[Python-Dev] Stackless Python - Pros and Cons

Eric S. Raymond esr@thyrsus.com
Sun, 6 Aug 2000 17:59:35 -0400


Jack Jansen <jack@oratrix.nl>:
> Could the defenders of Stackless Python please explain _why_ this is
> such a great idea? Just and Christian seem to swear by it, but I'd
> like to hear of some simple examples of programming tasks that will be 
> programmable in 50% less code with it (or 50% more understandable
> code, for that matter).

My interest in Stackless is that I want to see Icon-style generators,
co-routining, and first-class continuations in Python.  Generators and
co-routining are wrappers around continuations.  Something
functionally equivalent to the Stackless mods is needed to get us
there, because using the processor stack makes it very hard to do
continuations properly.

In their full generality, first-class continuations are hard to think
about and to explain clearly, and I'm not going to try here.  A large
part of Guido's reluctance to introduce them is precisely because they
are so hard to think about; he thinks it's a recipe for trouble stuff
in the language that *he* has trouble understanding, let alone other
people.

He has a point, and part of the debate going on in the group that has
been tracking this stuff (Guido, Barry Warsaw, Jeremy Hylton, Fred
Drake, Eric Tiedemann and myself) is whether Python should expose
support for first-class continuations or only "safer" packagings such
as coroutining and generators.  So for the moment just think of
continuations as the necessary primitive to implement coroutining and
generators.

You can think of a generator as a function that, internally, is coded 
as a special kind of loop.  Let's say, for example, that you want a function
that returns successive entries in the list "squares of integers".  In 
Python-with-generators, that would look something like this.

def square_generator():
    i = 1
    while 1:
	yield i**2
	i = i + 1

Calling this function five times in succession would return 1, 4, 9,
16, 25.  Now what would be going on under the hood is that the new primitive
"yield" says "return the given value, and save a continuation of this
function to be run next time the function is called".  The continuation 
saves the program counter and the state of automatic variables (the stack)
to be restored on the next call -- thus, execution effectively resumes just
after the yield statement.

This example probably does not look very interesting.  It's a very trivial
use of the facility.  But now suppose you had an analogous function 
implemented by a code loop that gets an X event and yields the event data!

Suddenly, X programs don't have to look like a monster loop with all the
rest of the code hanging off of them.  Instead, any function in the program
that needs to do stateful input parsing can just say "give me the next event"
and get it.  

In general, what generators let you do is invert control hierarchies
based on stateful loops or recursions.  This is extremely nice for
things like state machines and tree traversals -- you can bundle away
the control loop away in a generator, interrupt it, and restart it
without losing your place.

I want this feature a lot.  Guido has agreed in principle that we ought
to have generators, but there is not yet a well-defined path forward to
them.  Stackless may be the most promising route.

I was going to explain coroutines separately, but I realized while writing
this that the semantics of "yield" proposed above actually gives full
coroutining.  Two functions can ping-pong control back and forth among
themselves while retaining their individual stack states as a pair of
continuations.
-- 
		<a href="http://www.tuxedo.org/~esr">Eric S. Raymond</a>

"This country, with its institutions, belongs to the people who
inhabit it. Whenever they shall grow weary of the existing government,
they can exercise their constitutional right of amending it or their
revolutionary right to dismember it or overthrow it."
	-- Abraham Lincoln, 4 April 1861