PEP 255: Simple Generators

[Tim Peters]

[Glyph Lefkowitz]
> If python's answer to "what should we do with new feature X"? is
> "add a keyword" it will turn into Perl before too long.

Since that's very rarely Python's answer, this is slippery-slope FUD shouted
from a flat plane.

> ...
> And yes, we should wait indefinitely for microthreads.

You're welcome to, but, e.g., I've wanted Python generators for 10 years,
and have no pressing use for microthreads.  I don't object to you getting
the latter, but they're gross overkill for most things I want to do.
Generators are much simpler and often suffice.

> If generators are implemented as a syntax feature, then microthreads
> are implemented as a module

I see no logical connection there:  whether microthreads grow new syntax is
an independent issue.

> (or should we have syntax for that too?  'task' anyone?)

That's something for the microthreads PEP to stake out.  You know that
implementing the semantics efficiently in Jython too has been a severe
impediment to Stackless adoption, right?  You may very well *need* new
keywords for any gimmick like these to get around that:  Finn and Samuele
cannot change the JVM, and last I heard even Christian gave up on that tack
(which is saying something -- he routinely redefines truth <wink>).

> then we're again falling prey to a sin of perl; there will be more
> than one way to do it.

Generator semantics are simple and synchronous.  Nobody in their right mind
codes them "by hand" using a thread model instead (voice of experience
there -- and, no, I haven't always been in my right mind <wink>).  For the
same reasons, Python would retain "while" loops even if Guido added a
"goto".

BTW, you're missing the slippery slope argument that *may* have merit in
real life:  Just was almost certainly correct in predicting that the
adoption of generators is the best thing that could possibly happen for
Stackless's prospects.  The audience for microthreads is enthusiastic but
small, and the best shot it has is to ride in as a generalization of
something else.

> This is, after all, not a new *capability* of python per se, like the
> ability to parse XML or display 3D objects using OpenGL; it's just an
> attempt to build a particular idiom into the language, by providing a
> convenience.

The ability to resume a function, and efficiently, is a new capability --
and a powerful one, and a useful one.

> ...
> At least from my point of view, generators are a solution to a problem
> which does not exist.  There's nothing I can see which would be
> *significantly* easier using generators.

There's nothing I can see which would be significantly easier using
microthreads:  you could use real threads instead.  If you're eager to
ignore the *pragmatics* of generators, then I'm happy to repay the favor wrt
microthreads.  Reducing development time, increasing expressivity,
increasing efficiency, and improving scalability aren't fundamentally new
capabilities either -- they're "merely convenient" too, for sufficiently
strained rhetorical distortions of "merely convenient".

> In addition, the implementation side of this is just an efficiency hack.

To the contrary, Neil's implementation is so elegant it makes me smile --
calling it a hack is at best in bad taste.  If you looked into it instead of
ranting about it, you'd discover that the things he did to the eval loop are
necessary preconditions for implementing microthreads too -- the ability to
suspend a frame in midstream is fundamental to all these gimmicks.  This
will be the end of the path for most, but it takes you closer to your goal.

> It's not hard to make this work using threads; but since python doesn't
> have real microthreads, generators won't be built on top of anything
> except the interpreter itself; and they will hinder the future
> implementation of microthreads should they happen

To the contrary, they're probably microthreads' best hope.

> and probably cause some pretty fruity interoperability issues (if it's
> a syntax feature).

Possibly, depending on how lame the microthread implementation is <0.9
wink>.

> New users should not be using generators -- it's an advanced control
> structure with some interesting uses, but neither would I try to explain
> to a newbie how to write a task scheduler by putting hooks into ceval.c.
> Programming is just hard sometimes.  (This is one of the reasons I use
> python and *NOT* icon; for me, generators are not an important enough
> feature to worm their way into everything.)

Generators aren't becoming central to Python; they're an especially powerful
instance of 2.2's iterator interface (a point you should consider:
generators live in the iterator world, not in the thread world, and
iterators are ubiquitous in all kinds of apps; thread-like control flow has
a much smaller audience, and smaller still if the "threads" block at
synchronous I/O calls, and smaller still again if they don't act like "real
threads" when running in foreign code).

if-you-want-to-be-a-millionaire-you-only-have-to-collect-1000-dollars-
    1000-times<wink>-ly y'rs  - tim