[Python-Dev] Re: Minimal 'stackless' PEP using generators?

[Clark C. Evans]

Thanks for responding Terry.   I'm not sure if my exposition was clear.

On Mon, Aug 23, 2004 at 08:33:09PM -0400, Terry Reedy wrote:
| > 1.  The PEP would introduce a new 'builtin' class called 'Cooperate'
| >
| > 2.  Generator semantics would be altered so that 'yield X', where X
| >     is an instance of Cooperate, would automagically propigate to
| >     the outer-most non-generator.

Suppose I have three 'frames', two generators and one function, the
call stack looks like:

     g2
     g1
     fn
     
For this to occur, fn has called .next() on g1, which has called .next()
on g2.  The proposed behavior is that "yield magic.Collaborate" provides
the value returned by g1.next() within fn.  In particular, it would be
equivalent as if every "val = g2.next()" in g1 was rewritten:

    val = g2.next()
    if isinstance(val, magic.Cooperate):
        yield val

One could do this in three ways: (a) by a convention above, this is done
in twisted.flow, (b) by a bytecode hack, or (c) by modifying eval.c
to have equivalent behavior.

| 1. This alters and complexifies the current generator concept.

Correct; it introduces something similar to a corountine.

| 2. This imposes an isinstance(x, magic.Collaborate) function call cost on
| every yield in every generator.

Ok.  So, in addition to causing unexpected results (one would have to
check to see what sort of object to debug), it would also cause a 
performance hit.

| 3. 'automagic' it is.  The pertinent runtime call stack contains not
| generator objects but .next methods (or in CPython, method-wrappers).  So I
| presume you are proposing that yielded Collaborates magically plunk
| themselves into the first frame up the stack not associated such objects.

Right, the first frame that is not a generator, aka, not having a yield
statement.  And yes, it is different semantics.

| 4. If the returned to frame does not re-call the passed over suspended
| frames, they and anything they hold onto are leaked until gc'ed.

No.  See the above equivalence.  It might take a bit of thinking
to make the "C" implementation correct, but the semantics are 
farily straight-forward and should not create leaks.

| Here, the effect you seek could be accomplished by an explicit 'if not
| isinstance(x, Collaborate):' before the print statement (or, in general,
| the block of code that uses x).  But what if top is rewritten as
| def top(): return ("one", MyCooperate(), "two")
| Do you really want the behavior of middle to change, as with your proposal?

Yes.

| >     >>> def lower():
| >     >>>     """ this is not a generator, so it sees cooperate """
| >     >>>     for x in middle():
| >     >>>         print "lower", x
| 
| Now add 'if isinstance(x, Cooperate): return' and the suspended middle will
| be left in limbo.  Not very tidy ;-)

No, it would be just like a generator had 'if 1: return'; the suspended
middle would be cleaned up just as it is today.

...

However, you've convinced me on the semantic differences and the
performance hit.  I guess the proposal would have to present a
new keyword, say 'collaborate' and have to have its semantics
clearly specified.

Thank you _so_ much for your kind review,

Clark