[Python-ideas] Possible PEP 380 tweak

[Jacob Holm]

On 2010-10-28 01:46, Guido van Rossum wrote:
> Nick & Jacob,
> 
> Unfortunately other things are in need of my attention and I am
> quickly lagging behind on this thread.
> 

Too bad, but understandable.  I'll try to be brief(er).


> I'll try to respond to some issues without specific quoting.
> 
> If GeneratorReturn and finish() can be implemented in pure user code,
> then I think it should be up to every (framework) developer to provide
> their own API, using whatever constraints they chose. Without specific
> use cases it's hard to reason about API design.

GeneratorReturn and finish *can* be implemented in pure user code, as
long as you accept that the premature return has to use some other
mechanism than "return" or StopIteration.



> Still, I think it is
> reasonable to offer some basic behavior on the generator object, and I
> still think that the best compromise here is to let g.close() extract
> the return value from StopIteration if it catches it. If a framework
> decides not to use this, fine. For a user working without a framework
> this is still just a little nicer than having to figure out the
> required logic yourself.
> 

This works only as long as you don't actually use yield-from, making it
a bit of a strange match to that PEP.  To get it to work *with*
yield-from you need the reraised GeneratorExit to include the return
value (possibly None) from the inner generator.  I seem to have
convinced Nick that the problem is real and that a modification to the
expansion might be needed/desirable.



> I am aware of four relevant states for generators. Here's how they
> work (in current Python):
> 
> - initial state: execution is poised at the top of the function.
> g.throw() always bounces back the exception. g.close() moves it to the
> final state. g.next() starts it running. g.send() requires a None
> argument and is then the same as g.next().
> 
> - running state: the frame is active. none of g.next(), g.send(),
> g.throw() or g.close() work -- they all raise ValueError.
> 
> - suspended state: execution is suspended at a yield. g.close() raises
> GeneratorExit and if the generator catches this it can do whatever it
> pleases. If it then raises StopIteration or GeneratorExit, g.close()
> is happy, if it raises another exception g.close() just passes that
> through, if it yields a value g.close() complains and raises
> RuntimeError().
> 
> - finished (exhausted) state: the generator has returned. g.close()
> always return None. g.throw() always bounces back the exception.
> g.next() and g.send() always raise StopIteration.
> 
> I would be in favor of adding an introspection API to distinguish
> these four states and I think it would be a fine thing to add to
> Python 3.2 if anyone finds the time to produce a patch (Nick? You
> showed what these boil down to.)
> 
> I note that in the initial state a generator has no choice in how to
> respond because it hasnt't yet had the opportunity to set up a
> try/except, so in this state it acts pretty much the same as in the
> exhausted state when receiving a throw() or close().
> 

Yes, I forgot about this case in the versions of "finish" that I wrote.
 Nick showed a better version that handled it properly.


> Regarding built-in syntax for Go-like channels, let's first see an
> implementation in userland become successful *or* see that it's
> impossible to write an efficient one before adding more to the
> language.
> 

It is impossible in current python to use a for-loop or generator
expression to loop over a Go-like channel without using threads for
everything.  (The only way to suspend the iteration is to suspend the
thread, and then whatever code is supposed to write to the channel must
be running in another thread)  This is a shame, since the blocking
nature of channels otherwise make them ideal for cooperative multitasking.

Note, this restriction (no for-loop iteration without threads) does not
make channels useless in current python, just much less convenient to
work with.  That, unfortunately, makes it less likely that a userland
implementation will ever become successful.


> Note that having a different expansion of a for-loop based on the
> run-time value or type of the iterable cannot be done -- the expansion
> can only vary based on the syntactic form.
> 

The intent was to have a different expansion depending on the type of
function containing the for-loop (as in regular/cofunction).  I think I
made a few errors though, so the new expansion doesn't actually work
with regular iterables.  If I get around to fixing it I'll post the fix
in that thread.


> There are a few different conventions for using generators and
> yield-from; e.g. generators used as proper iterators with easy
> refactoring; generators used as tasks where yield X is used for
> blocking I/O operations; and generators used as "inverse generators"
> as in the parallel_reduce() example that initiated this thread. I
> don't particularly care about what kind of errors you get if a
> generator written for one convention is accidentally used by another
> convention, as long as it is made clear which convention is being used
> in each case. Frameworks/libraries can and probably should develop
> decorators to mark up the 2nd and 3rd conventions, but I don't think
> the *language* needs to go out of its way to enforce proper usage.
> 

Agreed, I think.

- Jacob