[Python-ideas] Revised revised revised PEP on yield-from

[Guido van Rossum]

[Resend, hopefully bag.python.org is fixed again]

On Mon, Feb 16, 2009 at 1:52 PM, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> Guido van Rossum wrote:
>> (a) I don't know if the PEP proposes that "yield from expr" should
>> return the last value returned by (i.e. sent to) a yield somewhere
>> deeply nested; I think this would be useful.
>
> No, it doesn't. Its value is the value passed to the
> 'return' statement that terminates the subgenerator
> (and generators are enhanced to permit return with a
> value).
>
> My reason for doing this is so you can use subgenerators
> like functions in a generator that's being used as
> a lightweight thread.

There better be a pretty darn good reason to do this. I really don't
like overloading return this way -- normally returning from a
generator is equivalent to falling off the end and raises
StopIteration, and I don't think you can change that easily.

>> (b) I hope the PEP also explains what to do if "expr" is not a
>> generator but some other kind of iterator.
>
> Yes, it does. Currently I'm proposing that, if the
> relevant methods are not defined, send() is treated
> like next(), and throw() and close() do what they
> would have done normally on the parent generator.

I'm not sure I like this interpretation of .send() -- it looks
asymmetrical with the way .send() to non-generator iterators is
treated in other contexts, where it is an error. I'm fine with the
other two though, so I'm not sure how strong my opposition should be.

>> (c) A quick skim of the PEP didn't show suggestions for how to
>> implement this.
>
> One way would be to simply emit the bytecode corresponding
> to the presented expansion, although that wouldn't be
> very efficient in terms of either speed or code size.

Also pretty complex given the special cases for .send(), .throw() and
.close() -- if it weren't for pass-through it could b a simplified
for-loop (leaving out the variable assignment), but because of the
pass-through it seems it would have to be ugly.

> The PEP also sketches an optimised implementation in
> which the generator has a slot which refers to the
> generator being delegated to. Calls to next(), send(),
> throw() and close() are forwarded via this slot if it
> is nonempty.

And that could in turn be a generator with another such slot, right?
Hopefully the testing for the presence of .throw, .send and .close
could be done once at the start of the yield-from and represented as a
set of flags (or perhaps the test could be delayed until the first
time it's needed).

I recommend that you produce a working implementation of this; who
knows what other issues you might run into (including, whether your
proposed interpretation of return from a generator above makes sense.

> There will still be a small overhead involved in the
> delegation, but it's only a chain of C function calls
> instead of Python ones, which ought to be a big
> improvement.

Agreed.

> It might be possible to reduce the overhead even further
> by following the chain of delegation pointers in a loop
> until reaching the end and then calling the end
> generator directly. It would be trickier to get right,
> though, because you'd have to be prepared to back up
> and try earlier generators in the face of StopIterations.

Well there you have a question that could be answered by trying to implement it.

--
--Guido van Rossum (home page: http://www.python.org/~guido/)