[Python-Dev] problem with recursive "yield from" delegation

[Mark Shannon]

Stefan Behnel wrote:
> Hi,
> 
> I found a problem in the current "yield from" implementation that I think
> is worth discussing:
> 
> http://bugs.python.org/issue14220
> 
[snip]

I've been experimenting with the implementation of PEP 380, and I found 
a couple of interesting things.

First of all, the semantics described in the PEP do not match the tests.
If you substitute the supposedly semantically equivalent code
based on normal yields for each yield from in the test code
(Lib/test/test_pep380.py) and run it, then it fails.

My second experiment involved stripping away all the code relating to 
yield from outside of the interpreter and changing the YIELD_FROM
bytecode to repeat itself, by setting the last instruction to the 
instruction immediately before itself.
To do this I added 4 lines of code and removed over 120 lines :)

This fails many of the tests*, but works for the most straightforward
use cases. Many of these failures seem to me to be more 'natural' than 
the current behaviour

It might be possible to fix most, or all?, of the other failures
by compiling "yield from" into a two opcode sequence:
YIELD_FROM_START and YIELD_FROM_REPEAT.
Both opcodes should be reasonably simple and __next__() and send() would 
not have to worry about the subiterator, although close() and throw()
might (the subiterator would be top-of-stack in the generator's frame)

Overall the semantics of PEP 380 seem far too complex,
trying to do several things at once.
An example:
Plain yield makes no distinction between a receiving a None and any 
other value, so send(None) and __next__() are the same. Yield from
makes this distinction so has to test for None, meaning the semantics of 
send now changes with its argument.

I would recommend changing one of two things in the PEP:
Either, close and throw should not close/throw in subiterators
(this would simplify the semantics and implementation immensely)
Or, only allow subgenerators, not subiterators
(this would fix the next/send problem).

I would also suggest a change in implementation to perform all yields
within the interpreter. A simpler implementation is likely to be a
more reliable one.

Finally, the PEP itself makes no mention of coroutines, stackless or 
greenlet in the alternatives or criticisms section. Perhaps it should.

Cheers,
Mark.

*Tests in PEP 380. It passes all other tests.