[Python-Dev] problem with recursive "yield from" delegation

[Stefan Behnel]

Hi,

I found a problem in the current "yield from" implementation that I think
is worth discussing:

http://bugs.python.org/issue14220

Test code:

    def g1():
        yield "y1"
        yield from g2()
        yield "y4"

    def g2():
        yield "y2"
        try:
            yield from gi
        except ValueError:
            pass  # catch "already running" error
        yield "y3"

    gi = g1()
    for y in gi:
        print("Yielded: %s" % (y,))

This is what it currently does:

1) g1() delegates to a new g2(), propagates its "y2" value and asks for the
next value

2) g2 delegates back to the g1 instance and asks for its next value

3) Python sees the active delegation in g1 and asks g2 for its next value

4) g2 sees that it's already running and throws an exception

Ok so far. Now:

5) the exception is propagated into g1 at call level 3) instead of the
original requestor g2 one level above

6) g1 undelegates and terminates by the exception

7) g2 catches the exception, yields "y3" and then terminates normally

8) g1 gets control back but has already terminated and does nothing

Effect: g1 does not yield "y4" anymore.

The problem is in steps 5) and 6), which are handled by g1 at the wrong
call level. They shouldn't lead to undelegation and termination in g1, just
to an exception being raised in g2.

I ran into this while trying to adapt the implementation for Cython, which
has a different generator type implementation but otherwise uses more or
less the same code now. But I'm not sure how to fix this one without major
changes to the implementation, especially not without special casing the
generator type on delegation (which won't work because CPython doesn't know
about Cython generators). Any ideas?

Stefan