[Python-ideas] Yield-From: Handling of GeneratorExit

[Greg Ewing]

Guido van Rossum wrote:

> The feature doesn't exist for the benefit of well-behaved generators.

I know. I just mean that it won't break any existing
correct generators, and should allow reasonably-written
future generators to behave reasonably.

> It exists to help people who don't understand generators well enough
> yet to only write well-behaved ones.

If you haven't delved into the details of generators,
you won't know about GeneratorExit, so you won't be
trying to catch it.

> I need a longer description of the problems that you are trying to
> solve here

It's a bit subtle, but I'll try to recap. Suppose
Fred writes the following generator:

   def fred():
     try:
       yield 1
       x = 17
     except GeneratorExit:
       x = 42
     print "x =", x

By current standards, this is a legitimate generator.
Now, the refactoring principle suggests that it
should be possible to rewrite it like this:

   def fred_outer():
     x = yield from fred_inner()
     print "x =", x

   def fred_inner():
     try:
       yield 1
       x = 17
     except GeneratorExit:
       x = 42
     return x

If we treat GeneratorExit just like any other exception
and throw it into the subgenerator, this does in fact
work.

Now for the problem: Suppose Mary comes along and wants
to re-use Fred's inner generator. She writes this:

   def mary():
     y = yield from fred_inner()
     print "y =", y
     yield 2

If close() is called on Mary's generator while it's
suspended inside the call to fred_inner(), a RuntimeError
occurs, because the GeneratorExit got swallowed and
Mary tried to do another yield.

This is not reasonable behaviour, because Mary didn't
do anything wrong. Neither did Fred do anything wrong
when he wrote fred_inner() -- it's a perfectly well-
behaved generator by current standards. But put the
two together and a broken generator results.

One way to fix this is to place a small restriction
on the refactoring principle: we state that you can't
factor out a block of code that catches GeneratorExit
and doesn't reraise it before exiting the block.

This allows us to treat GeneratorExit as a special case,
and always reraise it regardless of what the subiterator
does. Mary's generator is then no longer broken. Fred's
doesn't work any more, but he can't complain, because
he performed an invalid refactoring.

My proposal for changing the way close() works is just
an alternative way of tackling this problem that would
remove the need for special-casing GeneratorExit either
in the expansion or the statement of the refactoring
principle, and allow generators such as Fred's above
to work.

> Please, please, please, we need to stop the bikeshedding and scope
> expansion, and start converging to a *simpler* proposal.

I'm all in favour of simplicity, but it's not clear
what is simpler here. There's a tradeoff between
complexity in the yield-from expansion and complexity
in the behaviour of close().

BTW, if you're after simplicity, I still think that
using a different exception to return values from
generators, and using a different syntax to do so,
are both unnecessary complications.

-- 
Greg

>