[Python-ideas] x=(yield from) confusion [was:Yet another alternative name for yield-from]

[Guido van Rossum]

On Sun, Apr 5, 2009 at 7:46 AM, Jacob Holm <jh at improva.dk> wrote:
> The argument that we have no value to send before we have yielded is wrong.
> The generator containing the "yield-from" could easily have a value to send
> (or throw), and if iter(EXPR) returns a coroutine or a non-generator it
> could easily be ready to accept it.  That is the idea behind my attempted
> fixes to the @coroutine issue.

I think it's simpler to refrain from yield-from in that case and spell
it out. If the value to send doesn't come from outside the outer
generator, yield-from is not the solution.

> This is where the fun begins.  In an earlier thread we concluded that if the
> thrown exception is a StopIteration and the *same* StopIteration instance
> escapes the throw() call, it should be reraised rather than caught and
> turned into a RETVAL.  The reasoning was the following example:
>
> def inner():
>     for i in xrange(10):
>         yield i
>
> def outer():
>     yield from inner()
>     print "if StopIteration is thrown in we shouldn't get here"
>
> Which we wanted to be equivalent to:
>
> def outer():
>     for i in xrange(10):
>         yield i
>     print "if StopIteration is thrown in we shouldn't get here"
>
> The same argument goes for ReturnFromGenerator, so the expansion at this
> stage should be more like:
[snip]

This example and reasoning are invalid. You shouldn't be throwing
StopIteration (or ReturnFromGenerator) *into* a generator. That's
something that should only come *out*.

> Next issue is that the value returned by it.close() is thrown away by
> yield-from.  Here is a silly example:
>
> def inner():
>     i = 0
>     while True
>         try:
>             yield
>         except GeneratorExit:
>             return i
>         i += 1
>
> def outer():
>     try:
>         yield from inner()
>     except GeneratorExit:
>         # nothing I can write here will get me the value returned from inner()
>
> Also the trivial:
>
> def outer():
>     return yield from inner()
>
> Would swallow the return value as well.
>
> I have previously suggested attaching the return value to the (re)raised
> GeneratorExit, and/or saving the return value on the generator and making
> close return the value each time it is called.  We could also choose to
> define this as broken behavior and raise a RuntimeError, although it seems a
> bit strange to have yield-from treat it as an error when close doesn't.
> Silently having the yield-from construct swallow the returned value is my
> least favored option.

Attaching it to the GeneratorExit is just plain wrong -- this is an
exception you throw *in*, not something that is thrown out (except
when it bounces back).

One solution is not to use yield-from but write it out using yield and
send (just like the full expansion, but you can probably drop most of
the complexity for any particular example).

Another solution is not to use close() and GeneratorExit but some
application-specific exception to signal the end.

But perhaps it would be okay to change the GeneratorExit handler in
the expansion so that it passes through the return value with a
StopIteration exception:

rv = it.close()
if rv is None:
  raise StopIteration(rv)    # Or ReturnFromGenerator(rv)
else:
  raise

Alternatively, simpler:

it.throw(GeneratorExit)
# We only get here if it yielded a value
raise RuntimeError(...)

(Though this isn't exactly if we were to use duck typing.)

We could then write the first version of outer() like this:

def outer():
  try:
    yield from inner()
  except StopIteration as e:
    ...access return value as e.value...

and I think the second (trivial) outer() will return inner()'s return
value just fine, since it just passes through as a StopIteration
value.

> Like Greg, I am in favor of duck-typing this as closely as possible.

OK, noted. I think it's probably fine.

FWIW, I'm beginning to think that ReturnFromGenerator is a bit of a
nuisance, and that it's actually fine to allow "return value" inside a
generator to mean "raise StopIteration(value)" (well not quite at that
point in the code but once we are about to clean up the frame). Maybe
I've overstated the case for preventing beginners' mistakes. After all
they'll notice that their generator returns prematurely when they
include any kind of return value. Also if the StopIteration ends being
printed as a traceback the value will be printed, which is the kind of
hint newbies love.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)