[Python-ideas] x=(yield from) confusion [was:Yet another alternative name for yield-from]

[Guido van Rossum]

On Fri, Apr 3, 2009 at 10:47 AM, Jacob Holm <jh at improva.dk> wrote:
> You can see my original example at:
>
> http://mail.python.org/pipermail/python-ideas/2009-April/003841.html
>
> and a few arguments why I think it is better at:
>
> http://mail.python.org/pipermail/python-ideas/2009-April/003847.html
[...]
> The reason for closing would be that once you have computed the final
> result, you want whatever resources the coroutine is using to be freed.
> Since only the final result is assumed to be useful, it makes perfect sense
> to close the coroutine at the same time as you are requesting the final
> result.

Hm. I am beginning to see what you are asking for. Your averager
example is somewhat convincing.

Interestingly, in a sense it seems unrelated to yield-from: the
averager doesn't seem to need yield-from, it receives values sent to
it using send(). An alternative version (that works today), which I
find a bit clearer, uses exceptions instead of a sentinel value: when
you are done with sending it the sequence of values, you throw() a
special exception into it, and in response it raises another exception
back with a value attribute. I'm showing the usage example first, then
the support code.

Usage example:

@coroutine
def summer():
  sum = 0
  while True:
    try:
      value = yield
    except Terminate:
      raise Done(sum)
    else:
      sum += value

def main():
  a = summer()
  a.send(1)
  a.send(2)
  print finalize(a)

Support code:

class Terminate(Exception):
  """Exception thrown into the generator to ask it to stop."""

class Done(Exception):
  """Exception raised by the generator when it catches Terminate."""
  def __init__(self, value=None):
    self.value = value

def coroutine(func):
  """Decorator around a coroutine, to make the initial next() call."""
  def wrapper(*args, **kwds):
    g = func(*args, **kwds)
    g.next()
    return g
  return wrapper

def finalize(g):
  """Throw Terminate into a couroutine and extract the value from Done."""
  try:
    g.throw(Terminate)
  except Done as e:
    return e.value
  else:
    g.close()
    raise RuntimeError("Expected Done(<value>)")

I use a different exception to throw into the exception as what it
raises in response, so that mistakes (e.g. the generator not catching
Terminate) are caught, and no confusion can exist with the built-in
exceptions StopIteration and GeneratorExit.

Now I'll compare this manual version with your (Jacob Holm's) proposal:

- instead of Done you use GeneratorExit
- hence, instead of g.throw(Done) you can use g.close()
- instead of Terminate you use StopException
- you want g.close() to extract and return the value from StopException
- you use "return value" instead of "raise Done(value)"

The usage example then becomes, with original version indicated in comments:

@coroutine
def summer():
  sum = 0
  while True:
    try:
      value = yield
    except GeneratorExit:    # except Terminate:
      return sum    # raise Done(sum)
    else:
      sum += value

def main():
  a = summer()
  a.send(1)
  a.send(2)
  print a.close()    # print finalize(a)

At this point, I admin that I am not yet convinced. On the one hand,
my support code melts away, except for the @coroutine decorator. On
the other hand:

- the overloading of GeneratorExit and StopIteration reduces
diagnostics for common beginner's mistakes when writing regular
(iterator-style) generator code
- the usage example isn't much simpler
- the support code isn't rocket science
- the coroutine use case is specialized enough that a little support
seems okay (you still need @coroutine anyway)

The last three points don't sway me either way: they pit minor
conveniences against minor inconveniences.

However, the first point worries me a lot. The concern over
StopIteration can be dealt with by introducing a new exception that is
raised only by "return value" inside a generator.

But I'm also worried that the mere need to catch GeneratorExit for a
purpose other than resource cleanup will cause examples using it to
pop up on the web, which will then be copied and modified by clueless
beginners, and *increase* the probability of bad code being written.
(That's why I introduce *new* exceptions in my support code -- they
don't have predefined meanings in other contexts.)

Finally, I am not sure of the connection with "yield from". I don't
see a way to exploit it for this example. As an exercise, I
constructed an "averager" generator out of the above "summer" and a
similar "counter", and I didn't see a way to exploit "yield from". The
only connection seems to be PEP 380's proposal to turn "return value"
inside a generator into "raise StopIteration(value)", and that's the
one part of the PEP with which I have a problem anyway (the beginner's
issues above). Oh, and "yield from" competes with @couroutine over
when the initial next() call is made, which again suggests the two
styles (yield-from and coroutines) are incompatible.

All in all, I think I would be okay with turning "return value" inside
a generator into raising *some* exception, as long as that exception
is not StopIteration (nor derives from it, nor from GeneratorExit).
PEP 380 and its implementation would become just a tad more complex,
but I think that's worth it. Generators used as iterators would raise
a (normally) uncaught exception if they returned a value, and that's
my main requirement. I'm still not convince that more is needed, in
particular I'm still -0 on catching this value in gen_close() and
returning the value attribute from there.

As I've said before, I don't care whether "return None" would be
treated more like "return" or more like "return value" -- for
beginners' code I don't think it matters, and for advanced code they
should be equivalent.

I'll stop arguing for new syntax to return a value from a generator
(like Phillip Eby's proposed "return from yield with <value>"): I
don't think it adds enough to overcome the pain for the parser and
other tools.

Finally, as far as a name for the new exception, I think something
long like ReturnFromGenerator would be fine, since most of the time it
is handled implicitly by coroutine support code (whether this is user
code or gen_close()) or the yield-from implementation.

I'm sorry for being so long winded and yet somewhat inconclusive. I
wouldn't have bothered if I didn't think there was something worth
pursuing. But it sure seems elusive.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)