[Python-ideas] x=(yield from) confusion [was:Yet another alternative name for yield-from]

[Nick Coghlan]

Jim Jewett wrote:
> The times I did remember that (even) the expression form looped, I was
> still boggled that it would return something other than None after it
> was exhausted.  Greg's answer was that it was for threading, and the
> final return was the real value.  This seems like a different category
> of generator, but I could get my head around it -- so long as I forgot
> that the yield itself was returning anything useful.

Greg tried to clarify this a bit already, but I think Jacob's averager
example is an interesting case where it makes sense to both yield
multiple times and also "return a value".

While it is just a toy example, I believe it does a great job of
illustrating the control flow expectations. (Writing this email has
certainly clarified a lot of things about the PEP in my *own* mind).

The following reworking of Jacob's example assumes a couple of things
that differ from the current PEP:

- the particular colour my bikeshed is painted when it comes to
returning values from a generator is "return finally" (the idea being to
emphasise that this represents a special "final" value for the generator
that happens only after all of the normal yields are done).

- rather than trying to change the meaning of GeneratorExit and close(),
3 new generator methods would be added: next_return(), send_return() and
throw_return(). The new methods have the same signatures as their
existing counterparts, but if the generator raises GeneratorReturn, they
trap it and return the associated value instead. Like close(), they
complain with a RuntimeError if the generator doesn't finish. For example:

  def throw_return(self, *exc_info):
    try:
      self.throw(*exc_info)
      raise RuntimeError("Generator did not terminate")
    except GeneratorReturn as gr:
      return gr.value

(Note that I've also removed the 'yield raise' idea from the example -
if next() or send() triggers termination of the generator with an
exception other than StopIteration, then that exception is already
propagated into the calling scope by the existing generator machinery. I
realise Jacob was trying to make it possible to "yield an exception"
without terminating the coroutine, but that idea is well beyond the
scope of the PEP)

You then get:

  class CalcAverage(Exception): pass

  def averager(start=0):
    # averager that maintains a running average
    # and returns the final average when done
    count = 0
    exc = None
    sum = start
    while 1:
      avg = sum / count
      try:
        val = yield avg
      except CalcAverage:
        return finally avg
      sum += val
      count += 1

  avg = averager()
  avg.next() # start coroutine
  avg.send(1.0) # yields 1.0
  avg.send(2.0) # yields 1.5
  print avg.throw_return(CalcAverage) # prints 1.5

Now, suppose I want to write another toy coroutine that calculates the
averages of two sequences and then returns the difference:

  def average_diff(start=0):
    avg1 = yield from averager(start)
    avg2 = yield from averager(start)
    return finally avg2 - avg1

  diff = average_diff()
  diff.next() # start coroutine
              # yields 0.0
  avg.send(1.0) # yields 1.0
  avg.send(2.0) # yields 1.5
  diff.throw(CalcAverage) # Starts calculation of second average
                          # yields 0.0
  diff.send(2.0) # yields 2.0
  diff.send(3.0) # yields 2.5
  print diff.throw_return(CalcAverage) # Prints 1.0 (from "2.5 - 1.5")

The same example could be rewritten to use "None" as a sentinel value
instead of throwing in an exception (average_diff doesn't change, so I
haven't rewritten that part):

  def averager(start=0):
    count = 0
    exc = None
    sum = start
    while 1:
      avg = sum / count
      val = yield avg
      if val is None:
        return finally avg
      sum += val
      count += 1

  # yielded values are same as the throw_return() approach
  diff = average_diff()
  diff.next() # start coroutine
  diff.send(1.0)
  diff.send(2.0)
  diff.send(None) # Starts calculation of second average
  diff.send(2.0)
  diff.send(3.0)
  print diff.send_return(None) # Prints 1.0 (from "2.5 - 1.5")

Notice how the coroutines in this example can be thought of as simple
state machines that the calling code needs to know how to drive. That
state sequence is as much a part of the coroutine's signature as are the
arguments to the constructor and the final return value.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------