[Python-Dev] yield back-and-forth?

[Nick Coghlan]

Phillip J. Eby wrote:
>> Thoughts?
> 
> If we have to have a syntax, "yield from sub_generator()" seems clearer 
> than "yieldthrough", and doesn't require a new keyword.

Andrew Koenig suggested the same phrasing last year [1], and I liked it then. 
I don't like it any more, though, as I think it is too inflexible, and we have 
a better option available (specifically, stealing "continue with an argument" 
from PEP 340). The following ramblings (try to) explain my reasoning :)

Guido does raise an interesting point. The introduction of "send" and "throw" 
means that the current simple loop approach does not easily allow values to be 
passed down to nested generators, nor does not it correctly terminate nested 
generators in response to an invocation of "throw". Because of the explicit 
for loop, the nested generator only gets cleaned up in response to GC - it 
never sees the exception that occurs in the body of the for loop (at the point 
of the yield expression).

The "yield from iterable" concept could be translated roughly as follows:

   itr = iter(iterable)
   try:
       send_input = itr.send  # Can input values be passed down?
   except AttributeError:
       send_input = None
   try:
       next = itr.next()      # Get the first output
   except StopIteration:
       pass
   else:
       while 1:
           try:
               input = yield next  # yield and get input
           except:
               try:
                  throw_exc = itr.throw  # Can exception be passed down?
               except AttributeError:
                   raise                     # Nope, reraise
               else:
                   throw_exc(sys.exc_info()) # Yep, pass it down
           else:
               try:
                   if send_input is None:
                       if input is not None:
                           raise TypeError("Cannot send input!")
                       next = itr.next()
                   else:
                       next = send_input(input) # Pass input down
               except StopIteration:
                   break

I'm not particularly happy with this, though, as not only is it horribly 
implicit and magical, it's trivial to accidentally break the chain - consider 
what happens if you naively do:
   yield from (x*x for x in sub_generator())

The chain has been broken - the sub generator no longer sees either passed in 
values or thrown exceptions, as the generator expression intercepts them 
without passing them down. Even worse, IMO, is that the syntax is entirely 
inflexible - we have no easy way to manipulate either the results sent from 
the generator, or the input values passed to it.

However, an idea from Guido's PEP 340 helps with the "send" part of the story, 
involving passing an argument to continue:

   def main_generator():
       ...
       for value in sub_generator():
           continue yield value

Here, sub_generator's "send" method would be invoked with the result of the 
call to yield value. Manipulation in either direction (input or output) is 
trivial:

   def main_generator():
       ...
       for value in sub_generator():
           input = yield value*value   # Square the output values
           continue input*input        # Square the input values, too

You could even do odd things like yield each value twice, and then pass down 
pairs of inputs:

   def main_generator():
       ...
       for value in sub_generator():
           continue (yield value), (yield value)

The need to use a "continue" statement eliminates the temptation to use a 
generator expression, and makes it far less likely the downwards connection 
between the main generator and the sub generator will be accidentally broken.

Exception propagation is a different story. What do you want to propagate? All 
exceptions from the body of the for loop? Or just those from the yield statement?

Well, isn't factoring out exception processing part of what PEP 343 is for?

   # Simply make sure the generator is closed promptly
   def main_generator():
       ...
       with closing(sub_generator()) as gen:
           for value in gen:
               continue yield value

   # Or throw the real exception to the nested generator
   class throw_to(object):
       def __init__(self, gen):
           self.gen = gen
       def __enter__(self):
           return self.gen
       def __exit__(self, exc_type, *exc_details):
           if exc_type is not None:
               try:
                   self.gen.throw(exc_type, *exc_details)
               except StopIteration:
                   pass

   def main_generator():
       ...
       with throw_to(sub_generator()) as gen:
           for value in gen:
               continue yield value

   # We can even limit the propagated exceptions to those
   # from the outside world and leave the rest alone
   def main_generator():
       ...
       gen = sub_generator()
       for value in gen:
           with throw_to(gen):
               input = yield value
           continue input


Thanks to Jason's other thread, I even have a hypothetical use case for all 
this. Specifically, "generator decorators", that manipulate the state that 
holds while executing the generator. Suppose I have a number of generators 
that I want to work in high precision decimal, but yield results with normal 
precision. Doing this dance manually in every generator would be a pain, so 
let's use a decorator:

      def precise_gen(gen_func):
         def wrapper(*args, **kwds):
             orig_ctx = decimal.getcontext()
             with orig_ctx as ctx:
                 ctx.prec = HIGH_PRECISION
                 gen = gen_func(*args, **kwds)
                 for val in gen:
                     with orig_ctx:
                         val = +val
                         try:
                             yield val
                         except:
                             gen.throw() # make it default to sys.exc_info()
         wrapper.__name__ = gen_func.__name__
         wrapper.__dict__ = gen_func.__dict__
         wrapper.__doc__ = gen_func.__doc__
         return wrapper

So far, so good (although currently, unlike a raise statement, gen.throw() 
doesn't automatically default to sys.exc_info()). Tough luck, however, if you 
want to use this on a generator that accepts input values - those inputs will 
get dropped on the floor by the wrapper, and the wrapped generator will never 
see them.

"yield from" wouldn't help in the least, because there are other things going 
on in the for loop containing the yield statement. 'continue with argument', 
however, would work just fine:

      def precise_gen(gen_func):
         def wrapper(*args, **kwds):
             orig_ctx = decimal.getcontext()
             with orig_ctx as ctx:
                 ctx.prec = HIGH_PRECISION
                 gen = gen_func(*args, **kwds)
                 for val in gen:
                     with orig_ctx:
                         val = +val
                         try:
                             continue yield val
                         except:
                             gen.throw() # make it default to sys.exc_info()
         wrapper.__name__ = gen_func.__name__
         wrapper.__dict__ = gen_func.__dict__
         wrapper.__doc__ = gen_func.__doc__
         return wrapper

Now an arbitrary generator can be decorated with "@precise_gen", and it's 
internal operations will be carried out with high precision, while any interim 
results will be yielded using the caller's precision and the caller's rounding 
rules.

Cheers,
Nick.

[1] http://mail.python.org/pipermail/python-dev/2005-October/057410.html

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
             http://www.boredomandlaziness.org