[Python-Dev] PEP 380 - return value question and prototype implementation (was Thoughts fresh after EuroPython)

[P.J. Eby]

At 07:08 AM 7/24/2010 -0700, Guido van Rossum wrote:
>- After seeing Raymond's talk about monocle (search for it on PyPI) I
>am getting excited again about PEP 380 (yield from, return values from
>generators). Having read the PEP on the plane back home I didn't see
>anything wrong with it, so it could just be accepted in its current
>form.

I would like to reiterate (no pun intended) the suggestion of a 
special syntactic  form for the return, such as "yield return x", or 
"return with x" or something similar, to distinguish it from a normal 
generator return.

I think that when people are getting used to the idea of generators, 
it's important for them to get the idea that the function's "return 
value" isn't really a value, it's an iterator object.  Allowing a 
return value, but then having that value silently disappear, seems 
like it would delay that learning, so, a special form might help to 
make it clear that the generator in question is intended for use with 
a corresponding "yield from", and help avoid confusion on this.

(I could of course be wrong, and would defer to anyone who sees a 
better way to explain/teach around this issue.  In any event, I'm +1 
on the PEP otherwise.)

By the way, the PEP's "optimized" implementation could probably be 
done just by making generator functions containing yield-from 
statements return an object of a different type than the standard 
geniter.  Here's a Python implementation sketch, using a helper class 
and a decorator -- translation to a C version is likely 
straightforward, as it'll basically be this plus a light sprinkling 
of syntactic sugar.

So, in the pure-Python prototype (without syntax sugaring), usage 
would look like this:

     @From.container
     def some_generator(...):
         ...
         yield From(other_generator(...))  # equivalent to 'yield from'
         ...

     def other_generator(...):
         ...
         raise StopIteration(value)  # equivalent to 'return value'


We mark some_generator() with @From.container to indicate that it 
uses 'yield from' internally (which would happen automatically in the 
C/syntax sugar version).  We don't mark other_generator(), though, 
because it doesn't contain a 'yield from'.

Now, the implementation code (a slightly altered/watered-down version 
of a trampoline I've used before in 2.x, hopefully altered correctly 
for Python 3.x syntax/semantics):

class From:
     @classmethod
     def container(cls, func):
         def decorated(*args, **kw):
             return cls(func(*args, **kw))  # wrap generator in a 
From() instance
         return decorated

     def __new__(cls, geniter):
         if isinstance(geniter, cls):
             # It's already a 'From' instance, just return it
             return geniter
         self = object.__new__(cls, geniter)
         self.stack = [geniter]
         return self

     def __iter__(self):
         return self

     def __next__(self):
         return self._step()

     def send(self, value):
         return self._step(value)

     def throw(self, *exc_info):
         return self._step(None, exc_info)

     def _step(self, value=None, exc_info=()):
         if not self.stack:
             raise RuntimeError("Can't resume completed generator")
         try:
             while self.stack:
                 try:
                     it = self.stack[-1]
                     if exc_info:
                         try:
                             rv = it.throw(*exc_info)
                         finally:
                             exc_info = ()
                     elif value is not None:
                         rv = it.send(value)
                     else:
                         rv = it.next()
                 except:
                     value = None
                     exc_info = sys.exc_info()
                     if exc_info[0] is StopIteration:
                         # pass return value up the stack
                         value, = exc_info[1].args or (None,)
                         exc_info = ()   # but not the error
                     self.stack.pop()
                 else:
                     if isinstance(rv, From):
                         stack.extend(rv.stack)  # Call subgenerator
                         value, exc_info, rv = None, (), None
                     else:
                         return rv   # it's a value to yield/return
             else:
                 # Stack's empty, so exit w/current return value or error
                 if exc_info:
                     raise exc_info[1]
                 else:
                     return value
         finally:
             exc_info = ()   # don't let this create garbage

     def close(self):
         if self.stack:
             try:
                 # There's probably a cleaner way to do this in Py 3, I just
                 # don't know what it is off the top of my head...
                 raise GeneratorExit
             except GeneratorExit as e:
                 try:
                     self.throw(*sys.exc_info())
                 except (StopIteration, GeneratorExit):
                     pass
                 else:
                     raise RuntimeError("Generator(s) failed to close()")
                     # XXX probably needs some more code here to clean up stack

     def __del__(self):
         try:
             self.close()
         except:
             pass

As you can (hopefully) see, the main code path simply ends up 
delegating next/send/close etc. directly to the iterator on the top 
of the stack, so there isn't any multi-layer passthru going on, as in 
the "non-optimized" version of the spec in the PEP.