At 07:08 AM 7/24/2010 -0700, Guido van Rossum wrote:

- After seeing Raymond's talk about monocle (search for it on PyPI) I am getting excited again about PEP 380 (yield from, return values from generators). Having read the PEP on the plane back home I didn't see anything wrong with it, so it could just be accepted in its current form.

I would like to reiterate (no pun intended) the suggestion of a special syntactic form for the return, such as "yield return x", or "return with x" or something similar, to distinguish it from a normal generator return. I think that when people are getting used to the idea of generators, it's important for them to get the idea that the function's "return value" isn't really a value, it's an iterator object. Allowing a return value, but then having that value silently disappear, seems like it would delay that learning, so, a special form might help to make it clear that the generator in question is intended for use with a corresponding "yield from", and help avoid confusion on this. (I could of course be wrong, and would defer to anyone who sees a better way to explain/teach around this issue. In any event, I'm +1 on the PEP otherwise.) By the way, the PEP's "optimized" implementation could probably be done just by making generator functions containing yield-from statements return an object of a different type than the standard geniter. Here's a Python implementation sketch, using a helper class and a decorator -- translation to a C version is likely straightforward, as it'll basically be this plus a light sprinkling of syntactic sugar. So, in the pure-Python prototype (without syntax sugaring), usage would look like this: @From.container def some_generator(...): ... yield From(other_generator(...)) # equivalent to 'yield from' ... def other_generator(...): ... raise StopIteration(value) # equivalent to 'return value' We mark some_generator() with @From.container to indicate that it uses 'yield from' internally (which would happen automatically in the C/syntax sugar version). We don't mark other_generator(), though, because it doesn't contain a 'yield from'. Now, the implementation code (a slightly altered/watered-down version of a trampoline I've used before in 2.x, hopefully altered correctly for Python 3.x syntax/semantics): class From: @classmethod def container(cls, func): def decorated(*args, **kw): return cls(func(*args, **kw)) # wrap generator in a From() instance return decorated def __new__(cls, geniter): if isinstance(geniter, cls): # It's already a 'From' instance, just return it return geniter self = object.__new__(cls, geniter) self.stack = [geniter] return self def __iter__(self): return self def __next__(self): return self._step() def send(self, value): return self._step(value) def throw(self, *exc_info): return self._step(None, exc_info) def _step(self, value=None, exc_info=()): if not self.stack: raise RuntimeError("Can't resume completed generator") try: while self.stack: try: it = self.stack[-1] if exc_info: try: rv = it.throw(*exc_info) finally: exc_info = () elif value is not None: rv = it.send(value) else: rv = it.next() except: value = None exc_info = sys.exc_info() if exc_info[0] is StopIteration: # pass return value up the stack value, = exc_info[1].args or (None,) exc_info = () # but not the error self.stack.pop() else: if isinstance(rv, From): stack.extend(rv.stack) # Call subgenerator value, exc_info, rv = None, (), None else: return rv # it's a value to yield/return else: # Stack's empty, so exit w/current return value or error if exc_info: raise exc_info[1] else: return value finally: exc_info = () # don't let this create garbage def close(self): if self.stack: try: # There's probably a cleaner way to do this in Py 3, I just # don't know what it is off the top of my head... raise GeneratorExit except GeneratorExit as e: try: self.throw(*sys.exc_info()) except (StopIteration, GeneratorExit): pass else: raise RuntimeError("Generator(s) failed to close()") # XXX probably needs some more code here to clean up stack def __del__(self): try: self.close() except: pass As you can (hopefully) see, the main code path simply ends up delegating next/send/close etc. directly to the iterator on the top of the stack, so there isn't any multi-layer passthru going on, as in the "non-optimized" version of the spec in the PEP.