[Answering somewhat out of order; new proposal developed at the end.]<br><br>On Fri, Apr 3, 2009 at 4:25 PM, Jacob Holm <<a href="mailto:jh@improva.dk" target="_blank">jh@improva.dk</a>> wrote:<br>> I am still trying to get a clear picture of what kind of mistakes you are<br>
> trying to protect against.<br>> If it is people accidently writing return in a generator when they really<br>> mean yield, that is what I thought the proposal for an alternate syntax was<br>> for. That sounds like a good idea to me, especially if we could also ban or<br>
> discourage the use of normal return. But the alternate syntax doesn't have<br>> to mean a different exception.<br><br>I am leaning the other way now. New syntax for returning in a value is a high-cost proposition. Instead, I think we can guard against most of the same mistakes (mixing yield and return in a generator used as an iterator) by using a different exception to pass the value. This would delay the failure to runtime, but it would still fail loudly, which is good enough for me.<br>
<br>I want to name the new exception ReturnFromGenerator to minimize the similarity with GeneratorExit: if we had both GeneratorExit and GeneratorReturn there would be endless questions on the newbie forums about the differences between the two, and people might use the wrong one. Since ReturnFromGenerator goes *out* of the generator and GeneratorExit goes *in*, there really are no useful parallels, and similar names would cause confusion.<br>
<br>> I could easily see "return value" as a separate PEP, except PEP 380<br>> provides one of the better reasons for its inclusion. It might be good to<br>> figure out how this feature should work by itself before complicating things<br>
> by integrating it in the yield-from semantics.<br><br>Here are my curent thoughts on this. When a generator returns, the return statement is treated normally (whether or not it has a value) until the frame is about to be left (i.e. after any finally-clauses have run). Then, it is converted to StopIteration if there was no value or ReturnFromGenerator if there was a value. I don't care which one is picked for an explicit "return None" -- that should be decided by implementation expediency. (E.g. if one requires adding new opcodes and one doesn't, I'd pick the one that doesn't.)<br>
<br>Normal loops (for-loops, list comprehensions, other implied loops) only catch StopIteration, so that returning a value is still wrong here. But some other contexts treat ReturnFromGenerator similar as StopIteration except the latter conveys None and the former conveys an explicit value. This applies to yield-from as well as to explicit or implied closing of the generator (close() or deallocation).<br>
<br>So g.close() returns the value (I think I earlier said I didn't like that -- I turned around on this one). It's pseudo-code is roughly:<br><br><span style="font-family: courier new,monospace;">def close(it):</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> try:</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> it.throw(GeneratorExit)</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> except (GeneratorExit, StopIteration):</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> return None</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> except ReturnFromGenerator as e: # This block is really the only new thing</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> return e.value</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> # Other exceptions are passed out unchanged</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> else:</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> # throw() yielded a value -- unchanged</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> raise RuntimeError(.....)</span><br style="font-family: courier new,monospace;">
<br>Deleting a generator is like closing and printing (!) a traceback (to stderr) if close() raises an exception. A returned value it is just ignored. Explicit pseudo-code without falling back to close():<br><br><span style="font-family: courier new,monospace;">def __del__(it):</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> try:</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> it.throw(GeneratorExit)</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> except (GeneratorExit, StopIteration, ReturnFromGenerator):</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> pass</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> except:</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> # Some other exception happened</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> <print traceback></span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> else:</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> # throw() yielded another value</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> <print traceback></span><br>
<br>I have also worked out what I want yield-from to do, see end of this message.<br><br>[Guido]<br>>> Oh, and "yield from" competes with @couroutine over<br>>> when the initial next() call is made, which again suggests the two<br>
>> styles (yield-from and coroutines) are incompatible.<br>><br>> It is a serious problem, because one of the major points of the PEP is that<br>> it should be useful for refactoring coroutines. As a matter of fact, I<br>
> started another thread on this specific issue earlier today which only Nick<br>> has so far responded to. I think it is solvable, but requires some more<br>> work.<br><br>I think that's the thread where I asked you and Nick to stop making more proposals.I a worried that a solution would become too complex, and I want to keep the "naive" interpretation of "yield from EXPR" to be as close as possible to "for x in EXPR: yield x". I think the @coroutine generator (whether built-in or not) or explicit "priming" by a next() call is fine.<br>
<br>-----<br><br>So now let me develop my full thoughts on yield-from. This is unfortunately long, because I want to show some intermediate stages. I am using a green font for new code. I am using stages, where each stage provides a better approximation of the desired semantics. Note that each stage *adds* some semantics for corner cases that weren't handled the same way in the previous stage. Each stage proposes an expansion for "RETVAL = yield from EXPR". I am using Py3k syntax.<br>
<br>1. Stage one uses the for-loop equivalence:<br><br><span style="font-family: courier new,monospace; color: rgb(0, 0, 0);">for x in EXPR:</span><br style="font-family: courier new,monospace; color: rgb(0, 0, 0);"><span style="font-family: courier new,monospace; color: rgb(0, 0, 0);"> yield x</span><br style="color: rgb(0, 0, 0);">
<span style="font-family: courier new,monospace; color: rgb(0, 0, 0);">RETVAL = None</span><br><br>2. Stage two expands the for-loop into an explicit while-loop that has the same meaning. It also sets RETVAL when breaking out of the loop. This prepares for the subsequent stages. Note that we have an explicit iter(EXPR) call here, since that is what a for-loop does:<br>
<br><span style="font-family: courier new,monospace; color: rgb(0, 153, 0);">it = iter(EXPR)</span><br style="font-family: courier new,monospace; color: rgb(0, 153, 0);"><span style="font-family: courier new,monospace; color: rgb(0, 153, 0);">while True:</span><br style="font-family: courier new,monospace; color: rgb(0, 153, 0);">
<span style="font-family: courier new,monospace; color: rgb(0, 153, 0);"> try:</span><br style="font-family: courier new,monospace; color: rgb(0, 153, 0);"><span style="font-family: courier new,monospace; color: rgb(0, 153, 0);"> x = next(it)</span><br style="font-family: courier new,monospace; color: rgb(0, 153, 0);">
<span style="font-family: courier new,monospace; color: rgb(0, 153, 0);"> except StopIteration:</span><br style="font-family: courier new,monospace; color: rgb(0, 153, 0);"><span style="font-family: courier new,monospace; color: rgb(0, 153, 0);"> RETVAL = None; break</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> yield x</span><br><br>3. Stage three further rearranges stage 2 without making semantic changes, Again this prepares for later stages:<br><br><span style="font-family: courier new,monospace;">it = iter(EXPR)</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace; color: rgb(0, 153, 0);">try:</span><br style="font-family: courier new,monospace; color: rgb(0, 153, 0);"><span style="font-family: courier new,monospace; color: rgb(0, 153, 0);"> x = next(it)</span><br style="font-family: courier new,monospace; color: rgb(0, 153, 0);">
<span style="font-family: courier new,monospace; color: rgb(0, 153, 0);">except StopIteration:</span><br style="font-family: courier new,monospace; color: rgb(0, 153, 0);"><span style="font-family: courier new,monospace; color: rgb(0, 153, 0);"> RETVAL = e.value</span><br style="font-family: courier new,monospace; color: rgb(0, 153, 0);">
<span style="font-family: courier new,monospace; color: rgb(0, 153, 0);">else:</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> while True:</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> yield x</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> try:</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> x = next(x)</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> except StopIteration:</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> RETVAL = None; break</span><br><br>4. Stage four adds handling for ReturnFromGenerator, in both places where next() is called:<br><br><span style="font-family: courier new,monospace;">it = iter(EXPR)</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace; color: rgb(0, 0, 0);">try:</span><br style="font-family: courier new,monospace; color: rgb(0, 0, 0);">
<span style="font-family: courier new,monospace; color: rgb(0, 0, 0);"> x = next(it)</span><br style="font-family: courier new,monospace; color: rgb(0, 0, 0);">
<span style="font-family: courier new,monospace; color: rgb(0, 0, 0);">except StopIteration:</span><br style="font-family: courier new,monospace; color: rgb(0, 0, 0);">
<span style="font-family: courier new,monospace; color: rgb(0, 0, 0);"> RETVAL = e.value</span><br style="color: rgb(0, 0, 0);"><span style="font-family: courier new,monospace; color: rgb(0, 153, 0);">except ReturnFromGenerator as e:</span><br style="font-family: courier new,monospace; color: rgb(0, 153, 0);">
<span style="font-family: courier new,monospace; color: rgb(0, 153, 0);"> RETVAL = e.value; break</span><br style="font-family: courier new,monospace; color: rgb(0, 0, 0);">
<span style="font-family: courier new,monospace; color: rgb(0, 0, 0);">else:</span><br style="font-family: courier new,monospace; color: rgb(0, 0, 0);">
<span style="font-family: courier new,monospace; color: rgb(0, 0, 0);"> while True:</span><br style="font-family: courier new,monospace; color: rgb(0, 0, 0);">
<span style="font-family: courier new,monospace; color: rgb(0, 0, 0);"> yield x</span><br style="font-family: courier new,monospace; color: rgb(0, 0, 0);">
<span style="font-family: courier new,monospace; color: rgb(0, 0, 0);"> try:</span><br style="font-family: courier new,monospace; color: rgb(0, 0, 0);">
<span style="font-family: courier new,monospace; color: rgb(0, 0, 0);"> x = next(it)</span><br style="font-family: courier new,monospace; color: rgb(0, 0, 0);">
<span style="font-family: courier new,monospace; color: rgb(0, 0, 0);"> except StopIteration:</span><br style="font-family: courier new,monospace; color: rgb(0, 0, 0);">
<span style="font-family: courier new,monospace; color: rgb(0, 0, 0);"> RETVAL = None; break</span><br style="font-family: courier new,monospace; color: rgb(0, 0, 0);"><span style="font-family: courier new,monospace; color: rgb(0, 153, 0);"> except ReturnFromGenerator as e:</span><br style="font-family: courier new,monospace; color: rgb(0, 153, 0);">
<span style="font-family: courier new,monospace; color: rgb(0, 153, 0);"> RETVAL = e.value; break</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> yield x</span><br>
<br>5. Stage five shows what should happen if "yield x" above returns a value: it is passed into the subgenerator using send(). I am ignoring for now what happens if it is not a generator; this will be cleared up later. Note that the initial next() call does not change into a send() call, because there is no value to send before before we have yielded:<br>
<br><span style="font-family: courier new,monospace;">it = iter(EXPR)</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace; color: rgb(0, 0, 0);">try:</span><br style="font-family: courier new,monospace; color: rgb(0, 0, 0);"><span style="font-family: courier new,monospace; color: rgb(0, 0, 0);"> x = next(it)</span><br style="font-family: courier new,monospace; color: rgb(0, 0, 0);">
<span style="font-family: courier new,monospace; color: rgb(0, 0, 0);">except StopIteration:</span><br style="font-family: courier new,monospace; color: rgb(0, 0, 0);"><span style="font-family: courier new,monospace; color: rgb(0, 0, 0);"> RETVAL = None</span><br style="font-family: courier new,monospace; color: rgb(0, 0, 0);">
<span style="font-family: courier new,monospace; color: rgb(0, 0, 0);">except ReturnFromGenerator as e:</span><br style="font-family: courier new,monospace; color: rgb(0, 0, 0);"><span style="font-family: courier new,monospace; color: rgb(0, 0, 0);"> RETVAL = e.value</span><br style="font-family: courier new,monospace; color: rgb(0, 0, 0);">
<span style="font-family: courier new,monospace; color: rgb(0, 0, 0);">else:</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> while True:</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace; color: rgb(0, 153, 0);"> v = yield x</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> try:</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace; color: rgb(0, 153, 0);"> x = it.send(v)</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> except StopIteration:</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> RETVAL = None; break</span><br><span style="font-family: courier new,monospace; color: rgb(0, 0, 0);"> except ReturnFromGenerator as e:</span><br style="font-family: courier new,monospace; color: rgb(0, 0, 0);">
<span style="font-family: courier new,monospace; color: rgb(0, 0, 0);"> RETVAL = e.value; break</span><br><br>6. Stage six adds more refined semantics for when "yield x" raises an exception: it is thrown into the generator, except if it is GeneratorExit, in which case we close() the generator and re-raise it (in this case the loop cannot continue so we do not set RETVAL):<br>
<br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">it = iter(EXPR)</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace; color: rgb(0, 0, 0);">try:</span><br style="font-family: courier new,monospace; color: rgb(0, 0, 0);"><span style="font-family: courier new,monospace; color: rgb(0, 0, 0);"> x = next(it)</span><br style="font-family: courier new,monospace; color: rgb(0, 0, 0);">
<span style="font-family: courier new,monospace; color: rgb(0, 0, 0);">except StopIteration:</span><br style="font-family: courier new,monospace; color: rgb(0, 0, 0);"><span style="font-family: courier new,monospace; color: rgb(0, 0, 0);"> RETVAL = None</span><br style="font-family: courier new,monospace; color: rgb(0, 0, 0);">
<span style="font-family: courier new,monospace; color: rgb(0, 0, 0);">except ReturnFromGenerator as e:</span><br style="font-family: courier new,monospace; color: rgb(0, 0, 0);"><span style="font-family: courier new,monospace; color: rgb(0, 0, 0);"> RETVAL = e.value</span><br style="font-family: courier new,monospace; color: rgb(0, 0, 0);">
<span style="font-family: courier new,monospace; color: rgb(0, 0, 0);">else:</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> while True:</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace; color: rgb(0, 153, 0);"> try:</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace; color: rgb(0, 0, 0);"> v = yield x<br><span style="color: rgb(0, 153, 0);"> except GeneratorExit:</span><br style="color: rgb(0, 153, 0);"><span style="color: rgb(0, 153, 0);"> it.close()</span><br style="color: rgb(0, 153, 0);">
<span style="color: rgb(0, 153, 0);"> raise</span><br style="color: rgb(0, 153, 0);"><span style="color: rgb(0, 153, 0);"> except:</span><br style="color: rgb(0, 153, 0);"><span style="color: rgb(0, 153, 0);"> try:</span><br style="color: rgb(0, 153, 0);">
<span style="color: rgb(0, 153, 0);"> x = it.throw(*sys.exc_info())</span><br style="color: rgb(0, 153, 0);"><span style="color: rgb(0, 153, 0);"> except StopIteration:</span><br style="color: rgb(0, 153, 0);">
<span style="color: rgb(0, 153, 0);"> RETVAL = None; break</span><br style="color: rgb(0, 153, 0);"><span style="color: rgb(0, 153, 0);"> except ReturnFromGenerator as e:</span><br style="color: rgb(0, 153, 0);">
<span style="color: rgb(0, 153, 0);"> RETVAL = e.value; break</span><br style="color: rgb(0, 153, 0);"><span style="color: rgb(0, 153, 0);"> else:</span><br style="color: rgb(0, 0, 0);"> try:</span><br style="font-family: courier new,monospace; color: rgb(0, 0, 0);">
<span style="font-family: courier new,monospace; color: rgb(0, 0, 0);"> x = it.send(v)</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> except StopIteration:</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> RETVAL = None; break</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace; color: rgb(0, 0, 0);"> except ReturnFromGenerator as e:</span><br style="font-family: courier new,monospace; color: rgb(0, 0, 0);">
<span style="font-family: courier new,monospace; color: rgb(0, 0, 0);"> RETVAL = e.value; break</span><br><br>7. In stage 7 we finally ask ourselves what should happen if it is not a generator (but some other iterator). The best answer seems subtle: send() should degenerator to next(), and all exceptions should simply be re-raised. We can conceptually specify this by simply re-using the for-loop expansion:<br>
<br><span style="font-family: courier new,monospace;">it = iter(EXPR)</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace; color: rgb(0, 153, 0);">if <it is not a generator>:</span><br style="font-family: courier new,monospace; color: rgb(0, 153, 0);">
<span style="font-family: courier new,monospace; color: rgb(0, 153, 0);"> for x in it:</span><br style="font-family: courier new,monospace; color: rgb(0, 153, 0);"><span style="font-family: courier new,monospace; color: rgb(0, 153, 0);"> yield next(x)</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> RETVAL = None</span><br style="font-family: courier new,monospace; color: rgb(0, 153, 0);"><span style="font-family: courier new,monospace; color: rgb(0, 153, 0);">else:</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">
</span><span style="font-family: courier new,monospace; color: rgb(0, 0, 0);">try:</span><br style="font-family: courier new,monospace; color: rgb(0, 0, 0);"><span style="font-family: courier new,monospace; color: rgb(0, 0, 0);"> x = next(it)</span><br style="font-family: courier new,monospace; color: rgb(0, 0, 0);">
<span style="font-family: courier new,monospace;"> </span><span style="font-family: courier new,monospace; color: rgb(0, 0, 0);">except StopIteration:</span><br style="font-family: courier new,monospace; color: rgb(0, 0, 0);">
<span style="font-family: courier new,monospace; color: rgb(0, 0, 0);"> RETVAL = None</span><br style="font-family: courier new,monospace; color: rgb(0, 0, 0);"><span style="font-family: courier new,monospace;"> </span><span style="font-family: courier new,monospace; color: rgb(0, 0, 0);">except ReturnFromGenerator as e:</span><br style="font-family: courier new,monospace; color: rgb(0, 0, 0);">
<span style="font-family: courier new,monospace; color: rgb(0, 0, 0);"> RETVAL = e.value</span><br style="font-family: courier new,monospace; color: rgb(0, 0, 0);"><span style="font-family: courier new,monospace;"> </span><span style="font-family: courier new,monospace; color: rgb(0, 0, 0);">else:</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> while True:</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace; color: rgb(0, 0, 0);"> try:</span><br style="font-family: courier new,monospace; color: rgb(0, 0, 0);">
<span style="font-family: courier new,monospace; color: rgb(0, 0, 0);"> v = yield x<br> except GeneratorExit:<br> it.close()<br> raise<br> except:<br> try:<br> x = it.throw(*sys.exc_info())<br>
except StopIteration:<br>
RETVAL = None; break<br>
except ReturnFromGenerator as e:<br>
RETVAL = e.value; break<br> else:<br> try:</span><br style="font-family: courier new,monospace; color: rgb(0, 0, 0);">
<span style="font-family: courier new,monospace; color: rgb(0, 0, 0);"> x = it.send(v)</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> except StopIteration:</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> RETVAL = None; break</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace; color: rgb(0, 0, 0);"> except ReturnFromGenerator as e:</span><br style="font-family: courier new,monospace; color: rgb(0, 0, 0);">
<span style="font-family: courier new,monospace; color: rgb(0, 0, 0);"> RETVAL = e.value; break</span><br style="font-family: courier new,monospace;"><br>Note: I don't mean that we literally should have a separate code path for non-generators. But writing it this way adds the generator test to one place in the spec, which helps understanding why I am choosing these semantics. The entire code of stage 6 degenerates to stage 1 if we make the following substitutions:<br>
<br><span style="font-family: courier new,monospace;">it.send(v) -> next(v)</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">it.throw(sys.exc_info()) -> raise</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">it.close() -> pass</span><br><br>(Except for some edge cases if the incoming exception is StopIteration
or ReturnFromgenerator, so we'd have to do the test before entering the
try/except block around the throw() or send() call.)<br><br>We could do this based on the presence or absence of the send/throw/close attributes: this would be duck typing. Or we could use isinstance(it, types.GeneratorType). I'm not sure there are strong arguments for either interpretation. The type check might be a little faster. We could even check for an exact type, since GeneratorType is final. Perhaps the most important consideration is that if EXPR produces a file stream object (which has a close() method), it would not consistently be closed: it would be closed if the outer generator was closed before reaching the end, but not if the loop was allowed to run until the end of the file. So I'm leaning towards only making the generator-specific method calls if it is really a generator.<br>
<br>-- <br>--Guido van Rossum (home page: <a href="http://www.python.org/%7Eguido/" target="_blank">http://www.python.org/~guido/</a>)<br>
<br>