Revised**12 PEP on Yield-From
Draft 13 of the PEP. Adjusted the expansion so as not to suggest that a throw or close attribute with the value None should be treated as a missing method. -- Greg PEP: XXX Title: Syntax for Delegating to a Subgenerator Version: $Revision$ Last-Modified: $Date$ Author: Gregory Ewing <greg.ewing@canterbury.ac.nz> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 13-Feb-2009 Python-Version: 3.x Post-History: Abstract ======== A syntax is proposed for a generator to delegate part of its operations to another generator. This allows a section of code containing 'yield' to be factored out and placed in another generator. Additionally, the subgenerator is allowed to return with a value, and the value is made available to the delegating generator. The new syntax also opens up some opportunities for optimisation when one generator re-yields values produced by another. Motivation ========== A Python generator is a form of coroutine, but has the limitation that it can only yield to its immediate caller. This means that a piece of code containing a ``yield`` cannot be factored out and put into a separate function in the same way as other code. Performing such a factoring causes the called function to itself become a generator, and it is necessary to explicitly iterate over this second generator and re-yield any values that it produces. If yielding of values is the only concern, this can be performed without much difficulty using a loop such as :: for v in g: yield v However, if the subgenerator is to interact properly with the caller in the case of calls to ``send()``, ``throw()`` and ``close()``, things become considerably more difficult. As will be seen later, the necessary code is very complicated, and it is tricky to handle all the corner cases correctly. A new syntax will be proposed to address this issue. In the simplest use cases, it will be equivalent to the above for-loop, but it will also handle the full range of generator behaviour, and allow generator code to be refactored in a simple and straightforward way. Proposal ======== The following new expression syntax will be allowed in the body of a generator: :: yield from <expr> where <expr> is an expression evaluating to an iterable, from which an iterator is extracted. The iterator is run to exhaustion, during which time it yields and receives values directly to or from the caller of the generator containing the ``yield from`` expression (the "delegating generator"). Furthermore, when the iterator is another generator, the subgenerator is allowed to execute a ``return`` statement with a value, and that value becomes the value of the ``yield from`` expression. The full semantics of the ``yield from`` expression can be described in terms of the generator protocol as follows: * Any values that the iterator yields are passed directly to the caller. * Any values sent to the delegating generator using ``send()`` are passed directly to the iterator. If the sent value is None, the iterator's ``next()`` method is called. If the sent value is not None, the iterator's ``send()`` method is called. If the call raises StopIteration, the delegating generator is resumed. Any other exception is propagated to the delegating generator. * Exceptions other than GeneratorExit thrown into the delegating generator are passed to the ``throw()`` method of the iterator. If the call raises StopIteration, the delegating generator is resumed. Any other exception is propagated to the delegating generator. * If a GeneratorExit exception is thrown into the delegating generator, or the ``close()`` method of the delegating generator is called, then the ``close()`` method of the iterator is called if it has one. If this call results in an exception, it is propagated to the delegating generator. Otherwise, GeneratorExit is raised in the delegating generator. * The value of the ``yield from`` expression is the first argument to the ``StopIteration`` exception raised by the iterator when it terminates. * ``return expr`` in a generator causes ``StopIteration(expr)`` to be raised. Enhancements to StopIteration ----------------------------- For convenience, the ``StopIteration`` exception will be given a ``value`` attribute that holds its first argument, or None if there are no arguments. Formal Semantics ---------------- Python 3 syntax is used in this section. 1. The statement :: RESULT = yield from EXPR is semantically equivalent to :: _i = iter(EXPR) try: _y = next(_i) except StopIteration as _e: _r = _e.value else: while 1: try: _s = yield _y except GeneratorExit as _e: try: _m = getattr(_i, 'close') except AttributeError: pass else: _m() raise _e except BaseException as _e: try: _m = getattr(_i, 'throw') except AttributeError: raise _e else: try: _y = _m(*sys.exc_info()) except StopIteration as _e: _r = _e.value break else: try: if _s is None: _y = next(_i) else: _y = _i.send(_s) except StopIteration as _e: _r = _e.value break RESULT = _r 2. In a generator, the statement :: return value is semantically equivalent to :: raise StopIteration(value) except that, as currently, the exception cannot be caught by ``except`` clauses within the returning generator. 3. The StopIteration exception behaves as though defined thusly: :: class StopIteration(Exception): def __init__(self, *args): if len(args) > 0: self.value = args[0] else: self.value = None Exception.__init__(self, *args) Rationale ========= The Refactoring Principle ------------------------- The rationale behind most of the semantics presented above stems from the desire to be able to refactor generator code. It should be possible to take an section of code containing one or more ``yield`` expressions, move it into a separate function (using the usual techniques to deal with references to variables in the surrounding scope, etc.), and call the new function using a ``yield from`` expression. The behaviour of the resulting compound generator should be, as far as reasonably practicable, the same as the original unfactored generator in all situations, including calls to ``next()``, ``send()``, ``throw()`` and ``close()``. The semantics in cases of subiterators other than generators has been chosen as a reasonable generalization of the generator case. The proposed semantics have the following limitations with regard to refactoring: * A block of code that catches GeneratorExit without subsequently re-raising it cannot be factored out while retaining exactly the same behaviour. * Factored code may not behave the same way as unfactored code if a StopIteration exception is thrown into the delegating generator. With use cases for these being rare to non-existent, it was not considered worth the extra complexity required to support them. Finalization ------------ There was some debate as to whether explicitly finalizing the delegating generator by calling its ``close()`` method while it is suspended at a ``yield from`` should also finalize the subiterator. An argument against doing so is that it would result in premature finalization of the subiterator if references to it exist elsewhere. Consideration of non-refcounting Python implementations led to the decision that this explicit finalization should be performed, so that explicitly closing a factored generator has the same effect as doing so to an unfactored one in all Python implementations. The assumption made is that, in the majority of use cases, the subiterator will not be shared. The rare case of a shared subiterator can be accommodated by means of a wrapper that blocks ``throw()`` and ``close()`` calls, or by using a means other than ``yield from`` to call the subiterator. Generators as Threads --------------------- A motivation for generators being able to return values concerns the use of generators to implement lightweight threads. When using generators in that way, it is reasonable to want to spread the computation performed by the lightweight thread over many functions. One would like to be able to call a subgenerator as though it were an ordinary function, passing it parameters and receiving a returned value. Using the proposed syntax, a statement such as :: y = f(x) where f is an ordinary function, can be transformed into a delegation call :: y = yield from g(x) where g is a generator. One can reason about the behaviour of the resulting code by thinking of g as an ordinary function that can be suspended using a ``yield`` statement. When using generators as threads in this way, typically one is not interested in the values being passed in or out of the yields. However, there are use cases for this as well, where the thread is seen as a producer or consumer of items. The ``yield from`` expression allows the logic of the thread to be spread over as many functions as desired, with the production or consumption of items occuring in any subfunction, and the items are automatically routed to or from their ultimate source or destination. Concerning ``throw()`` and ``close()``, it is reasonable to expect that if an exception is thrown into the thread from outside, it should first be raised in the innermost generator where the thread is suspended, and propagate outwards from there; and that if the thread is terminated from outside by calling ``close()``, the chain of active generators should be finalised from the innermost outwards. Syntax ------ The particular syntax proposed has been chosen as suggestive of its meaning, while not introducing any new keywords and clearly standing out as being different from a plain ``yield``. Optimisations ------------- Using a specialised syntax opens up possibilities for optimisation when there is a long chain of generators. Such chains can arise, for instance, when recursively traversing a tree structure. The overhead of passing ``next()`` calls and yielded values down and up the chain can cause what ought to be an O(n) operation to become, in the worst case, O(n\*\*2). A possible strategy is to add a slot to generator objects to hold a generator being delegated to. When a ``next()`` or ``send()`` call is made on the generator, this slot is checked first, and if it is nonempty, the generator that it references is resumed instead. If it raises StopIteration, the slot is cleared and the main generator is resumed. This would reduce the delegation overhead to a chain of C function calls involving no Python code execution. A possible enhancement would be to traverse the whole chain of generators in a loop and directly resume the one at the end, although the handling of StopIteration is more complicated then. Use of StopIteration to return values ------------------------------------- There are a variety of ways that the return value from the generator could be passed back. Some alternatives include storing it as an attribute of the generator-iterator object, or returning it as the value of the ``close()`` call to the subgenerator. However, the proposed mechanism is attractive for a couple of reasons: * Using a generalization of the StopIteration exception makes it easy for other kinds of iterators to participate in the protocol without having to grow an extra attribute or a close() method. * It simplifies the implementation, because the point at which the return value from the subgenerator becomes available is the same point at which the exception is raised. Delaying until any later time would require storing the return value somewhere. Rejected Ideas -------------- Some ideas were discussed but rejected. Suggestion: There should be some way to prevent the initial call to next(), or substitute it with a send() call with a specified value, the intention being to support the use of generators wrapped so that the initial next() is performed automatically. Resolution: Outside the scope of the proposal. Such generators should not be used with ``yield from``. Suggestion: If closing a subiterator raises StopIteration with a value, return that value from the ``close()`` call to the delegating generator. Resolution: Undesirable for a number of reasons. The purpose of closing a generator is to ensure proper cleanup, not to obtain a meaningful return value. Also, it would be unreliable unless the return value were stored so as to be available to subsequent close calls, which would cause it to persist for longer than expected. Suggestion: If ``close()`` is not to return a value, then raise an exception if StopIteration with a non-None value occurs. Resolution: No clear reason to do so. Ignoring a return value is not considered an error anywhere else in Python. Criticisms ========== Under this proposal, the value of a ``yield from`` expression would be derived in a very different way from that of an ordinary ``yield`` expression. This suggests that some other syntax not containing the word ``yield`` might be more appropriate, but no acceptable alternative has so far been proposed. Rejected alternatives include ``call``, ``delegate`` and ``gcall``. It has been suggested that some mechanism other than ``return`` in the subgenerator should be used to establish the value returned by the ``yield from`` expression. However, this would interfere with the goal of being able to think of the subgenerator as a suspendable function, since it would not be able to return values in the same way as other functions. The use of an exception to pass the return value has been criticised as an "abuse of exceptions", without any concrete justification of this claim. In any case, this is only one suggested implementation; another mechanism could be used without losing any essential features of the proposal. Alternative Proposals ===================== Proposals along similar lines have been made before, some using the syntax ``yield *`` instead of ``yield from``. While ``yield *`` is more concise, it could be argued that it looks too similar to an ordinary ``yield`` and the difference might be overlooked when reading code. To the author's knowledge, previous proposals have focused only on yielding values, and thereby suffered from the criticism that the two-line for-loop they replace is not sufficiently tiresome to write to justify a new syntax. By dealing with the full generator protocol, this proposal provides considerably more benefit. Additional Material =================== Some examples of the use of the proposed syntax are available, and also a prototype implementation based on the first optimisation outlined above. `Examples and Implementation`_ .. _Examples and Implementation: http://www.cosc.canterbury.ac.nz/greg.ewing/python/yield-from/ Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:
Greg Ewing wrote:
* ``return expr`` in a generator causes ``StopIteration(expr)`` to be raised.
One minor nit here - this bullet point is somewhat ambiguous as to where the raised exception is visible. It is probably worth mentioning explicitly that as with existing "return" statements in generators, the StopIteration exception won't be seen in the generator's own frame. As someone else mentioned on another recent draft, it may be worth including some of the toy examples we were playing with in some of the discussion threads. Sure, they weren't all that practical, but they seemed to do a good job of getting the data flow concepts across. Otherwise looks good to me. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------
Nick Coghlan wrote:
It is mentioned later, in the formal semantics,
but I agree that it probably wouldn't hurt to mention it earlier as well.
It may be good enough just to include links to the relevant messages in the PEP. Getting some small examples in the actual docs is probably more important.
Otherwise looks good to me.
There is a minor issue with the handling of throw in the expansion, but otherwise I agree. Cheers - Jacob
Greg Ewing wrote:
A few more details about the expansion. First a minor nit. There is no need to use getattr if the name is constant and you are going to catch AttributeError anyway. Just use _i.close and _i.throw. Next a more serious issue. The current use of *sys.exc_info() in the throw handling is actually wrong. Getting the "throw" attribute may run arbitrary python code which could easily replace the exception. We should probably add _x = sys.exc_info() as the first line in the "except BaseException as _e" block, and change _y = _m(*sys.exc_info()) to: _y = _m(*_x) Alternatively, we could get the signature of throw() fixed so that it matches the "raise" statement again. Then we can drop the use of sys.exc_info(), and just use: _y = _m(_e) But that is probably out of scope for this PEP. Other than that, I think the expansion matches what we have agreed on. FWIW I still consider an expansion using functools.partial to be more readable because it centralizes the StopIteration handling and reduces the code nesting. Here is an updated version of such an expansion that is semantically equivalent to the PEP rev 13 + my suggested fix for the sys.exc_info() issue: _i = iter(EXPR) _p = functools.partial(next, _i) while 1: try: _y = _p() except StopIteration as _e: _r = _e.value break try: _s = yield _y except GeneratorExit as _e: try: _m = _i.close except AttributeError: pass else: _m() raise _e except BaseException as _e: _x = sys.exc_info() try: _m = _i.throw except AttributeError: raise _e _p = functools.partial(_m, *_x) else: if _s is None: _p = functools.partial(next, _i) else: _p = functools.partial(_i.send, _s) RESULT = _r If you don't like it, fine. Since it is only a presentation detail, I won't push for it. I *would* like to hear your reasoning, but will accept whatever conclusion you come to, I think :) Anyway, it looks to me like we are almost done. What are the chances of getting this into 3.1 and 2.7? Hopefully-soon-using-yield-from-for-real-ly yours - Jacob
Jacob Holm wrote:
Yep, good idea.
Good catch (and I agree with your first suggested fix of grabbing the exception details before retrieving the method - the with statement expansion in PEP 343 had to do something similar for similar reasons).
Yep (using sys.exc_info() also translates more cleanly back to Python 2.x)
Nobody would ever implement it that way though - using partial like that in the formal semantic definition implies a whole heap of temporary objects that just won't exist in practice. Better to use the more verbose expansion that is much closer in spirit to the way it would actually be implemented. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------
Nick Coghlan wrote:
The temporary objects don't bother me. That is really a deep implementation detail. As for being closer in spirit, the real implementation is already going to be very different so I don't see that as a problem either. FWIW, it is quite easy to write same style expansion without using functools.partial, like this: _i = iter(EXPR) _m, _a = next, (_i,) while 1: try: _y = _m(*_a) except StopIteration as _e: _r = _e.value break try: _s = yield _y except GeneratorExit as _e: try: _m = _i.close except AttributeError: pass else: _m() raise _e except BaseException as _e: _a = sys.exc_info() try: _m = _i.throw except AttributeError: raise _e else: if _s is None: _m, _a = next, (_i,) else: _m, _a = _i.send, (_s,) RESULT = _r This is even a line shorter than the version using functools.partial, and the temporary _a tuples used actually match what happens in a normal function call anyway (I think)... Anyway, as I said it is not all that important. It is a presentation detail, and as such very subjective. I can agree to disagree about what version is clearer. Cheers - Jacob
Jacob Holm wrote:
Now that we passed the magic three or four threshold, is it not easier to read if we pick some better names? _iter = iter(EXPR) _call, _arg = next, _iter while 1: try: _out = _call(_arg) except StopIteration as _except: _result = _except.value break try: _in = yield _out except GeneratorExit as _except: try: _close = _iter.close except AttributeError: pass else: _close() raise _except except BaseException as _except: _a = sys.exc_info() try: _call = _iter.throw except AttributeError: raise _except else: if _in is None: _call, _arg = next, _iter else: _call, _arg = _iter.send, _in RESULT = _result --Scott David Daniels Scott.Daniels@Acm.Org
Scott David Daniels wrote:
^^^^^ should be (_iter,)
^^^^ should be *_arg
^^ should be _arg, forcing the other changes.
^^^^^ should be (_iter,)
else: _call, _arg = _iter.send, _in
^^^ should be (_in,)
RESULT = _result
As noted inline, you missed one of the _a's and so falsely assumed there would always only be one argument to _call. In the case of throw there is 3. I don't really care what the variables are called. That bikeshed discussion is not one I want to participate in. Your names are as good as any I guess. Anyway, this is moot unless Greg agrees that this style of expansion is a good idea in the first place. Not-bikeshedding-ly yours - Jacob
Jacob Holm wrote:
Quite right. I can't have been thinking very straight when I did that!
I'm not sure if this is really necessary, since function calls save/restore the exception being handled, if I understand correctly. But I suppose it can't hurt to clarify this.
FWIW I still consider an expansion using functools.partial to be more readable
That's a matter of opinion -- I find it harder to follow because it separates the logic for deciding which method to call from the place where it's called. Also I would rather express the expansion in terms of core language features as far as possible rather than relying on something imported from a library.
Anyway, it looks to me like we are almost done. What are the chances of getting this into 3.1 and 2.7?
You'd have to ask Guido. -- Greg
Greg Ewing wrote:
For some reason I thought that the save/restore only applied to what would be raised by a bare raise and not to sys.exc_info(). I have just tested it with a small script and it appears I was wrong. Sorry for the noise.
It may be slightly harder to follow, but it is easier to see that StopIteration is treated the same for the three operations, and it is a bit shorter and less deeply nested. Anyway, that is only my opinion and in this case it is yours that count.
As shown in my response to Nick, the use of functools.partial is not actually needed for this style of expansion. It is just as easy to collect the function and arguments in different variables and use _m(*_a) in the call. I'll shut up about it now. If you still don't like it that's fine. Cheers - Jacob
Greg, I am still busy understanding what your PEP means to the framework that I have been building. I believe that, for practical usage, there is still something missing, or at least, not clear. Suppose we would like to write a HTTP protocol stack using generators, and we have something like (actual working code, read 'yield from' for every 'yield'): try: reqArgs = yield readRe(REGEXP.REQUEST, MAXREQUESTSIZE) except OverflowError: yield requestEntityTooLarge() yield HTTP.CRLF return headers = parseHeaders(reqArgs) yield processBody(headers) The function 'readRe' will read and buffer a part of the request as defined by the regular expression REGEXP.REQUEST, and it will raise OverflowError when it keeps reading while never making a match. The point is that readRe accepts chunks of data that are not aligned to protocol boundaries. This is a typical boundary clash as Jackson calls it (I tend to think of this stuff as JSP pipelines) and JSP describes how to solve it. But to be able to solve it, the readRe generator must be able to indicate that it has superfluous data, and this data must be processed by other generators. In the case of the example, 'readRe' might have been reading parts of the body (assuming a POST request). After I created 'compose' I started implementing practical stuff like this, and it soon turned out that 'compose' must support boundary clashes or all but toy problems would still be unsolvable with generators. Therefor 'compose' now has a feature to return a value together with remaining data: raise StopIteration(retval, remainingChunk1, remainingChunk2, ...) This will push retval to the delegating generator as a return value for yield, and then feed the remainingChunk's to whatever generators come next. In your PEP, this would be a return statement of course. Have you though about this? How would you solve it? Best regards, Erik E.J. Groeneveld Seek You Too
Dear Greg, 2009/4/19 Greg Ewing <greg.ewing@canterbury.ac.nz>:
Draft 13 of the PEP.
I have implemented this in Weightless, and changed my implementation as to work with BaseException instead of Exception. This works well. However, I was not able to make an exception for GeneratorExit, see next point.
I tried to implement this, but I failed. The reason is that a generator's close() checks for an exception being raised by its generator. When no exception has been raised, it will raise the RuntimeError('generator ignored GeneratorExit'). And when an exception has been raised (regardless what type), it will exit just normally. So the phrase
If this call results in an exception, it is propagated to the delegating generator.
applies only to the RuntimeError close() might throw. And the ramaining phrase
Otherwise, GeneratorExit is raised in the delegating generator.
makes it behave no different than for all other types of exceptions. Was this intended? If Yes, I suggest to make the text clearer and more specific about it. If No, then what is the correct expansion? Best regards Erik Groeneveld
Hi Erik Erik Groeneveld wrote:
In other words, the ValueError in the following example is swallowed by close():
This looks like close() doesn't actually behave as it should according to PEP342. Interesting... I would call that a bug.
The assumption was that close() would behave as described in PEP342, which would make the description in the yield-from PEP380 correct. I can't believe that none of us actually tested that... Cheers - Jacob
Hi Jacob, 2009/4/22 Jacob Holm <jh@improva.dk>:
I have the same code here, it does not raise an exception indeed.
This looks like close() doesn't actually behave as it should according to PEP342. Interesting...
From PEP342 (Thanks Jacob, for the reference):
4. Add a close() method for generator-iterators, which raises GeneratorExit at the point where the generator was paused. If the generator then raises StopIteration (by exiting normally, or due to already being closed) or GeneratorExit (by not catching the exception), close() returns to its caller. If the generator yields a value, a RuntimeError is raised. If the generator raises any other exception, it is propagated to the caller. close() does nothing if the generator has already exited due to an exception or normal exit.
I would call that a bug.
I agree with that.
I can't believe that none of us actually tested that...
Someone did! Erik
On Wed, Apr 22, 2009 at 06:05:11PM +0200, Erik Groeneveld wrote:
I don't understand why you're both counting this as a bug. It looks like exactly the behavior specified in PEP 342. When g.close() is evaluated, a GeneratorExit is thrown to the suspended 'yield' expression in foo. That exception is is not caught, so g terminates without executing the rest of its code. The 'raise ValueError' line is never executed. -- Jim Pryor jim@jimpryor.net
Greg, Jacob, Jim,
Thanks for signaling this. The example was wrong. I was confused by my code implicitly turning a GeneratorExit into StopIteration. In my unittests I have: def f2(): try: yield except GeneratorExit: pass # implicit raise StopIteration here and I added a duplicate to test the problem more explicitly: def f3(): try: yield except GeneratorExit: pass yield # does not raise an exception but yields None Thanks a lot, I was now able to complete all unittests and code according to the new PEP. There is one problem left however. The code dealing with GeneratorExit has to undo the work of close() a bit. To account for: "If this call results in an exception, it is propagated to the delegating generator. Otherwise, GeneratorExit is raised in the delegating generator." and knowing that close() raises a RuntimeError when a generator ignores GeneratorExit, I have: try: generator.close() except RuntimeError: pass raise GeneratorExit But this code cannot tell if the generator intended to raise a RuntimeError. Indeed, I can't make this test work with RuntimeError (see commented lines): msg = [] def f8(): try: yield f9() #except RuntimeError, e: except ValueError, e: msg.append(str(e)) raise StopIteration() def f9(): try: yield except GeneratorExit: msg.append('GeneratorExit turned into ValueError') #raise RuntimeError('stop here') raise ValueError('stop here') yield g8 = compose(f8()) g8.next() try: g8.throw(GeneratorExit()) self.fail('must raise StopIteration') except StopIteration: pass self.assertEquals(['GeneratorExit turned into ValueError', 'stop here'], msg) I wonder what you think about this and how to get this right. Erik
2009/4/23 Jacob Holm <jh@improva.dk>:
These two line were exactly what I started out with, but I see that expanded it because I had a different interpretation of the PEP. I interpreted it as when the generator did not raise an exception, but close does, it is a different situation. I'll think about it more deeply. Thanks btw! Erik
HTH -Jacob
Erik Groeneveld wrote:
Keep in mind that as far as the PEP is concerned, it's not necessarily dealing with a generator, just some object that happens to implement certain methods. It has no idea what's going on inside the close() method -- it can only go by the end result of the call. -- Greg
Erik Groeneveld wrote:
Eh? That's not what happens according to the following experiment: def g(): try: yield except GeneratorExit: raise ValueError("Blarg!") gi = g() gi.next() gi.close() which produces Traceback (most recent call last): File "g.py", line 9, in <module> gi.close() File "g.py", line 5, in g raise ValueError("Blarg!") ValueError: Blarg! -- Greg
I created a pure Python decorator-based implementation of PEP 380, based on the semantics in this latest draft (Revised**12). There's a "simple" version and an "optimized" version: the simple version is easier to follow; the optimized version does special handling for nested yield from calls. They're both now posted at ActiveState: http://code.activestate.com/recipes/576727/ http://code.activestate.com/recipes/576728/
From the descriptions:
Hope others find them a useful contribution. Of course, I welcome any feedback. -- Jim Pryor profjim@jimpryor.net
Greg Ewing wrote:
* ``return expr`` in a generator causes ``StopIteration(expr)`` to be raised.
One minor nit here - this bullet point is somewhat ambiguous as to where the raised exception is visible. It is probably worth mentioning explicitly that as with existing "return" statements in generators, the StopIteration exception won't be seen in the generator's own frame. As someone else mentioned on another recent draft, it may be worth including some of the toy examples we were playing with in some of the discussion threads. Sure, they weren't all that practical, but they seemed to do a good job of getting the data flow concepts across. Otherwise looks good to me. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------
Nick Coghlan wrote:
It is mentioned later, in the formal semantics,
but I agree that it probably wouldn't hurt to mention it earlier as well.
It may be good enough just to include links to the relevant messages in the PEP. Getting some small examples in the actual docs is probably more important.
Otherwise looks good to me.
There is a minor issue with the handling of throw in the expansion, but otherwise I agree. Cheers - Jacob
Greg Ewing wrote:
A few more details about the expansion. First a minor nit. There is no need to use getattr if the name is constant and you are going to catch AttributeError anyway. Just use _i.close and _i.throw. Next a more serious issue. The current use of *sys.exc_info() in the throw handling is actually wrong. Getting the "throw" attribute may run arbitrary python code which could easily replace the exception. We should probably add _x = sys.exc_info() as the first line in the "except BaseException as _e" block, and change _y = _m(*sys.exc_info()) to: _y = _m(*_x) Alternatively, we could get the signature of throw() fixed so that it matches the "raise" statement again. Then we can drop the use of sys.exc_info(), and just use: _y = _m(_e) But that is probably out of scope for this PEP. Other than that, I think the expansion matches what we have agreed on. FWIW I still consider an expansion using functools.partial to be more readable because it centralizes the StopIteration handling and reduces the code nesting. Here is an updated version of such an expansion that is semantically equivalent to the PEP rev 13 + my suggested fix for the sys.exc_info() issue: _i = iter(EXPR) _p = functools.partial(next, _i) while 1: try: _y = _p() except StopIteration as _e: _r = _e.value break try: _s = yield _y except GeneratorExit as _e: try: _m = _i.close except AttributeError: pass else: _m() raise _e except BaseException as _e: _x = sys.exc_info() try: _m = _i.throw except AttributeError: raise _e _p = functools.partial(_m, *_x) else: if _s is None: _p = functools.partial(next, _i) else: _p = functools.partial(_i.send, _s) RESULT = _r If you don't like it, fine. Since it is only a presentation detail, I won't push for it. I *would* like to hear your reasoning, but will accept whatever conclusion you come to, I think :) Anyway, it looks to me like we are almost done. What are the chances of getting this into 3.1 and 2.7? Hopefully-soon-using-yield-from-for-real-ly yours - Jacob
Jacob Holm wrote:
Yep, good idea.
Good catch (and I agree with your first suggested fix of grabbing the exception details before retrieving the method - the with statement expansion in PEP 343 had to do something similar for similar reasons).
Yep (using sys.exc_info() also translates more cleanly back to Python 2.x)
Nobody would ever implement it that way though - using partial like that in the formal semantic definition implies a whole heap of temporary objects that just won't exist in practice. Better to use the more verbose expansion that is much closer in spirit to the way it would actually be implemented. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------
Nick Coghlan wrote:
The temporary objects don't bother me. That is really a deep implementation detail. As for being closer in spirit, the real implementation is already going to be very different so I don't see that as a problem either. FWIW, it is quite easy to write same style expansion without using functools.partial, like this: _i = iter(EXPR) _m, _a = next, (_i,) while 1: try: _y = _m(*_a) except StopIteration as _e: _r = _e.value break try: _s = yield _y except GeneratorExit as _e: try: _m = _i.close except AttributeError: pass else: _m() raise _e except BaseException as _e: _a = sys.exc_info() try: _m = _i.throw except AttributeError: raise _e else: if _s is None: _m, _a = next, (_i,) else: _m, _a = _i.send, (_s,) RESULT = _r This is even a line shorter than the version using functools.partial, and the temporary _a tuples used actually match what happens in a normal function call anyway (I think)... Anyway, as I said it is not all that important. It is a presentation detail, and as such very subjective. I can agree to disagree about what version is clearer. Cheers - Jacob
Jacob Holm wrote:
Now that we passed the magic three or four threshold, is it not easier to read if we pick some better names? _iter = iter(EXPR) _call, _arg = next, _iter while 1: try: _out = _call(_arg) except StopIteration as _except: _result = _except.value break try: _in = yield _out except GeneratorExit as _except: try: _close = _iter.close except AttributeError: pass else: _close() raise _except except BaseException as _except: _a = sys.exc_info() try: _call = _iter.throw except AttributeError: raise _except else: if _in is None: _call, _arg = next, _iter else: _call, _arg = _iter.send, _in RESULT = _result --Scott David Daniels Scott.Daniels@Acm.Org
Scott David Daniels wrote:
^^^^^ should be (_iter,)
^^^^ should be *_arg
^^ should be _arg, forcing the other changes.
^^^^^ should be (_iter,)
else: _call, _arg = _iter.send, _in
^^^ should be (_in,)
RESULT = _result
As noted inline, you missed one of the _a's and so falsely assumed there would always only be one argument to _call. In the case of throw there is 3. I don't really care what the variables are called. That bikeshed discussion is not one I want to participate in. Your names are as good as any I guess. Anyway, this is moot unless Greg agrees that this style of expansion is a good idea in the first place. Not-bikeshedding-ly yours - Jacob
Jacob Holm wrote:
Quite right. I can't have been thinking very straight when I did that!
I'm not sure if this is really necessary, since function calls save/restore the exception being handled, if I understand correctly. But I suppose it can't hurt to clarify this.
FWIW I still consider an expansion using functools.partial to be more readable
That's a matter of opinion -- I find it harder to follow because it separates the logic for deciding which method to call from the place where it's called. Also I would rather express the expansion in terms of core language features as far as possible rather than relying on something imported from a library.
Anyway, it looks to me like we are almost done. What are the chances of getting this into 3.1 and 2.7?
You'd have to ask Guido. -- Greg
Greg Ewing wrote:
For some reason I thought that the save/restore only applied to what would be raised by a bare raise and not to sys.exc_info(). I have just tested it with a small script and it appears I was wrong. Sorry for the noise.
It may be slightly harder to follow, but it is easier to see that StopIteration is treated the same for the three operations, and it is a bit shorter and less deeply nested. Anyway, that is only my opinion and in this case it is yours that count.
As shown in my response to Nick, the use of functools.partial is not actually needed for this style of expansion. It is just as easy to collect the function and arguments in different variables and use _m(*_a) in the call. I'll shut up about it now. If you still don't like it that's fine. Cheers - Jacob
Greg, I am still busy understanding what your PEP means to the framework that I have been building. I believe that, for practical usage, there is still something missing, or at least, not clear. Suppose we would like to write a HTTP protocol stack using generators, and we have something like (actual working code, read 'yield from' for every 'yield'): try: reqArgs = yield readRe(REGEXP.REQUEST, MAXREQUESTSIZE) except OverflowError: yield requestEntityTooLarge() yield HTTP.CRLF return headers = parseHeaders(reqArgs) yield processBody(headers) The function 'readRe' will read and buffer a part of the request as defined by the regular expression REGEXP.REQUEST, and it will raise OverflowError when it keeps reading while never making a match. The point is that readRe accepts chunks of data that are not aligned to protocol boundaries. This is a typical boundary clash as Jackson calls it (I tend to think of this stuff as JSP pipelines) and JSP describes how to solve it. But to be able to solve it, the readRe generator must be able to indicate that it has superfluous data, and this data must be processed by other generators. In the case of the example, 'readRe' might have been reading parts of the body (assuming a POST request). After I created 'compose' I started implementing practical stuff like this, and it soon turned out that 'compose' must support boundary clashes or all but toy problems would still be unsolvable with generators. Therefor 'compose' now has a feature to return a value together with remaining data: raise StopIteration(retval, remainingChunk1, remainingChunk2, ...) This will push retval to the delegating generator as a return value for yield, and then feed the remainingChunk's to whatever generators come next. In your PEP, this would be a return statement of course. Have you though about this? How would you solve it? Best regards, Erik E.J. Groeneveld Seek You Too
Dear Greg, 2009/4/19 Greg Ewing <greg.ewing@canterbury.ac.nz>:
Draft 13 of the PEP.
I have implemented this in Weightless, and changed my implementation as to work with BaseException instead of Exception. This works well. However, I was not able to make an exception for GeneratorExit, see next point.
I tried to implement this, but I failed. The reason is that a generator's close() checks for an exception being raised by its generator. When no exception has been raised, it will raise the RuntimeError('generator ignored GeneratorExit'). And when an exception has been raised (regardless what type), it will exit just normally. So the phrase
If this call results in an exception, it is propagated to the delegating generator.
applies only to the RuntimeError close() might throw. And the ramaining phrase
Otherwise, GeneratorExit is raised in the delegating generator.
makes it behave no different than for all other types of exceptions. Was this intended? If Yes, I suggest to make the text clearer and more specific about it. If No, then what is the correct expansion? Best regards Erik Groeneveld
Hi Erik Erik Groeneveld wrote:
In other words, the ValueError in the following example is swallowed by close():
This looks like close() doesn't actually behave as it should according to PEP342. Interesting... I would call that a bug.
The assumption was that close() would behave as described in PEP342, which would make the description in the yield-from PEP380 correct. I can't believe that none of us actually tested that... Cheers - Jacob
Hi Jacob, 2009/4/22 Jacob Holm <jh@improva.dk>:
I have the same code here, it does not raise an exception indeed.
This looks like close() doesn't actually behave as it should according to PEP342. Interesting...
From PEP342 (Thanks Jacob, for the reference):
4. Add a close() method for generator-iterators, which raises GeneratorExit at the point where the generator was paused. If the generator then raises StopIteration (by exiting normally, or due to already being closed) or GeneratorExit (by not catching the exception), close() returns to its caller. If the generator yields a value, a RuntimeError is raised. If the generator raises any other exception, it is propagated to the caller. close() does nothing if the generator has already exited due to an exception or normal exit.
I would call that a bug.
I agree with that.
I can't believe that none of us actually tested that...
Someone did! Erik
On Wed, Apr 22, 2009 at 06:05:11PM +0200, Erik Groeneveld wrote:
I don't understand why you're both counting this as a bug. It looks like exactly the behavior specified in PEP 342. When g.close() is evaluated, a GeneratorExit is thrown to the suspended 'yield' expression in foo. That exception is is not caught, so g terminates without executing the rest of its code. The 'raise ValueError' line is never executed. -- Jim Pryor jim@jimpryor.net
Greg, Jacob, Jim,
Thanks for signaling this. The example was wrong. I was confused by my code implicitly turning a GeneratorExit into StopIteration. In my unittests I have: def f2(): try: yield except GeneratorExit: pass # implicit raise StopIteration here and I added a duplicate to test the problem more explicitly: def f3(): try: yield except GeneratorExit: pass yield # does not raise an exception but yields None Thanks a lot, I was now able to complete all unittests and code according to the new PEP. There is one problem left however. The code dealing with GeneratorExit has to undo the work of close() a bit. To account for: "If this call results in an exception, it is propagated to the delegating generator. Otherwise, GeneratorExit is raised in the delegating generator." and knowing that close() raises a RuntimeError when a generator ignores GeneratorExit, I have: try: generator.close() except RuntimeError: pass raise GeneratorExit But this code cannot tell if the generator intended to raise a RuntimeError. Indeed, I can't make this test work with RuntimeError (see commented lines): msg = [] def f8(): try: yield f9() #except RuntimeError, e: except ValueError, e: msg.append(str(e)) raise StopIteration() def f9(): try: yield except GeneratorExit: msg.append('GeneratorExit turned into ValueError') #raise RuntimeError('stop here') raise ValueError('stop here') yield g8 = compose(f8()) g8.next() try: g8.throw(GeneratorExit()) self.fail('must raise StopIteration') except StopIteration: pass self.assertEquals(['GeneratorExit turned into ValueError', 'stop here'], msg) I wonder what you think about this and how to get this right. Erik
2009/4/23 Jacob Holm <jh@improva.dk>:
These two line were exactly what I started out with, but I see that expanded it because I had a different interpretation of the PEP. I interpreted it as when the generator did not raise an exception, but close does, it is a different situation. I'll think about it more deeply. Thanks btw! Erik
HTH -Jacob
Erik Groeneveld wrote:
Keep in mind that as far as the PEP is concerned, it's not necessarily dealing with a generator, just some object that happens to implement certain methods. It has no idea what's going on inside the close() method -- it can only go by the end result of the call. -- Greg
Erik Groeneveld wrote:
Eh? That's not what happens according to the following experiment: def g(): try: yield except GeneratorExit: raise ValueError("Blarg!") gi = g() gi.next() gi.close() which produces Traceback (most recent call last): File "g.py", line 9, in <module> gi.close() File "g.py", line 5, in g raise ValueError("Blarg!") ValueError: Blarg! -- Greg
I created a pure Python decorator-based implementation of PEP 380, based on the semantics in this latest draft (Revised**12). There's a "simple" version and an "optimized" version: the simple version is easier to follow; the optimized version does special handling for nested yield from calls. They're both now posted at ActiveState: http://code.activestate.com/recipes/576727/ http://code.activestate.com/recipes/576728/
From the descriptions:
Hope others find them a useful contribution. Of course, I welcome any feedback. -- Jim Pryor profjim@jimpryor.net
participants (7)
-
Erik Groeneveld -
Greg Ewing -
Jacob Holm -
Jim Pryor -
Mathias Panzenböck -
Nick Coghlan -
Scott David Daniels