Revised**10 PEP on Yield-From
Draft 11 of the PEP. Changes in this version: - GeneratorExit always calls close() and is always reraised. - Special handling of thrown-in StopIterations removed, since Guido doesn't think you should be doing that in the first place. - Expansion uses next(_i) instead of _i.next() and doesn't mention cacheing of methods. -- Greg PEP: XXX Title: Syntax for Delegating to a Subgenerator Version: $Revision$ Last-Modified: $Date$ Author: Gregory Ewing <greg.ewing@canterbury.ac.nz> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 13-Feb-2009 Python-Version: 3.x Post-History: Abstract ======== A syntax is proposed for a generator to delegate part of its operations to another generator. This allows a section of code containing 'yield' to be factored out and placed in another generator. Additionally, the subgenerator is allowed to return with a value, and the value is made available to the delegating generator. The new syntax also opens up some opportunities for optimisation when one generator re-yields values produced by another. Motivation ========== A Python generator is a form of coroutine, but has the limitation that it can only yield to its immediate caller. This means that a piece of code containing a ``yield`` cannot be factored out and put into a separate function in the same way as other code. Performing such a factoring causes the called function to itself become a generator, and it is necessary to explicitly iterate over this second generator and re-yield any values that it produces. If yielding of values is the only concern, this can be performed without much difficulty using a loop such as :: for v in g: yield v However, if the subgenerator is to interact properly with the caller in the case of calls to ``send()``, ``throw()`` and ``close()``, things become considerably more difficult. As will be seen later, the necessary code is very complicated, and it is tricky to handle all the corner cases correctly. A new syntax will be proposed to address this issue. In the simplest use cases, it will be equivalent to the above for-loop, but it will also handle the full range of generator behaviour, and allow generator code to be refactored in a simple and straightforward way. Proposal ======== The following new expression syntax will be allowed in the body of a generator: :: yield from <expr> where <expr> is an expression evaluating to an iterable, from which an iterator is extracted. The iterator is run to exhaustion, during which time it yields and receives values directly to or from the caller of the generator containing the ``yield from`` expression (the "delegating generator"). Furthermore, when the iterator is another generator, the subgenerator is allowed to execute a ``return`` statement with a value, and that value becomes the value of the ``yield from`` expression. The full semantics of the ``yield from`` expression can be described in terms of the generator protocol as follows: * Any values that the iterator yields are passed directly to the caller. * Any values sent to the delegating generator using ``send()`` are passed directly to the iterator. If the sent value is None, the iterator's ``next()`` method is called. If the sent value is not None, the iterator's ``send()`` method is called. Any exception resulting from attempting to call ``next`` or ``send`` is raised in the delegating generator. * Exceptions other than GeneratorExit passed to the ``throw()`` method of the delegating generator are forwarded to the ``throw()`` method of the iterator. Any exception resulting from attempting to call ``throw()`` are propagated to the delegating generator. * If a GeneratorExit exception is thrown into the delegating generator, the ``close()`` method of the iterator is called if it has one. If this call results in an exception, it is propagated to the delegating generator. Otherwise, the GeneratorExit is reraised in the delegating generator. The implicit GeneratorExit resulting from closing the delegating generator is treated as though it were passed in using ``throw()``. * The value of the ``yield from`` expression is the first argument to the ``StopIteration`` exception raised by the iterator when it terminates. * ``return expr`` in a generator causes ``StopIteration(expr)`` to be raised. Enhancements to StopIteration ----------------------------- For convenience, the ``StopIteration`` exception will be given a ``value`` attribute that holds its first argument, or None if there are no arguments. Formal Semantics ---------------- Python 3 syntax is used in this section. 1. The statement :: RESULT = yield from EXPR is semantically equivalent to :: _i = iter(EXPR) try: _y = next(_i) except StopIteration as _e: _r = _e.value else: while 1: try: _s = yield _y except GeneratorExit: _m = getattr(_i, 'close', None) if _m is not None: _m() raise except: _m = getattr(_i, 'throw', None) if _m is not None: _y = _m(*sys.exc_info()) else: raise else: try: if _s is None: _y = next(_i) else: _y = _i.send(_s) except StopIteration as _e: _r = _e.value break RESULT = _r 2. In a generator, the statement :: return value is semantically equivalent to :: raise StopIteration(value) except that, as currently, the exception cannot be caught by ``except`` clauses within the returning generator. 3. The StopIteration exception behaves as though defined thusly: :: class StopIteration(Exception): def __init__(self, *args): if len(args) > 0: self.value = args[0] else: self.value = None Exception.__init__(self, *args) Rationale ========= The Refactoring Principle ------------------------- The rationale behind most of the semantics presented above stems from the desire to be able to refactor generator code. It should be possible to take an section of code containing one or more ``yield`` expressions, move it into a separate function (using the usual techniques to deal with references to variables in the surrounding scope, etc.), and call the new function using a ``yield from`` expression. The behaviour of the resulting compound generator should be, as far as possible, exactly the same as the original unfactored generator in all situations, including calls to ``next()``, ``send()``, ``throw()`` and ``close()``. The semantics in cases of subiterators other than generators has been chosen as a reasonable generalization of the generator case. Finalization ------------ There was some debate as to whether explicitly finalizing the delegating generator by calling its ``close()`` method while it is suspended at a ``yield from`` should also finalize the subiterator. An argument against doing so is that it would result in premature finalization of the subiterator if references to it exist elsewhere. Consideration of non-refcounting Python implementations led to the decision that this explicit finalization should be performed, so that explicitly closing a factored generator has the same effect as doing so to an unfactored one in all Python implementations. The assumption made is that, in the majority of use cases, the subiterator will not be shared. The rare case of a shared subiterator can be accommodated by means of a wrapper that blocks ``throw()`` and ``close()`` calls, or by using a means other than ``yield from`` to call the subiterator. Generators as Threads --------------------- A motivation for generators being able to return values concerns the use of generators to implement lightweight threads. When using generators in that way, it is reasonable to want to spread the computation performed by the lightweight thread over many functions. One would like to be able to call a subgenerator as though it were an ordinary function, passing it parameters and receiving a returned value. Using the proposed syntax, a statement such as :: y = f(x) where f is an ordinary function, can be transformed into a delegation call :: y = yield from g(x) where g is a generator. One can reason about the behaviour of the resulting code by thinking of g as an ordinary function that can be suspended using a ``yield`` statement. When using generators as threads in this way, typically one is not interested in the values being passed in or out of the yields. However, there are use cases for this as well, where the thread is seen as a producer or consumer of items. The ``yield from`` expression allows the logic of the thread to be spread over as many functions as desired, with the production or consumption of items occuring in any subfunction, and the items are automatically routed to or from their ultimate source or destination. Concerning ``throw()`` and ``close()``, it is reasonable to expect that if an exception is thrown into the thread from outside, it should first be raised in the innermost generator where the thread is suspended, and propagate outwards from there; and that if the thread is terminated from outside by calling ``close()``, the chain of active generators should be finalised from the innermost outwards. Syntax ------ The particular syntax proposed has been chosen as suggestive of its meaning, while not introducing any new keywords and clearly standing out as being different from a plain ``yield``. Optimisations ------------- Using a specialised syntax opens up possibilities for optimisation when there is a long chain of generators. Such chains can arise, for instance, when recursively traversing a tree structure. The overhead of passing ``next()`` calls and yielded values down and up the chain can cause what ought to be an O(n) operation to become, in the worst case, O(n\*\*2). A possible strategy is to add a slot to generator objects to hold a generator being delegated to. When a ``next()`` or ``send()`` call is made on the generator, this slot is checked first, and if it is nonempty, the generator that it references is resumed instead. If it raises StopIteration, the slot is cleared and the main generator is resumed. This would reduce the delegation overhead to a chain of C function calls involving no Python code execution. A possible enhancement would be to traverse the whole chain of generators in a loop and directly resume the one at the end, although the handling of StopIteration is more complicated then. Use of StopIteration to return values ------------------------------------- There are a variety of ways that the return value from the generator could be passed back. Some alternatives include storing it as an attribute of the generator-iterator object, or returning it as the value of the ``close()`` call to the subgenerator. However, the proposed mechanism is attractive for a couple of reasons: * Using a generalization of the StopIteration exception makes it easy for other kinds of iterators to participate in the protocol without having to grow an extra attribute or a close() method. * It simplifies the implementation, because the point at which the return value from the subgenerator becomes available is the same point at which the exception is raised. Delaying until any later time would require storing the return value somewhere. Originally it was proposed to simply extend StopIteration to accept a value. However, it was felt desirable by some to have a mechanism for detecting the erroneous use of a value-returning generator in a context that is not aware of generator return values. Using an exception that is a superclass of StopIteration means that code knowing about generator return values only has one exception to catch, and code that does not know about them will fail to catch the new exception. Criticisms ========== Under this proposal, the value of a ``yield from`` expression would be derived in a very different way from that of an ordinary ``yield`` expression. This suggests that some other syntax not containing the word ``yield`` might be more appropriate, but no acceptable alternative has so far been proposed. Rejected alternatives include ``call``, ``delegate`` and ``gcall``. It has been suggested that some mechanism other than ``return`` in the subgenerator should be used to establish the value returned by the ``yield from`` expression. However, this would interfere with the goal of being able to think of the subgenerator as a suspendable function, since it would not be able to return values in the same way as other functions. The use of an exception to pass the return value has been criticised as an "abuse of exceptions", without any concrete justification of this claim. In any case, this is only one suggested implementation; another mechanism could be used without losing any essential features of the proposal. Alternative Proposals ===================== Proposals along similar lines have been made before, some using the syntax ``yield *`` instead of ``yield from``. While ``yield *`` is more concise, it could be argued that it looks too similar to an ordinary ``yield`` and the difference might be overlooked when reading code. To the author's knowledge, previous proposals have focused only on yielding values, and thereby suffered from the criticism that the two-line for-loop they replace is not sufficiently tiresome to write to justify a new syntax. By dealing with the full generator protocol, this proposal provides considerably more benefit. Additional Material =================== Some examples of the use of the proposed syntax are available, and also a prototype implementation based on the first optimisation outlined above. `Examples and Implementation`_ .. _Examples and Implementation: http://www.cosc.canterbury.ac.nz/greg.ewing/python/yield-from/ Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:
Hi Greg A few comments on the latest PEP 380 version (rev11d)... 1) IIRC, the use of sys.exc_info() is not needed in 3.x as all exceptions have a __traceback__ attribute. 2) The expansion is not handling StopIterations raised as a result of calling _i.throw(). They should be treated just like any other StopIteration that ends the yield-from. A simpler expansion based on 1) and 2) but otherwise identical is: _i = iter(EXPR) try: _y = next(_i) while 1: try: _s = yield _y except GeneratorExit: _m = getattr(_i, 'close', None) if _m is not None: _m() raise except BaseException as _e: _m = getattr(_i, 'throw', None) if _m is not None: _y = _m(_e) else: raise else: if _s is None: _y = next(_i) else: _y = _i.send(_s) except StopIteration as _e: _r = _e.value RESULT = _r 3) If the subiterator has a close() but doesn't have throw() it won't be closed when throw() is called on the outer generator. This is fine with me, I am just not sure if it is intentional. 4) If the subiterator has a close() but doesn't have send() it won't be closed when a send() on the outer generator causes an AttributeError in the expansion. Again this is fine with me, I am just not sure if it is intentional. 5) The last paragraph in the "Use of StopIteration to return values" section, seems to be a leftover from an earlier draft of the PEP that used a different exception. 6) Several of the issues we have been discussing on python-ideas are not mentioned at all: * The "initial next()" issue should at least be described and listed as out of scope. * The "what should close() do if it catches StopIteration with a value" issue I don't think we have resolved either way. Since we are not going to store the value, only the first close() would be able to return it. Under those conditions, I no longer think that returning the value is a good idea. If we are not storing or returning the value, I think close() should raise an exception. Either reraise the StopIteration, so that the caller has a chance to get the value that way, or raise a RuntimeError, because it is meaningless to return a value as response to a GeneratorExit when that value cannot later be accessed by anything and it is therefore most likely a bug. * The special-casing of StopIteration should probably be mentioned as a rejected idea. Not special-casing it does break the refactoring principle, and I think it important to mention that in some way. * There may be other issues I have forgotten at the moment. 7) By not mentioning caching, you are effectively saying the methods won't be cached. I have exactly one use for this. The fastest pure-python "full" workaround I can find for the "initial next()" issue is a wrapper using Nicks self-modifying class hack. With this the delegation cost is less than 1/3 of any other approach I have tried. (But still 13 times higher than a yield-from without the wrapper when using your patch). All that means is that adding caching later would be likely to break some code that relied on the exact semantics as described in the PEP. Other than that, everything looks fine. Best regards - Jacob
Jacob Holm wrote:
1) IIRC, the use of sys.exc_info() is not needed in 3.x as all exceptions have a __traceback__ attribute.
The 3.0 docs still list the signature of throw() as having 3 args, though, so I'll leave it that way for now.
2) The expansion is not handling StopIterations raised as a result of calling _i.throw().
Yes, I was a bit too aggressive in ripping out StopIteration handling there. :-)
A simpler expansion based on 1) and 2) but otherwise identical is:
I had a try block around everything in an earlier version, but I changed it because it was prone to catching too much. I think I'll stick with separate targeted try blocks, because it's easier to make sure I'm catching only what I intend to catch.
I'm not worried about those much. The important thing is for explicit finalization to work as expected.
Yes, I need to remove that.
6) Several of the issues we have been discussing on python-ideas are not mentioned at all:
I'll add some discussion about these.
7) By not mentioning caching, you are effectively saying the methods won't be cached.
That's what I mean to say. Guido has pointed out that Python doesn't generally do cacheing if it would change the behaviour of the code, which means we can't do cacheing here. There's still a fast path available if the subiterator is a generator, which is the case I'm mostly concerned about, so I'm not as worried as I was about not being able to cache send(). -- Greg
Jacob Holm wrote:
If you care about a generator's return value *don't* just call "close()" on it. Use "throw(GeneratorExit)" instead so you can catch the exception and interrogate the return value yourself. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------
Nick Coghlan wrote:
I already know I can do that. My point was that if close is defined to not return the value, then returning a value in response to a GeneratorExit is suspect and likely to be a bug in the generator. I have since realised that there can be valid cases where being able to share the exit code path between the normal and GeneratorExit cases simplifies things. Also, Greg has convinced me that the type of error I was worried about is not specific to generators, so there is no particular reason to do the extra work of detecting them here. So I now agree we can swallow the return value without raising an exception. Cheers - Jacob
Jacob Holm wrote:
So I now agree we can swallow the return value without raising an exception.
Yeah, I made the mistake of replying before I finished catching up on the post-holiday email collection - I eventually got to those later threads and saw you had already changed your mind. Hopefully Greg can now get hold of Guido and Benjamin in time to get the PEP over the line for 3.1b1 :) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------
Greg, Please forgive me for hooking into this discussion so late. Below are my late comments to your original PEP, and below those some new stuff. I have been writing weightless/compose which does exactly what your PEP is trying to accomplish. I'll check my stuff against this PEP. I really appreciate your initiative! It helps me a lot. 2009/4/15 Greg Ewing <greg.ewing@canterbury.ac.nz>:
These are the exact problems I can't solve neatly in weightless/compose:
this allows a programmer to express the intention of just returning a generator or wanting to delegate the work to a 'subgenerator'. Weightless/compose now just descends into every generator, while this is certainly not always wanted. Great I think, I like the syntax.
In Weightless/compose, after several different tries I settled for mimicking returning a value by using raise StopIteration(returnvalue). As return in a generator raises StopIteration(), I think it is very natural to use return like this in a generator (if fact I wished it would be possible sometimes, not being aware of python-ideas). So I like it too.
Clear.
Clear. I have implemented this by just calling send(...) either with None or with a value. The VM dispatches that to next() when the value is None, I assume.
I let any Exception propagate using the throw() method. I believe this will not correctly handle GeneratorExit as outlined in the discussion before. I'll have to change this I think.
I have a hard time understanding what this would mean in a pure python implementation. I added both bullets to my unittests to work it out later.
The implicit GeneratorExit resulting from closing the delegating generator is treated as though it were passed in using ``throw()``.
By "closing the delegating generator" you mean "from the outside, call close() on it"? It then will raise the GeneratorExit exception, and I understand it. I added a unittest as well.
I assume that 'return 1 2 3' will have one return value being a tuple (1,2,3) which is one argument to StopIteration(), and which is unpacked when 'yield from' returns?
I am using StopIteration's 'args' atrribute? But after reading the motivation below, it could indeed confuse other generators, and a separate StopIteration would be better, I think.
I'll take this one with me, as I really need some time to compare it to my own code. I'll come back to it later.
Clear.
I probably miss the point, could you explain why this is needed?
Yes! Exactly. I just call this supporting 'program decomposition'. For clearity, you could probably add the name of the refactoring, it is called 'extract method' isn't it?
I agree completely. I went through some lenght to get proper clean-up, and I solved it similarly.
Yes, I believe you make sure that: try: x = yield from y() except SomeError: return 'HELP' actually does catch the SomeError exception when raised in y(), or one it its descendants?
Next section I skipped, I you don't mind.
I agree. And I begin to understand the need for that value attribute. Ok.
[...]
For now I have one more fundamental question left over. Will the the delegating generator remain on the call-stack or not? The current behaviour is that a stack-frame is created for a generator, which is not disposed when next()/send() returns, but kept somewere. When a new call to next()/send() happens, the same stack-frame is put back on the call-stack. This is crucial because it acts as the o-so-valuable closure. I came across this problem because I was writing code that traverses the call-stack in order to find some place to put 'generator-local' variables (like thread-local). My implementation in weightless/compose does not physically keep the generators on the call-stack (I don't know how to do that in Python), but keep it's own stack. I would have to extend 'compose' to not only search on the real call-stack but also traverse it own semi-call-stack if/when it finds an instance of itself on the real call-stack. I am writing code like: def x(): somevar = 10 yield from y() then it I write in y: def y(): frame = currentframe().f_back while 'somevar' not in frame.f_locals: frame = frame.f_back return frame.f_locals['somevar'] would this find the variable in x? Best regards, Erik
Erik Groeneveld wrote:
By "closing the delegating generator" you mean "from the outside, call close() on it"?
Yes, that's right.
I assume that 'return 1 2 3' will have one return value being a tuple (1,2,3)
Um, you can't write 'return 1 2 3'. You can write 'return 1, 2, 3' which returns a tuple.
The 'value' attribute is purely a convenience -- you can get it from args[0] just as well.
For clearity, you could probably add the name of the refactoring, it is called 'extract method' isn't it?
Quite likely it's called that in some book or other, I wouldn't really know. I don't think of refactorings as having names, I just do them.
That's the idea.
Originally it was proposed to simply extend StopIteration to accept a value...
That whole paragraph shouldn't be there, it's left over from an earlier version. Guido originally wanted to use a different exception for returning with a value, but he changed his mind.
Will the the delegating generator remain on the call-stack or not?
It certainly shows up in tracebacks, but I'm not actually sure whether it will be seen by sys._getframe() while a subiterator is running in my current implementation. I'll try an experiment to find out. Even if it does, it's probably not something you should rely on, since a different implementation that optimizes more aggressively could behave differently. In the case of thread-local storage for generator based threads, I don't think I would try to attach them to a generator frame, because a thread isn't just a single generator, it's a collection of generators. Rather I'd use the scheduler to manage them. The scheduler knows when it's switching from one thread to another and can switch in the appropriate set of variables. -- Greg
Hi Greg A few comments on the latest PEP 380 version (rev11d)... 1) IIRC, the use of sys.exc_info() is not needed in 3.x as all exceptions have a __traceback__ attribute. 2) The expansion is not handling StopIterations raised as a result of calling _i.throw(). They should be treated just like any other StopIteration that ends the yield-from. A simpler expansion based on 1) and 2) but otherwise identical is: _i = iter(EXPR) try: _y = next(_i) while 1: try: _s = yield _y except GeneratorExit: _m = getattr(_i, 'close', None) if _m is not None: _m() raise except BaseException as _e: _m = getattr(_i, 'throw', None) if _m is not None: _y = _m(_e) else: raise else: if _s is None: _y = next(_i) else: _y = _i.send(_s) except StopIteration as _e: _r = _e.value RESULT = _r 3) If the subiterator has a close() but doesn't have throw() it won't be closed when throw() is called on the outer generator. This is fine with me, I am just not sure if it is intentional. 4) If the subiterator has a close() but doesn't have send() it won't be closed when a send() on the outer generator causes an AttributeError in the expansion. Again this is fine with me, I am just not sure if it is intentional. 5) The last paragraph in the "Use of StopIteration to return values" section, seems to be a leftover from an earlier draft of the PEP that used a different exception. 6) Several of the issues we have been discussing on python-ideas are not mentioned at all: * The "initial next()" issue should at least be described and listed as out of scope. * The "what should close() do if it catches StopIteration with a value" issue I don't think we have resolved either way. Since we are not going to store the value, only the first close() would be able to return it. Under those conditions, I no longer think that returning the value is a good idea. If we are not storing or returning the value, I think close() should raise an exception. Either reraise the StopIteration, so that the caller has a chance to get the value that way, or raise a RuntimeError, because it is meaningless to return a value as response to a GeneratorExit when that value cannot later be accessed by anything and it is therefore most likely a bug. * The special-casing of StopIteration should probably be mentioned as a rejected idea. Not special-casing it does break the refactoring principle, and I think it important to mention that in some way. * There may be other issues I have forgotten at the moment. 7) By not mentioning caching, you are effectively saying the methods won't be cached. I have exactly one use for this. The fastest pure-python "full" workaround I can find for the "initial next()" issue is a wrapper using Nicks self-modifying class hack. With this the delegation cost is less than 1/3 of any other approach I have tried. (But still 13 times higher than a yield-from without the wrapper when using your patch). All that means is that adding caching later would be likely to break some code that relied on the exact semantics as described in the PEP. Other than that, everything looks fine. Best regards - Jacob
Jacob Holm wrote:
1) IIRC, the use of sys.exc_info() is not needed in 3.x as all exceptions have a __traceback__ attribute.
The 3.0 docs still list the signature of throw() as having 3 args, though, so I'll leave it that way for now.
2) The expansion is not handling StopIterations raised as a result of calling _i.throw().
Yes, I was a bit too aggressive in ripping out StopIteration handling there. :-)
A simpler expansion based on 1) and 2) but otherwise identical is:
I had a try block around everything in an earlier version, but I changed it because it was prone to catching too much. I think I'll stick with separate targeted try blocks, because it's easier to make sure I'm catching only what I intend to catch.
I'm not worried about those much. The important thing is for explicit finalization to work as expected.
Yes, I need to remove that.
6) Several of the issues we have been discussing on python-ideas are not mentioned at all:
I'll add some discussion about these.
7) By not mentioning caching, you are effectively saying the methods won't be cached.
That's what I mean to say. Guido has pointed out that Python doesn't generally do cacheing if it would change the behaviour of the code, which means we can't do cacheing here. There's still a fast path available if the subiterator is a generator, which is the case I'm mostly concerned about, so I'm not as worried as I was about not being able to cache send(). -- Greg
Jacob Holm wrote:
If you care about a generator's return value *don't* just call "close()" on it. Use "throw(GeneratorExit)" instead so you can catch the exception and interrogate the return value yourself. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------
Nick Coghlan wrote:
I already know I can do that. My point was that if close is defined to not return the value, then returning a value in response to a GeneratorExit is suspect and likely to be a bug in the generator. I have since realised that there can be valid cases where being able to share the exit code path between the normal and GeneratorExit cases simplifies things. Also, Greg has convinced me that the type of error I was worried about is not specific to generators, so there is no particular reason to do the extra work of detecting them here. So I now agree we can swallow the return value without raising an exception. Cheers - Jacob
Jacob Holm wrote:
So I now agree we can swallow the return value without raising an exception.
Yeah, I made the mistake of replying before I finished catching up on the post-holiday email collection - I eventually got to those later threads and saw you had already changed your mind. Hopefully Greg can now get hold of Guido and Benjamin in time to get the PEP over the line for 3.1b1 :) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------
Greg, Please forgive me for hooking into this discussion so late. Below are my late comments to your original PEP, and below those some new stuff. I have been writing weightless/compose which does exactly what your PEP is trying to accomplish. I'll check my stuff against this PEP. I really appreciate your initiative! It helps me a lot. 2009/4/15 Greg Ewing <greg.ewing@canterbury.ac.nz>:
These are the exact problems I can't solve neatly in weightless/compose:
this allows a programmer to express the intention of just returning a generator or wanting to delegate the work to a 'subgenerator'. Weightless/compose now just descends into every generator, while this is certainly not always wanted. Great I think, I like the syntax.
In Weightless/compose, after several different tries I settled for mimicking returning a value by using raise StopIteration(returnvalue). As return in a generator raises StopIteration(), I think it is very natural to use return like this in a generator (if fact I wished it would be possible sometimes, not being aware of python-ideas). So I like it too.
Clear.
Clear. I have implemented this by just calling send(...) either with None or with a value. The VM dispatches that to next() when the value is None, I assume.
I let any Exception propagate using the throw() method. I believe this will not correctly handle GeneratorExit as outlined in the discussion before. I'll have to change this I think.
I have a hard time understanding what this would mean in a pure python implementation. I added both bullets to my unittests to work it out later.
The implicit GeneratorExit resulting from closing the delegating generator is treated as though it were passed in using ``throw()``.
By "closing the delegating generator" you mean "from the outside, call close() on it"? It then will raise the GeneratorExit exception, and I understand it. I added a unittest as well.
I assume that 'return 1 2 3' will have one return value being a tuple (1,2,3) which is one argument to StopIteration(), and which is unpacked when 'yield from' returns?
I am using StopIteration's 'args' atrribute? But after reading the motivation below, it could indeed confuse other generators, and a separate StopIteration would be better, I think.
I'll take this one with me, as I really need some time to compare it to my own code. I'll come back to it later.
Clear.
I probably miss the point, could you explain why this is needed?
Yes! Exactly. I just call this supporting 'program decomposition'. For clearity, you could probably add the name of the refactoring, it is called 'extract method' isn't it?
I agree completely. I went through some lenght to get proper clean-up, and I solved it similarly.
Yes, I believe you make sure that: try: x = yield from y() except SomeError: return 'HELP' actually does catch the SomeError exception when raised in y(), or one it its descendants?
Next section I skipped, I you don't mind.
I agree. And I begin to understand the need for that value attribute. Ok.
[...]
For now I have one more fundamental question left over. Will the the delegating generator remain on the call-stack or not? The current behaviour is that a stack-frame is created for a generator, which is not disposed when next()/send() returns, but kept somewere. When a new call to next()/send() happens, the same stack-frame is put back on the call-stack. This is crucial because it acts as the o-so-valuable closure. I came across this problem because I was writing code that traverses the call-stack in order to find some place to put 'generator-local' variables (like thread-local). My implementation in weightless/compose does not physically keep the generators on the call-stack (I don't know how to do that in Python), but keep it's own stack. I would have to extend 'compose' to not only search on the real call-stack but also traverse it own semi-call-stack if/when it finds an instance of itself on the real call-stack. I am writing code like: def x(): somevar = 10 yield from y() then it I write in y: def y(): frame = currentframe().f_back while 'somevar' not in frame.f_locals: frame = frame.f_back return frame.f_locals['somevar'] would this find the variable in x? Best regards, Erik
Erik Groeneveld wrote:
By "closing the delegating generator" you mean "from the outside, call close() on it"?
Yes, that's right.
I assume that 'return 1 2 3' will have one return value being a tuple (1,2,3)
Um, you can't write 'return 1 2 3'. You can write 'return 1, 2, 3' which returns a tuple.
The 'value' attribute is purely a convenience -- you can get it from args[0] just as well.
For clearity, you could probably add the name of the refactoring, it is called 'extract method' isn't it?
Quite likely it's called that in some book or other, I wouldn't really know. I don't think of refactorings as having names, I just do them.
That's the idea.
Originally it was proposed to simply extend StopIteration to accept a value...
That whole paragraph shouldn't be there, it's left over from an earlier version. Guido originally wanted to use a different exception for returning with a value, but he changed his mind.
Will the the delegating generator remain on the call-stack or not?
It certainly shows up in tracebacks, but I'm not actually sure whether it will be seen by sys._getframe() while a subiterator is running in my current implementation. I'll try an experiment to find out. Even if it does, it's probably not something you should rely on, since a different implementation that optimizes more aggressively could behave differently. In the case of thread-local storage for generator based threads, I don't think I would try to attach them to a generator frame, because a thread isn't just a single generator, it's a collection of generators. Rather I'd use the scheduler to manage them. The scheduler knows when it's switching from one thread to another and can switch in the appropriate set of variables. -- Greg
participants (4)
-
Erik Groeneveld -
Greg Ewing -
Jacob Holm -
Nick Coghlan