x=(yield from) confusion [was:Yet another alternative name for yield-from]
Summary: I can understand a generator that returns something useful with each yield. I can understand a generator that is used only for cooperative multi-tasking, and returns nothing useful until the end. yield from *as an expression* only really makes sense if the generator is sending useful information *both* ways. I can understand that sort of generator only while reading the PEP; the code smell is strong enough that I forget it by the next day. Details: On 4/2/09, Nick Coghlan <ncoghlan@gmail.com> wrote:
Jim Jewett wrote:
If the "gencall" exhausts the generator f ... If the "return value" of the generator really is important ...
The intermediate values aren't necessarily discarded by "yield from" though: they're passed out to whoever is consuming the values yielded by the outermost generator.
I think this may be where I start to have trouble. When I see the statement form: def outer_g(): yield from inner_g() I expect inner_g to suspend execution, and it isn't that hard to remember that it might do so several times. I also expect the results of inner_g.next() to be passed on out to outer_g's own caller, thereby suspending outer_g. So far, so good. But when yield from is used as an expression, and I see: x = yield from g() I somehow expect only a single call to g.next(), whose value gets assigned to x, and not passed out. I did read the PEP, in several versions, and understood it at the time ... and still managed (several times) to forget and misinterpret by a day or two later. And yes, I realize it doesn't make too much sense to call g.next() only once -- so I kept looking for a loop around that yield from statement. When there wasn't one, I shrugged it off like a with clause -- and still didn't remember the actual PEP-intended meaning. The times I did remember that (even) the expression form looped, I was still boggled that it would return something other than None after it was exhausted. Greg's answer was that it was for threading, and the final return was the real value. This seems like a different category of generator, but I could get my head around it -- so long as I forgot that the yield itself was returning anything useful. -jJ
On Thu, Apr 2, 2009 at 6:43 PM, Jim Jewett <jimjjewett@gmail.com> wrote:
yield from *as an expression* only really makes sense if the generator is sending useful information *both* ways. I can understand that sort of generator only while reading the PEP; the code smell is strong enough that I forget it by the next day.
Read Dave Beazley's coroutines tutorial (dabeaz.com/couroutines) and check out the contortions in the scheduler to support subgenerators (Part 8). -- --Guido van Rossum (home page: http://www.python.org/~guido/)
On 4/2/09, Guido van Rossum <guido@python.org> wrote:
On Thu, Apr 2, 2009 at 6:43 PM, Jim Jewett <jimjjewett@gmail.com> wrote:
yield from *as an expression* only really makes sense if the generator is sending useful information *both* ways. I can understand that sort of generator only while reading the PEP; the code smell is strong enough that I forget it by the next day.
Read Dave Beazley's coroutines tutorial (dabeaz.com/couroutines) and check out the contortions in the scheduler to support subgenerators (Part 8).
I have. I still don't see it really helping with anything except maybe (in http://dabeaz.com/coroutines/pyos8.py) the Task.run method. Even there, I don't see an expression form helping very much. Passing through the intermediate yields could be nice, but a statement can do that. Grabbing a separate final value could change the "try ... except StopIteration" into an "x=yield from", but ... I'm not sure that could really work, because I don't see how all the code inside the try block goes away. (It might go away if you moved that logic from the task to the scheduler, but then the protocol seems to get even more complicated, as the scheduler has to distinguish between "I've used up this time slot", as well as "I have intermediate results, but might be called again", and "I'm done".) -jJ
Jim Jewett wrote:
yield from *as an expression* only really makes sense if the generator is sending useful information *both* ways.
No, that's not the only way it makes sense. In my multitasking example, none of the yields send or receive any values. But they're still needed, because they define the points at which the task can be suspended.
The times I did remember that (even) the expression form looped,
The yield-from expression itself doesn't loop. What it does do is yield multiple times, if the generator being called yields multiple times. But it has to be driven by whatever is calling the whole thing making a sufficient number of next() or send() calls, in a loop or otherwise. In hindsight, the wording in the PEP about the subiterator being "run to exhaustion" might be a bit misleading. I'l see if I can find a better way to phrase it. -- Greg
On 4/3/09, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
Jim Jewett wrote:
yield from *as an expression* only really makes sense if the generator is sending useful information *both* ways.
No, that's not the only way it makes sense. In my multitasking example, none of the yields send or receive any values.
err... I didn't mean both directions, I meant "from the callee to the caller as a yielded value" and "from the callee to the caller as a final return value that can't be yielded normally."
But they're still needed, because they define the points at which the task can be suspended.
If they don't send or receive values, then why do they need to be expressions instead of statements?
The times I did remember that (even) the expression form looped,
The yield-from expression itself doesn't loop. What it does do is yield multiple times,
That sounds to me like an implicit loop. yield from iter <==> for val in iter: yield val So the outside generator won't progress to its own next line (and subsequent yield) until it has finished looping over the inner generator. -jJ
Jim Jewett wrote:
err... I didn't mean both directions, I meant "from the callee to the caller as a yielded value" and "from the callee to the caller as a final return value that can't be yielded normally."
I think perhaps we're misunderstanding each other. You seemed to be saying that the only time you would want a generator to return a value is when you were also using it to either send or receive values using yield, and I was just pointing out that's not true. If that's not what you meant, you'll have to explain more clearly what you do mean. -- Greg
On Fri, Apr 3, 2009 at 9:15 PM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
Jim Jewett wrote:
err... I didn't mean both directions, I meant "from the callee to the caller as a yielded value" and "from the callee to the caller as a final return value that can't be yielded normally."
I think perhaps we're misunderstanding each other. You seemed to be saying that the only time you would want a generator to return a value is when you were also using it to either send or receive values using yield,
Almost exactly the opposite. I can see cases where you want the interim yields, and I can see cases where you want the final result -- but I'm asking how common it is to need *both*, and whether we should really be going to such effort for it. If you only need one or the other, I think the PEP can be greatly simplified. -jJ
If that's not what you meant, you'll have to explain more clearly what you do mean.
-- Greg
Jim Jewett wrote:
Almost exactly the opposite. I can see cases where you want the interim yields, and I can see cases where you want the final result -- but I'm asking how common it is to need *both*, and whether we should really be going to such effort for it.
If you only need one or the other, I think the PEP can be greatly simplified.
One of the things you may want to use coroutines with is a trampoline scheduler for handling asynchronous IO. In that case, the inner coroutine may want to yield an IO wait object so the scheduler can add the relevant descriptor to the main select() loop. In such a case, the interim yielded values would go back to the scheduler, while the final result would go back to the calling coroutine. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------
Jim Jewett wrote:
I can see cases where you want the interim yields, and I can see cases where you want the final result -- but I'm asking how common it is to need *both*, and whether we should really be going to such effort for it.
If you only need one or the other, I think the PEP can be greatly simplified.
How, exactly? I'd need convincing that you wouldn't just end up with the same amount of complexity arranged differently. -- Greg
Jim Jewett wrote:
The times I did remember that (even) the expression form looped, I was still boggled that it would return something other than None after it was exhausted. Greg's answer was that it was for threading, and the final return was the real value. This seems like a different category of generator, but I could get my head around it -- so long as I forgot that the yield itself was returning anything useful.
Greg tried to clarify this a bit already, but I think Jacob's averager example is an interesting case where it makes sense to both yield multiple times and also "return a value". While it is just a toy example, I believe it does a great job of illustrating the control flow expectations. (Writing this email has certainly clarified a lot of things about the PEP in my *own* mind). The following reworking of Jacob's example assumes a couple of things that differ from the current PEP: - the particular colour my bikeshed is painted when it comes to returning values from a generator is "return finally" (the idea being to emphasise that this represents a special "final" value for the generator that happens only after all of the normal yields are done). - rather than trying to change the meaning of GeneratorExit and close(), 3 new generator methods would be added: next_return(), send_return() and throw_return(). The new methods have the same signatures as their existing counterparts, but if the generator raises GeneratorReturn, they trap it and return the associated value instead. Like close(), they complain with a RuntimeError if the generator doesn't finish. For example: def throw_return(self, *exc_info): try: self.throw(*exc_info) raise RuntimeError("Generator did not terminate") except GeneratorReturn as gr: return gr.value (Note that I've also removed the 'yield raise' idea from the example - if next() or send() triggers termination of the generator with an exception other than StopIteration, then that exception is already propagated into the calling scope by the existing generator machinery. I realise Jacob was trying to make it possible to "yield an exception" without terminating the coroutine, but that idea is well beyond the scope of the PEP) You then get: class CalcAverage(Exception): pass def averager(start=0): # averager that maintains a running average # and returns the final average when done count = 0 exc = None sum = start while 1: avg = sum / count try: val = yield avg except CalcAverage: return finally avg sum += val count += 1 avg = averager() avg.next() # start coroutine avg.send(1.0) # yields 1.0 avg.send(2.0) # yields 1.5 print avg.throw_return(CalcAverage) # prints 1.5 Now, suppose I want to write another toy coroutine that calculates the averages of two sequences and then returns the difference: def average_diff(start=0): avg1 = yield from averager(start) avg2 = yield from averager(start) return finally avg2 - avg1 diff = average_diff() diff.next() # start coroutine # yields 0.0 avg.send(1.0) # yields 1.0 avg.send(2.0) # yields 1.5 diff.throw(CalcAverage) # Starts calculation of second average # yields 0.0 diff.send(2.0) # yields 2.0 diff.send(3.0) # yields 2.5 print diff.throw_return(CalcAverage) # Prints 1.0 (from "2.5 - 1.5") The same example could be rewritten to use "None" as a sentinel value instead of throwing in an exception (average_diff doesn't change, so I haven't rewritten that part): def averager(start=0): count = 0 exc = None sum = start while 1: avg = sum / count val = yield avg if val is None: return finally avg sum += val count += 1 # yielded values are same as the throw_return() approach diff = average_diff() diff.next() # start coroutine diff.send(1.0) diff.send(2.0) diff.send(None) # Starts calculation of second average diff.send(2.0) diff.send(3.0) print diff.send_return(None) # Prints 1.0 (from "2.5 - 1.5") Notice how the coroutines in this example can be thought of as simple state machines that the calling code needs to know how to drive. That state sequence is as much a part of the coroutine's signature as are the arguments to the constructor and the final return value. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------
Hi Nick, Your reworking of my "averager" example has highlighted another issue for me, which I will get to below. First a few comments on your message. Nick Coghlan wrote:
[snip]
The following reworking of Jacob's example assumes a couple of things that differ from the current PEP:
- the particular colour my bikeshed is painted when it comes to returning values from a generator is "return finally" (the idea being to emphasise that this represents a special "final" value for the generator that happens only after all of the normal yields are done).
We should probably drop that particular bikeshed discussion until we actually know the details of what the construct should do, esp in the context of close(). I am starting to lose track of all the different possible versions.
- rather than trying to change the meaning of GeneratorExit and close(), 3 new generator methods would be added: next_return(), send_return() and throw_return(). The new methods have the same signatures as their existing counterparts, but if the generator raises GeneratorReturn, they trap it and return the associated value instead. Like close(), they complain with a RuntimeError if the generator doesn't finish. For example:
def throw_return(self, *exc_info): try: self.throw(*exc_info) raise RuntimeError("Generator did not terminate") except GeneratorReturn as gr: return gr.value
I don't much like the idea of adding these methods, but that is not the point of this mail.
(Note that I've also removed the 'yield raise' idea from the example - if next() or send() triggers termination of the generator with an exception other than StopIteration, then that exception is already propagated into the calling scope by the existing generator machinery. I realise Jacob was trying to make it possible to "yield an exception" without terminating the coroutine, but that idea is well beyond the scope of the PEP)
I think it was pretty clearly marked as out of scope for this PEP, but I still like the idea.
You then get:
class CalcAverage(Exception): pass
def averager(start=0): # averager that maintains a running average # and returns the final average when done count = 0 exc = None sum = start while 1: avg = sum / count try: val = yield avg except CalcAverage: return finally avg sum += val count += 1
avg = averager() avg.next() # start coroutine avg.send(1.0) # yields 1.0 avg.send(2.0) # yields 1.5 print avg.throw_return(CalcAverage) # prints 1.5
This version has a bug. It will raise ZeroDivisionError on the initial next() call used to start the generator. A better version if you insist on yielding the running average, would be: def averager(start=0): # averager that maintains a running average # and returns the final average when done count = 0 sum = start avg = None while 1: try: val = yield avg except CalcAverage: return finally avg sum += val count += 1 avg = sum/count
Now, suppose I want to write another toy coroutine that calculates the averages of two sequences and then returns the difference:
def average_diff(start=0): avg1 = yield from averager(start) avg2 = yield from averager(start) return finally avg2 - avg1
diff = average_diff() diff.next() # start coroutine # yields 0.0 avg.send(1.0) # yields 1.0 avg.send(2.0) # yields 1.5 diff.throw(CalcAverage) # Starts calculation of second average # yields 0.0 diff.send(2.0) # yields 2.0 diff.send(3.0) # yields 2.5 print diff.throw_return(CalcAverage) # Prints 1.0 (from "2.5 - 1.5")
(There is another minor bug here: the two avg.send() calls should have been diff.send()). Now for my problem. The original averager example was inspired by the tutorial http://dabeaz.com/coroutines/ that Guido pointed to. (Great stuff, btw). One pattern that is recommended by the tutorial and used throughout is to decorate all coroutines with a decorator like: def coroutine(func): def start(*args,**kwargs): cr = func(*args,**kwargs) cr.next() return cr return start The idea is that it saves you from the initial next() call used to start the coroutine. The problem is that you cannot use such a decorated coroutine in any flavor of the yield-from expression we have considered so far, because the yield-from will start out by doing an *additional* next call and yield that value. I have a few vague ideas of how we might change "yield from" to support this, but nothing concrete enough to put here. Is this a problem we should try to fix, and if so, how? not-trying-to-be-difficult-ly yours - Jacob
Jacob Holm wrote:
- the particular colour my bikeshed is painted when it comes to returning values from a generator is "return finally" (the idea being to emphasise that this represents a special "final" value for the generator that happens only after all of the normal yields are done).
We should probably drop that particular bikeshed discussion until we actually know the details of what the construct should do, esp in the context of close(). I am starting to lose track of all the different possible versions.
Note that the syntax for returning values from generators is largely independent of the semantics. Guido has pointed out that disallowing the naive "return EXPR" in generators is an important learning tool for inexperienced generator users, and I think he's right. "return finally" reads pretty well and doesn't add a new keyword, while still allowing generator return values to be written easily. I haven't seen other suggestions I particularly like, so I figured I'd run with that one for the revised example :)
- rather than trying to change the meaning of GeneratorExit and close(), 3 new generator methods would be added: next_return(), send_return() and throw_return(). The new methods have the same signatures as their existing counterparts, but if the generator raises GeneratorReturn, they trap it and return the associated value instead. Like close(), they complain with a RuntimeError if the generator doesn't finish. For example:
def throw_return(self, *exc_info): try: self.throw(*exc_info) raise RuntimeError("Generator did not terminate") except GeneratorReturn as gr: return gr.value
I don't much like the idea of adding these methods, but that is not the point of this mail.
They don't have to be generator methods - they could easily be functions in a coroutine module. However, I definitely prefer the idea of new methods or functions that support a variety of interaction styles over trying to redefine generator finalisation tools (i.e. GeneratorExit and close()) to cover this completely different use case. Why create a potential backwards compatibility problem for ourselves when there are equally clean alternative solutions? I also don't like the idea of imposing a specific coroutine return idiom in the PEP - better to have a system that supports both sentinel values (via next_return() and send_return()) and sentinel exceptions (via send_throw()).
Now for my problem. The original averager example was inspired by the tutorial http://dabeaz.com/coroutines/ that Guido pointed to. (Great stuff, btw). One pattern that is recommended by the tutorial and used throughout is to decorate all coroutines with a decorator like:
def coroutine(func): def start(*args,**kwargs): cr = func(*args,**kwargs) cr.next() return cr return start
The idea is that it saves you from the initial next() call used to start the coroutine. The problem is that you cannot use such a decorated coroutine in any flavor of the yield-from expression we have considered so far, because the yield-from will start out by doing an *additional* next call and yield that value.
I have a few vague ideas of how we might change "yield from" to support this, but nothing concrete enough to put here. Is this a problem we should try to fix, and if so, how?
Hmm, that's a tricky one. It sounds like it is definitely an issue the PEP needs to discuss, but I don't currently have an opinion as to what it should say.
not-trying-to-be-difficult-ly yours
We have a long way to go before we even come close to consuming as many pixels as PEP 308 or PEP 343 - a fact for which Greg is probably grateful ;) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------
Nick Coghlan wrote:
Jacob Holm wrote:
We should probably drop that particular bikeshed discussion until we actually know the details of what the construct should do, esp in the context of close(). I am starting to lose track of all the different possible versions.
Note that the syntax for returning values from generators is largely independent of the semantics. Guido has pointed out that disallowing the naive "return EXPR" in generators is an important learning tool for inexperienced generator users, and I think he's right.
I agree that a separate syntax for returning a value from a generator/coroutine is propably a good idea. (I am still not convinced we need a separate exception for it, but that is a separate discussion). I even think it would be a good idea to deprecate the use of the normal "return" in generators, but that is probably not going to happen.
"return finally" reads pretty well and doesn't add a new keyword, while still allowing generator return values to be written easily. I haven't seen other suggestions I particularly like, so I figured I'd run with that one for the revised example :)
You can call it whatever you want, as long as it works predictably for the use cases we are finding. [snip]
Now for my problem. The original averager example was inspired by the tutorial http://dabeaz.com/coroutines/ that Guido pointed to. (Great stuff, btw). One pattern that is recommended by the tutorial and used throughout is to decorate all coroutines with a decorator like:
def coroutine(func): def start(*args,**kwargs): cr = func(*args,**kwargs) cr.next() return cr return start
The idea is that it saves you from the initial next() call used to start the coroutine. The problem is that you cannot use such a decorated coroutine in any flavor of the yield-from expression we have considered so far, because the yield-from will start out by doing an *additional* next call and yield that value.
I have a few vague ideas of how we might change "yield from" to support this, but nothing concrete enough to put here. Is this a problem we should try to fix, and if so, how?
Hmm, that's a tricky one. It sounds like it is definitely an issue the PEP needs to discuss, but I don't currently have an opinion as to what it should say.
Here is one possible fix, never mind the syntax. We could change the yield from expression from the current: RESULT = yield from EXPR by adding an extra form, possibly one of: RESULT = yield STARTEXPR from EXPR RESULT = yield from EXPR with STARTEXPR RESULT = yield from EXPR as NAME starting with STARTEXPR(NAME) And letting STARTEXPR if given take the place of the initial _i.next() in the expansion(s). The point is we need to yield *something* first, before rerouting all send(), next() and throw() calls to the subiterator.
not-trying-to-be-difficult-ly yours
We have a long way to go before we even come close to consuming as many pixels as PEP 308 or PEP 343 - a fact for which Greg is probably grateful ;)
Him and everybody else I would think. But AFAICT we are not even close to finished, so we may get there yet. - Jacob
Jacob Holm wrote:
Here is one possible fix, never mind the syntax. We could change the yield from expression from the current:
RESULT = yield from EXPR
by adding an extra form, possibly one of:
RESULT = yield STARTEXPR from EXPR RESULT = yield from EXPR with STARTEXPR RESULT = yield from EXPR as NAME starting with STARTEXPR(NAME)
And letting STARTEXPR if given take the place of the initial _i.next() in the expansion(s). The point is we need to yield *something* first, before rerouting all send(), next() and throw() calls to the subiterator.
Another possible fix would be to have new syntax for specifying that the initial call to the coroutine should be using send or throw instead. This could be seen as a restriction on what could be used as STARTEXPR(NAME) in the earlier syntax idea. Yet another fix that requires no extra syntax would be to store the latest value yielded by the generator in a property on the generator (raising AttributeError before the first yield). Then the initial: _y = _i.next() could be replaced with: try: _y = _i.gi_latest_yield # or whatever its name would be. except AttributeError: _y = _i.next() The benefit of this version is that it requires no new syntax, it avoids the extra next() call for coroutines, and it opens some new ways of using generators. It also supports almost everything that would be possible with the syntax-based fix. (Everything if the property is actually writable, but I don't really see a use for that except perhaps for deleting it). I can even remember that I have wanted such a property before, although I don't recall the exact use case. One bad thing about it is that the initial yield made by the yield-from is then the value that the coroutine decorator was meant to discard (usually None). That might be a reason for allowing the property to be writable, or for a change in syntax after all. On the other hand, if this is a problem you can manually call the coroutine the way you want before using it in yield-from, which would then initialize the value exactly like with the second syntax idea. Of the three ideas so far, I much prefer the one without extra syntax. - Jacob
Jacob Holm wrote:
Yet another fix that requires no extra syntax would be to store the latest value yielded by the generator in a property on the generator (raising AttributeError before the first yield). Then the initial:
_y = _i.next()
could be replaced with:
try: _y = _i.gi_latest_yield # or whatever its name would be. except AttributeError: _y = _i.next()
The benefit of this version is that it requires no new syntax, it avoids the extra next() call for coroutines, and it opens some new ways of using generators. It also supports almost everything that would be possible with the syntax-based fix. (Everything if the property is actually writable, but I don't really see a use for that except perhaps for deleting it).
I can even remember that I have wanted such a property before, although I don't recall the exact use case.
One bad thing about it is that the initial yield made by the yield-from is then the value that the coroutine decorator was meant to discard (usually None). That might be a reason for allowing the property to be writable, or for a change in syntax after all. On the other hand, if this is a problem you can manually call the coroutine the way you want before using it in yield-from, which would then initialize the value exactly like with the second syntax idea.
Of the three ideas so far, I much prefer the one without extra syntax.
This issue is still bouncing around in my brain, so I don't have a lot say about it yet, but a special attribute on the generator-iterator object that the yield from expression could check was the first possible approach that occurred to me. Although, rather than it being the "latest yield" from the generator, I was thinking more of just an ordinary attribute that a @coroutine decorator could set to indicate what to yield when firing it up with 'yield from'. On your syntax ideas, note that the parser can't do anything tricky with expressions of the form "yield EXPR" - the parser will treat that as a normal yield and get confused if you try to add anything after it. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------
Nick Coghlan wrote:
This issue is still bouncing around in my brain, so I don't have a lot say about it yet, but a special attribute on the generator-iterator object that the yield from expression could check was the first possible approach that occurred to me.
Ok, keep it bouncing.
Although, rather than it being the "latest yield" from the generator, I was thinking more of just an ordinary attribute that a @coroutine decorator could set to indicate what to yield when firing it up with 'yield from'.
I made it the latest yield because I have had a use case for that in the past, and it seemed like a natural thing to do.
On your syntax ideas, note that the parser can't do anything tricky with expressions of the form "yield EXPR" - the parser will treat that as a normal yield and get confused if you try to add anything after it.
I don't really like the idea of new syntax anyway, now that it seems there is a way to avoid it. But thanks for the reminder. - Jacob
On Sat, Apr 4, 2009 at 5:01 AM, Jacob Holm <jh@improva.dk> wrote:
Another possible fix would be to have new syntax for specifying that the initial call to the coroutine should be using send or throw instead. This could be seen as a restriction on what could be used as STARTEXPR(NAME) in the earlier syntax idea.
All, please stop making more proposals. I've got it all in my head but no time to write it up right now. Hopefully before the weekend is over I'll find the time. -- --Guido van Rossum (home page: http://www.python.org/~guido/)
Jacob Holm wrote:
RESULT = yield STARTEXPR from EXPR RESULT = yield from EXPR with STARTEXPR RESULT = yield from EXPR as NAME starting with STARTEXPR(NAME)
And letting STARTEXPR if given take the place of the initial _i.next()
No, that's not satisfactory at all, because it introduces a spurious value into the stream of yielded values seen by the user of the outer generator. For refactoring to work correctly, the first value yielded by the yield-from expression *must* be the first value yielded by the subgenerator. There's no way of achieving that when using the Beazley decorator, because it thinks the first yielded value is of no interest and discards it.
We have a long way to go before we even come close to consuming as many pixels as PEP 308 or PEP 343 - a fact for which Greg is probably grateful ;)
At least Guido will know if someone manages to unsubscribe him from the list -- he won't be getting 500 messages about yield-from every day. :-) -- Greg
Nick Coghlan wrote:
"return finally" reads pretty well and doesn't add a new keyword
Still doesn't mean anything, though. Ordinary returns happen "finally" too (most of the time, anyway), so what's the difference? -- Greg
Greg Ewing wrote:
Nick Coghlan wrote:
"return finally" reads pretty well and doesn't add a new keyword
Still doesn't mean anything, though. Ordinary returns happen "finally" too (most of the time, anyway), so what's the difference?
It needs to be mnemonic, not literal. The difference between a generator return and a normal function return is that with a generator you will typically have at least one yield before the actual return, whereas with a normal function, return or an exception are the only way to leave the function's frame. So "return finally" is meant to help you remember that it happens only after all the yields are done. Guido has said he is OK with losing the novice assistance on this one though, so it's probably a moot point. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------
On 4/3/09, Nick Coghlan <ncoghlan@gmail.com> wrote:
Greg tried to clarify this a bit already, but I think Jacob's averager example is an interesting case where it makes sense to both yield multiple times and also "return a value".
def averager(start=0): # averager that maintains a running average # and returns the final average when done count = 0 exc = None sum = start while 1: avg = sum / count try: val = yield avg except CalcAverage: return finally avg sum += val count += 1
It looks to me like it returns (or yields) the running average either way. I see a reason to send in a sentinel value, saying "Don't update the average, just tell me the current value." I don't see why that sentinel has to terminate the generator, nor do I see why that final average has to be returned rather than yielded. -jJ
Jim Jewett wrote:
It looks to me like it returns (or yields) the running average either way.
That is because Nick has mangled my beautiful example - sorry Nick :) You can see my original example at: http://mail.python.org/pipermail/python-ideas/2009-April/003841.html and a few arguments why I think it is better at: http://mail.python.org/pipermail/python-ideas/2009-April/003847.html
I see a reason to send in a sentinel value, saying "Don't update the average, just tell me the current value."
I don't see why that sentinel has to terminate the generator, nor do I see why that final average has to be returned rather than yielded.
Yielding the current value on each send was not part of the original example because I was thinking in terms of well-behaved coroutines as described in http://dabeaz.com/coroutines/. I agree that it makes sense for running averages, but it is not that hard to come up with similar examples where the intermediate state is not really useful and/or may be expensive to compute. The reason for closing would be that once you have computed the final result, you want whatever resources the coroutine is using to be freed. Since only the final result is assumed to be useful, it makes perfect sense to close the coroutine at the same time as you are requesting the final result. - Jacob
On 4/3/09, Jacob Holm <jh@improva.dk> wrote:
Yielding the current value on each send was not part of the original example ...
So in the original you cared about the final value, but not the intermediate yields. My question is whether there is a sane case where you care about *both*, *and* you care about distinguishing them. And is this case common really enough that we don't want it marked up with something more explicit, like a sentinel or a raise?
The reason for closing would be that once you have computed the final result, you want whatever resources the coroutine is using to be freed. Since only the final result is assumed to be useful, it makes perfect sense to close the coroutine at the same time as you are requesting the final result.
def outer(unfinished=object()): g=inner(unfinished) for result in g: yield unfinished # cooperative multi-tasking, so co-operate if result is not unfinished: break ... I see some value in simplifying that, or adding more power. But I'm not convinced the current proposal actually is much simpler, or that the extra power wouldn't be better written in a more explicit manner. I think the above still translates into def outer(unfinished=object()): # # now also need to set an initial value of result # # *OR* distinguish intermediate from final results. # result=unfinished g=inner(unfinished) # # loop is now implicit # while result is unfinished: # result = yield from g result = yield from g ... -jJ
On Fri, Apr 3, 2009 at 10:47 AM, Jacob Holm <jh@improva.dk> wrote:
You can see my original example at:
http://mail.python.org/pipermail/python-ideas/2009-April/003841.html
and a few arguments why I think it is better at:
http://mail.python.org/pipermail/python-ideas/2009-April/003847.html [...] The reason for closing would be that once you have computed the final result, you want whatever resources the coroutine is using to be freed. Since only the final result is assumed to be useful, it makes perfect sense to close the coroutine at the same time as you are requesting the final result.
Hm. I am beginning to see what you are asking for. Your averager example is somewhat convincing. Interestingly, in a sense it seems unrelated to yield-from: the averager doesn't seem to need yield-from, it receives values sent to it using send(). An alternative version (that works today), which I find a bit clearer, uses exceptions instead of a sentinel value: when you are done with sending it the sequence of values, you throw() a special exception into it, and in response it raises another exception back with a value attribute. I'm showing the usage example first, then the support code. Usage example: @coroutine def summer(): sum = 0 while True: try: value = yield except Terminate: raise Done(sum) else: sum += value def main(): a = summer() a.send(1) a.send(2) print finalize(a) Support code: class Terminate(Exception): """Exception thrown into the generator to ask it to stop.""" class Done(Exception): """Exception raised by the generator when it catches Terminate.""" def __init__(self, value=None): self.value = value def coroutine(func): """Decorator around a coroutine, to make the initial next() call.""" def wrapper(*args, **kwds): g = func(*args, **kwds) g.next() return g return wrapper def finalize(g): """Throw Terminate into a couroutine and extract the value from Done.""" try: g.throw(Terminate) except Done as e: return e.value else: g.close() raise RuntimeError("Expected Done(<value>)") I use a different exception to throw into the exception as what it raises in response, so that mistakes (e.g. the generator not catching Terminate) are caught, and no confusion can exist with the built-in exceptions StopIteration and GeneratorExit. Now I'll compare this manual version with your (Jacob Holm's) proposal: - instead of Done you use GeneratorExit - hence, instead of g.throw(Done) you can use g.close() - instead of Terminate you use StopException - you want g.close() to extract and return the value from StopException - you use "return value" instead of "raise Done(value)" The usage example then becomes, with original version indicated in comments: @coroutine def summer(): sum = 0 while True: try: value = yield except GeneratorExit: # except Terminate: return sum # raise Done(sum) else: sum += value def main(): a = summer() a.send(1) a.send(2) print a.close() # print finalize(a) At this point, I admin that I am not yet convinced. On the one hand, my support code melts away, except for the @coroutine decorator. On the other hand: - the overloading of GeneratorExit and StopIteration reduces diagnostics for common beginner's mistakes when writing regular (iterator-style) generator code - the usage example isn't much simpler - the support code isn't rocket science - the coroutine use case is specialized enough that a little support seems okay (you still need @coroutine anyway) The last three points don't sway me either way: they pit minor conveniences against minor inconveniences. However, the first point worries me a lot. The concern over StopIteration can be dealt with by introducing a new exception that is raised only by "return value" inside a generator. But I'm also worried that the mere need to catch GeneratorExit for a purpose other than resource cleanup will cause examples using it to pop up on the web, which will then be copied and modified by clueless beginners, and *increase* the probability of bad code being written. (That's why I introduce *new* exceptions in my support code -- they don't have predefined meanings in other contexts.) Finally, I am not sure of the connection with "yield from". I don't see a way to exploit it for this example. As an exercise, I constructed an "averager" generator out of the above "summer" and a similar "counter", and I didn't see a way to exploit "yield from". The only connection seems to be PEP 380's proposal to turn "return value" inside a generator into "raise StopIteration(value)", and that's the one part of the PEP with which I have a problem anyway (the beginner's issues above). Oh, and "yield from" competes with @couroutine over when the initial next() call is made, which again suggests the two styles (yield-from and coroutines) are incompatible. All in all, I think I would be okay with turning "return value" inside a generator into raising *some* exception, as long as that exception is not StopIteration (nor derives from it, nor from GeneratorExit). PEP 380 and its implementation would become just a tad more complex, but I think that's worth it. Generators used as iterators would raise a (normally) uncaught exception if they returned a value, and that's my main requirement. I'm still not convince that more is needed, in particular I'm still -0 on catching this value in gen_close() and returning the value attribute from there. As I've said before, I don't care whether "return None" would be treated more like "return" or more like "return value" -- for beginners' code I don't think it matters, and for advanced code they should be equivalent. I'll stop arguing for new syntax to return a value from a generator (like Phillip Eby's proposed "return from yield with <value>"): I don't think it adds enough to overcome the pain for the parser and other tools. Finally, as far as a name for the new exception, I think something long like ReturnFromGenerator would be fine, since most of the time it is handled implicitly by coroutine support code (whether this is user code or gen_close()) or the yield-from implementation. I'm sorry for being so long winded and yet somewhat inconclusive. I wouldn't have bothered if I didn't think there was something worth pursuing. But it sure seems elusive. -- --Guido van Rossum (home page: http://www.python.org/~guido/)
Hi Guido Thank you for taking the time. I'll try to be brief so I don't spend more of it than necesary... (I'll probably fail though) Guido van Rossum wrote:
Hm. I am beginning to see what you are asking for. Your averager example is somewhat convincing.
Interestingly, in a sense it seems unrelated to yield-from: the averager doesn't seem to need yield-from, it receives values sent to it using send().
It is related to yield-from only because the "return value from generator" concept is introduced there. I was trying to work out the details of how I would like that to work in the context of GeneratorExit and needed a near-trivial example. [snip "summer" example using existing features and comparison with version using suggested new features]
At this point, I admin that I am not yet convinced. On the one hand, my support code melts away, except for the @coroutine decorator. On the other hand:
- the overloading of GeneratorExit and StopIteration reduces diagnostics for common beginner's mistakes when writing regular (iterator-style) generator code - the usage example isn't much simpler - the support code isn't rocket science - the coroutine use case is specialized enough that a little support seems okay (you still need @coroutine anyway)
The last three points don't sway me either way: they pit minor conveniences against minor inconveniences.
However, the first point worries me a lot. The concern over StopIteration can be dealt with by introducing a new exception that is raised only by "return value" inside a generator.
I am still trying to get a clear picture of what kind of mistakes you are trying to protect against. If it is people accidently writing return in a generator when they really mean yield, that is what I thought the proposal for an alternate syntax was for. That sounds like a good idea to me, especially if we could also ban or discourage the use of normal return. But the alternate syntax doesn't have to mean a different exception. Since you are no longer pushing an alternative syntax for return but still want a different exception, I'll assume there is some other beginner mistake you are worried about. My guess is it is some mistake at the places where the generator is used, but I am having a hard time figuring out where the mistake could be in ignoring the returned value. Perhaps you (or someone who has more time) can provide an example where this is a bad thing?
But I'm also worried that the mere need to catch GeneratorExit for a purpose other than resource cleanup will cause examples using it to pop up on the web, which will then be copied and modified by clueless beginners, and *increase* the probability of bad code being written. (That's why I introduce *new* exceptions in my support code -- they don't have predefined meanings in other contexts.)
That worry I can understand.
Finally, I am not sure of the connection with "yield from". I don't see a way to exploit it for this example. As an exercise, I constructed an "averager" generator out of the above "summer" and a similar "counter", and I didn't see a way to exploit "yield from". The only connection seems to be PEP 380's proposal to turn "return value" inside a generator into "raise StopIteration(value)", and that's the one part of the PEP with which I have a problem anyway (the beginner's issues above).
Yes, the only connection is that this is where "return value" is introduced. I could easily see "return value" as a separate PEP, except PEP 380 provides one of the better reasons for its inclusion. It might be good to figure out how this feature should work by itself before complicating things by integrating it in the yield-from semantics.
Oh, and "yield from" competes with @couroutine over when the initial next() call is made, which again suggests the two styles (yield-from and coroutines) are incompatible.
It is a serious problem, because one of the major points of the PEP is that it should be useful for refactoring coroutines. As a matter of fact, I started another thread on this specific issue earlier today which only Nick has so far responded to. I think it is solvable, but requires some more work.
All in all, I think I would be okay with turning "return value" inside a generator into raising *some* exception, as long as that exception is not StopIteration (nor derives from it, nor from GeneratorExit). PEP 380 and its implementation would become just a tad more complex, but I think that's worth it. Generators used as iterators would raise a (normally) uncaught exception if they returned a value, and that's my main requirement. I'm still not convince that more is needed, in particular I'm still -0 on catching this value in gen_close() and returning the value attribute from there.
As long as close is not catching it without also returning the value. That would be *really* annoying.
As I've said before, I don't care whether "return None" would be treated more like "return" or more like "return value" -- for beginners' code I don't think it matters, and for advanced code they should be equivalent.
I'll stop arguing for new syntax to return a value from a generator (like Phillip Eby's proposed "return from yield with <value>"): I don't think it adds enough to overcome the pain for the parser and other tools.
And here I was starting to think that a new syntax for return could solve the problem of beginner mistakes without needing a new exception.
Finally, as far as a name for the new exception, I think something long like ReturnFromGenerator would be fine, since most of the time it is handled implicitly by coroutine support code (whether this is user code or gen_close()) or the yield-from implementation.
GeneratorReturn is the name we have been using for this beast so far, but I really don't care what it is called.
I'm sorry for being so long winded and yet somewhat inconclusive. I wouldn't have bothered if I didn't think there was something worth pursuing. But it sure seems elusive.
And I thank you again for your time. Best regards - Jacob
Jacob Holm wrote:
Since you are no longer pushing an alternative syntax for return but still want a different exception, I'll assume there is some other beginner mistake you are worried about. My guess is it is some mistake at the places where the generator is used, but I am having a hard time figuring out where the mistake could be in ignoring the returned value. Perhaps you (or someone who has more time) can provide an example where this is a bad thing?
I can't speak for Guido, but the two easy beginner mistakes I think are worth preventing: - using 'return' where you meant 'yield' (however, if even 'return finally' doesn't appeal to Guido as alternative syntax for "no, this is a coroutine, I really mean it" then I'm fine with that) - trying to iterate normally over a coroutine instead of calling it appropriately (raising GeneratorReturn instead of StopIteration means that existing iterative code will let the new exception escape rather than silently suppressing the return exception) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------
Nick Coghlan wrote:
Jacob Holm wrote:
Since you are no longer pushing an alternative syntax for return but still want a different exception, I'll assume there is some other beginner mistake you are worried about. My guess is it is some mistake at the places where the generator is used, but I am having a hard time figuring out where the mistake could be in ignoring the returned value. Perhaps you (or someone who has more time) can provide an example where this is a bad thing?
I can't speak for Guido, but the two easy beginner mistakes I think are worth preventing:
- using 'return' where you meant 'yield' (however, if even 'return finally' doesn't appeal to Guido as alternative syntax for "no, this is a coroutine, I really mean it" then I'm fine with that)
Good, this one I understand.
- trying to iterate normally over a coroutine instead of calling it appropriately (raising GeneratorReturn instead of StopIteration means that existing iterative code will let the new exception escape rather than silently suppressing the return exception)
But this one I still don't get. Let me try a couple of cases: 1) We have a coroutine that expects you to call send and/or throw with specific values, and ends up returning a value. A beginner may try to iterate over it, but will most likely get an exception on the first next() call because the input is not valid. Or he would get an infinite loop because None is not changing the state of the coroutine. In any case, it is unlikely that he will get to see either StopIteration or the new exception, because the input is not what the coroutine expects. The new exception doesn't help here. 2) We have a generator that e.g. pulls values from a file, yielding the processed values as it goes along, and returning some form of summary at the end. If I iterate over it with a for-loop, I get all the values asn usual ... followed by an exception. Why do I have to get an exception there just because the generator has some information that its implementer thought I might want? Ignoring the value in this case seems perfectly reasonable, so having to catch an exception is just noise here. 3) We have a coroutine that computes something expensive, occationally yielding to let other code run. It neither sends or receives values, just uses yield for cooperative multitasking. When it is done it returns a value. If you loop over this coroutine, you will get a bunch of Nones, followed by the new exception. You could argue that the new exception helps you here. One way of accessing the returned value would be to catch it and look at an attribute. However, for this case I would prefer to just call close on the generator to get the value afterwards. A beginner might be helped by the unexpected exception, but I think even a beginner would find that something strange was going on when the only value he gets for the loop variable is None. He might even look up the documentation for the coroutine he was calling and see how it was supposed to be used. 4) ... ? Do you have other concrete use cases I haven't thought of where a the new exception would help? - Jacob
On Fri, Apr 3, 2009 at 6:47 PM, Jacob Holm <jh@improva.dk> wrote:
1) We have a coroutine that expects you to call send and/or throw with specific values, and ends up returning a value. A beginner may try to iterate over it, but will most likely get an exception on the first next() call because the input is not valid. Or he would get an infinite loop because None is not changing the state of the coroutine. In any case, it is unlikely that he will get to see either StopIteration or the new exception, because the input is not what the coroutine expects. The new exception doesn't help here.
2) We have a generator that e.g. pulls values from a file, yielding the processed values as it goes along, and returning some form of summary at the end. If I iterate over it with a for-loop, I get all the values asn usual ... followed by an exception. Why do I have to get an exception there just because the generator has some information that its implementer thought I might want? Ignoring the value in this case seems perfectly reasonable, so having to catch an exception is just noise here.
Sorry, I read this message after writing a long response to an earlier message of you where I rejected the idea of new syntax for returning a value from a generator. I find this example somewhat convincing, and more so because the extra processing of ReturnFromGenerator makes my proposal a bit messy: there are three "except StopIteration" clauses, all with parallel "except ReturnFromGenerator" clauses. Though the real implementation would probably merge all that code into a single C-level function. And new syntax *is* a much bigger burden than a new exception. I think I need to ponder this for a while and think more about how important it really is to hold the hand of newbies trying to write a vanilla generator, vs. how important this use case really is (it's easily solved with a class, for example).
3) We have a coroutine that computes something expensive, occationally yielding to let other code run. It neither sends or receives values, just uses yield for cooperative multitasking. When it is done it returns a value. If you loop over this coroutine, you will get a bunch of Nones, followed by the new exception. You could argue that the new exception helps you here. One way of accessing the returned value would be to catch it and look at an attribute. However, for this case I would prefer to just call close on the generator to get the value afterwards. A beginner might be helped by the unexpected exception, but I think even a beginner would find that something strange was going on when the only value he gets for the loop variable is None. He might even look up the documentation for the coroutine he was calling and see how it was supposed to be used.
Well if you have nothing else to do you could just use "yield from" over this coroutine and get the return value through that syntax. And if you're going to call next() on it with other activities in between, you have to catch StopIteration from that next() call anyway -- you would have to also catch ReturnFromGenerator to extract the value. I don't believe that once the generator has raised StopIteration or ReturnFromGenerator, the return value should be saved somewhere to be retrieved with an explicit close() call -- I want to be able to free all resources once the generator frame is dead. -- --Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum wrote:
how important this use case really is (it's easily solved with a class, for example).
Yes, it's my feeling that a class would be better for this kind of thing too. Let's not lose sight of the fundamental motivation for all this, the way I see it at least: yield-from is primarily to permit factoring of generator code. Any proposals for enhancements or extensions ought to be justified in relation to that.
I don't believe that once the generator has raised StopIteration or ReturnFromGenerator, the return value should be saved somewhere to be retrieved with an explicit close() call -- I want to be able to free all resources once the generator frame is dead.
I agree with that. As a corollary, I *don't* think that close() should return the value of a ReturnFromGenerator even if it gets one, because unless the value is stored, you'll only get it the first time close() is called, and only if the generator has not already completed normally. That would make it too unreliable for any practical use as far as I can see. -- Greg
Greg Ewing wrote:
Guido van Rossum wrote:
I don't believe that once the generator has raised StopIteration or ReturnFromGenerator, the return value should be saved somewhere to be retrieved with an explicit close() call -- I want to be able to free all resources once the generator frame is dead.
I agree with that.
I don't think it is common to keep the generator object alive long after the generator is closed, so I don't see the problem in keeping the value so it can be returned by the next close() call.
As a corollary, I *don't* think that close() should return the value of a ReturnFromGenerator even if it gets one, because unless the value is stored, you'll only get it the first time close() is called, and only if the generator has not already completed normally. That would make it too unreliable for any practical use as far as I can see.
And I think it is a mistake to have close() swallow the return value. If it catches ReturnFromGenerator, it should also return the value or raise a RuntimeError. In my order of preference: 1. "return value" in a generator raises StopIteration(value). Any exception raised by next, send or throw sets a return value on the generator on the way out. If the exception is a StopIteration or GeneratorExit (see below) that was not the argument to throw, the value is taken from there, else it is None. Any next(), send(), or throw() operation on a closed generator raises a new StopIteration using the saved value. When close catches a StopIteration or GeneratorExit it returns the value. After yield-from calls close as part of its GeneratorExit handling, it raises a new GeneratorExit with the returned value. The GeneratorExit part lets "def outer(): return yield from inner()" behave as expected. 2. Same as #1 but using ReturnFromGenerator(value) instead of StopIteration. 3. Same as #1 but without attaching return value to GeneratorExit in yield-from. 4. Same as #3 but using ReturnFromGenerator(value) instead of StopIteration. 5. Same as #1 but without storing the value on the generator. 6. Same as #5 but using ReturnFromGenerator(value) instead of StopIteration. 7. "return value" in a generator raises ReturnFromGenerator(value). close() doesn't catch it. 8. "return value" in a generator raises ReturnFromGenerator(value). close() catches ReturnFromGenerator and raises a RuntimeError. 9. Anything else that has been suggested. In particular anything where close() catches ReturnFromGenerator without either returning the value or raising another exception. Too many options? This is just a small subset of what has been discussed. - Jacob
Jacob Holm wrote:
Greg Ewing wrote:
Guido van Rossum wrote:
I don't believe that once the generator has raised StopIteration or ReturnFromGenerator, the return value should be saved somewhere to be retrieved with an explicit close() call -- I want to be able to free all resources once the generator frame is dead.
I agree with that.
I don't think it is common to keep the generator object alive long after the generator is closed, so I don't see the problem in keeping the value so it can be returned by the next close() call.
I don't think close() means to me what it means to you... close() to me means "I'm done with this, it should have been exhausted already, but just to make sure all the resources held by the internal frame are released properly, I'm shutting it down explicitly" In other words, the *normal* flow for close() should be the "frame has already terminated, so just return immediately" path, not the "frame hasn't terminated yet, so throw GeneratorExit in and complain if the frame doesn't terminate" path. You're trying to move throwing GeneratorExit into the internal frame from the exceptional path to the normal path and I don't think that is a good idea. Far better to attach the return value to StopIteration as in Greg's original proposal (since Guido no longer appears to be advocating a separate exception for returning a value, we should be able to just go back to Greg's original approach) and use a normal next(), send() or throw() call along with a helper function to catch the StopIteration. Heck, with that approach, you can even write a context manager to catch the result for you: @contextmanager class cr_result(object): def __init__(self, cr): self.coroutine = cr self.result = None def __enter__(self): return self def __exit__(self, et, ev, tb): if et is StopIteration: self.result = ev.value return True # Trap StopIteration # Anything else is propagated with cr_result(a) as x: # Call a.next()/a.send()/a.throw() as you like # Use x.result to retrieve the coroutine's value Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------
Nick Coghlan wrote:
Jacob Holm wrote:
I don't think it is common to keep the generator object alive long after the generator is closed, so I don't see the problem in keeping the value so it can be returned by the next close() call.
I don't think close() means to me what it means to you... close() to me means "I'm done with this, it should have been exhausted already, but just to make sure all the resources held by the internal frame are released properly, I'm shutting it down explicitly"
In other words, the *normal* flow for close() should be the "frame has already terminated, so just return immediately" path, not the "frame hasn't terminated yet, so throw GeneratorExit in and complain if the frame doesn't terminate" path.
That is why #1-6 in my list took care to extract the value from StopIteration and attach it to the generator. Doing it like that allows you to ask for the value after the generator is exhausted normally, using either next() or close(). This is interesting because it allows you to loop over the generator with a normal for-loop and *still* get the return value after the loop if you want it. (You have to construct the generator before the loop instead of in the for-loop statement itself, and call close() on it afterwards, but that is easy). It also makes it possible for close to reliably return the value. The idea of saving the value on the generator is more basic than the idea of having close return a value. It means that calling next on an exhausted generator will keep raising StopIteration with the same value. If you don't save the return value on the generator, only the first StopIteration will have a value, the rest will always have None as their value.
You're trying to move throwing GeneratorExit into the internal frame from the exceptional path to the normal path and I don't think that is a good idea.
I think it is exacltly the right thing for the use cases I have. Anything else requires extra support code to get a similar api. (Extra exceptions to throw in and/or out, an alternative close function to catch the extra exceptions, probably other things as well). Whether or not it is a good idea to use GeneratorExit for this, I think it is important that a "return value from GeneratorExit" does not silently throw away the value. In other words, if close does *not* return the value it gets from StopIteration, it should raise an exception if that value is not None. One option is to let close() reraise the StopIteration if it has a non-None value. This matches Guidos suggestion for a way to access the return value after a GeneratorExit in yield-from without changing his suggested expansion. If the return value from the generator isn't stored and I can't have close() return the value, this would be my preference. Another option (if you insist that it is an error to return a value after a GeneratorExit) is to let close() raise a RuntimeError when it catches a StopIteration with a non-None value. - Jacob
Jacob Holm wrote:
Another option (if you insist that it is an error to return a value after a GeneratorExit) is to let close() raise a RuntimeError when it catches a StopIteration with a non-None value.
Why do you consider it OK for close() to throw away all of the values the generator might have yielded in the future, but not OK for it to throw away the generator's return value? The objection I have to having close() return a value is that it encourages people to start using GeneratorExit in their normal generator control flow and I think that's a really bad idea (on par with calling sys.exit() and then trapping SystemExit to terminate a search loop - perfectly legal from a language point of view, but a really bad plan nonetheless). Now, the fact that repeatedly calling next()/send()/throw() on a finished generator is meant to keep reraising the same StopIteration that was thrown when the generator first terminated is a *much* better justification for preserving the return value on the generator object. But coupling that with the idea of close() doing anything more than giving an unfinished generator a final chance to release any resources it is holding is mixing two completely different ideas. Better to just add a "value" property to generators that raises a RuntimeError if the generator frame hasn't terminated yet (probably along with a "finished" property to allow LBYL interrogation of the generator state). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------
Nick Coghlan wrote:
Jacob Holm wrote:
Another option (if you insist that it is an error to return a value after a GeneratorExit) is to let close() raise a RuntimeError when it catches a StopIteration with a non-None value.
Why do you consider it OK for close() to throw away all of the values the generator might have yielded in the future, but not OK for it to throw away the generator's return value?
Because the return value is actually computed and returned from the generator, *then* thrown away. If there is no way to access the value, it should be considered an error to return it and flagged as such. What the generator might have yielded if close wasn't called doesn't interest me the slightest.
The objection I have to having close() return a value is that it encourages people to start using GeneratorExit in their normal generator control flow and I think that's a really bad idea (on par with calling sys.exit() and then trapping SystemExit to terminate a search loop - perfectly legal from a language point of view, but a really bad plan nonetheless).
Yes, I understand that this is how you think of GeneratorExit.
Now, the fact that repeatedly calling next()/send()/throw() on a finished generator is meant to keep reraising the same StopIteration that was thrown when the generator first terminated is a *much* better justification for preserving the return value on the generator object.
Ok
But coupling that with the idea of close() doing anything more than giving an unfinished generator a final chance to release any resources it is holding is mixing two completely different ideas.
Better to just add a "value" property to generators that raises a RuntimeError if the generator frame hasn't terminated yet (probably along with a "finished" property to allow LBYL interrogation of the generator state).
Why not a single property raising AttributeError until the frame is terminated? (Not that I really care as long as I can access the value without having access to the original StopIteration). If the value is stored, I am fine with close not returning it or raising an exception. - Jacob
On Sat, Apr 4, 2009 at 4:27 PM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
Let's not lose sight of the fundamental motivation for all this, the way I see it at least: yield-from is primarily to permit factoring of generator code. Any proposals for enhancements or extensions ought to be justified in relation to that.
I still don't think that refactoring should drive the design exclusively. Refactoring is *one* thing that becomes easier with yield-from. But I want the design to look pretty from as many angles as possible.
I don't believe that once the generator has raised StopIteration or ReturnFromGenerator, the return value should be saved somewhere to be retrieved with an explicit close() call -- I want to be able to free all resources once the generator frame is dead.
I agree with that.
As a corollary, I *don't* think that close() should return the value of a ReturnFromGenerator even if it gets one, because unless the value is stored, you'll only get it the first time close() is called, and only if the generator has not already completed normally. That would make it too unreliable for any practical use as far as I can see.
Throwing in GeneratorExit and catching the ReturnFromGenerator exception would have the same problem though, so I'm not sure I buy this argument. -- --Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum wrote:
Throwing in GeneratorExit and catching the ReturnFromGenerator exception would have the same problem though, so I'm not sure I buy this argument.
I'm not advocating doing that. My view is that both calling close() on the generator and throwing GeneratorExit into it are things you only do to make sure the generator cleans up. You can't expect to get a meaningful return value either way. -- Greg
Guido van Rossum wrote:
I still don't think that refactoring should drive the design exclusively. Refactoring is *one* thing that becomes easier with yield-from. But I want the design to look pretty from as many angles as possible.
Certainly. But I feel that any extra features should be in some sense extensions or generalizations of what is needed to support refactoring. Otherwise the scope of the proposal can expand without bound. -- Greg
[Answering somewhat out of order; new proposal developed at the end.] On Fri, Apr 3, 2009 at 4:25 PM, Jacob Holm <jh@improva.dk> wrote:
I am still trying to get a clear picture of what kind of mistakes you are trying to protect against. If it is people accidently writing return in a generator when they really mean yield, that is what I thought the proposal for an alternate syntax was for. That sounds like a good idea to me, especially if we could also ban or discourage the use of normal return. But the alternate syntax doesn't have to mean a different exception.
I could easily see "return value" as a separate PEP, except PEP 380 provides one of the better reasons for its inclusion. It might be good to figure out how this feature should work by itself before complicating
I am leaning the other way now. New syntax for returning in a value is a high-cost proposition. Instead, I think we can guard against most of the same mistakes (mixing yield and return in a generator used as an iterator) by using a different exception to pass the value. This would delay the failure to runtime, but it would still fail loudly, which is good enough for me. I want to name the new exception ReturnFromGenerator to minimize the similarity with GeneratorExit: if we had both GeneratorExit and GeneratorReturn there would be endless questions on the newbie forums about the differences between the two, and people might use the wrong one. Since ReturnFromGenerator goes *out* of the generator and GeneratorExit goes *in*, there really are no useful parallels, and similar names would cause confusion. things
by integrating it in the yield-from semantics.
Oh, and "yield from" competes with @couroutine over when the initial next() call is made, which again suggests the two styles (yield-from and coroutines) are incompatible.
It is a serious problem, because one of the major points of the PEP is
Here are my curent thoughts on this. When a generator returns, the return statement is treated normally (whether or not it has a value) until the frame is about to be left (i.e. after any finally-clauses have run). Then, it is converted to StopIteration if there was no value or ReturnFromGenerator if there was a value. I don't care which one is picked for an explicit "return None" -- that should be decided by implementation expediency. (E.g. if one requires adding new opcodes and one doesn't, I'd pick the one that doesn't.) Normal loops (for-loops, list comprehensions, other implied loops) only catch StopIteration, so that returning a value is still wrong here. But some other contexts treat ReturnFromGenerator similar as StopIteration except the latter conveys None and the former conveys an explicit value. This applies to yield-from as well as to explicit or implied closing of the generator (close() or deallocation). So g.close() returns the value (I think I earlier said I didn't like that -- I turned around on this one). It's pseudo-code is roughly: def close(it): try: it.throw(GeneratorExit) except (GeneratorExit, StopIteration): return None except ReturnFromGenerator as e: # This block is really the only new thing return e.value # Other exceptions are passed out unchanged else: # throw() yielded a value -- unchanged raise RuntimeError(.....) Deleting a generator is like closing and printing (!) a traceback (to stderr) if close() raises an exception. A returned value it is just ignored. Explicit pseudo-code without falling back to close(): def __del__(it): try: it.throw(GeneratorExit) except (GeneratorExit, StopIteration, ReturnFromGenerator): pass except: # Some other exception happened <print traceback> else: # throw() yielded another value <print traceback> I have also worked out what I want yield-from to do, see end of this message. [Guido] that
it should be useful for refactoring coroutines. As a matter of fact, I started another thread on this specific issue earlier today which only Nick has so far responded to. I think it is solvable, but requires some more work.
I think that's the thread where I asked you and Nick to stop making more proposals.I a worried that a solution would become too complex, and I want to keep the "naive" interpretation of "yield from EXPR" to be as close as possible to "for x in EXPR: yield x". I think the @coroutine generator (whether built-in or not) or explicit "priming" by a next() call is fine. ----- So now let me develop my full thoughts on yield-from. This is unfortunately long, because I want to show some intermediate stages. I am using a green font for new code. I am using stages, where each stage provides a better approximation of the desired semantics. Note that each stage *adds* some semantics for corner cases that weren't handled the same way in the previous stage. Each stage proposes an expansion for "RETVAL = yield from EXPR". I am using Py3k syntax. 1. Stage one uses the for-loop equivalence: for x in EXPR: yield x RETVAL = None 2. Stage two expands the for-loop into an explicit while-loop that has the same meaning. It also sets RETVAL when breaking out of the loop. This prepares for the subsequent stages. Note that we have an explicit iter(EXPR) call here, since that is what a for-loop does: it = iter(EXPR) while True: try: x = next(it) except StopIteration: RETVAL = None; break yield x 3. Stage three further rearranges stage 2 without making semantic changes, Again this prepares for later stages: it = iter(EXPR) try: x = next(it) except StopIteration: RETVAL = e.value else: while True: yield x try: x = next(x) except StopIteration: RETVAL = None; break 4. Stage four adds handling for ReturnFromGenerator, in both places where next() is called: it = iter(EXPR) try: x = next(it) except StopIteration: RETVAL = e.value except ReturnFromGenerator as e: RETVAL = e.value; break else: while True: yield x try: x = next(it) except StopIteration: RETVAL = None; break except ReturnFromGenerator as e: RETVAL = e.value; break yield x 5. Stage five shows what should happen if "yield x" above returns a value: it is passed into the subgenerator using send(). I am ignoring for now what happens if it is not a generator; this will be cleared up later. Note that the initial next() call does not change into a send() call, because there is no value to send before before we have yielded: it = iter(EXPR) try: x = next(it) except StopIteration: RETVAL = None except ReturnFromGenerator as e: RETVAL = e.value else: while True: v = yield x try: x = it.send(v) except StopIteration: RETVAL = None; break except ReturnFromGenerator as e: RETVAL = e.value; break 6. Stage six adds more refined semantics for when "yield x" raises an exception: it is thrown into the generator, except if it is GeneratorExit, in which case we close() the generator and re-raise it (in this case the loop cannot continue so we do not set RETVAL): it = iter(EXPR) try: x = next(it) except StopIteration: RETVAL = None except ReturnFromGenerator as e: RETVAL = e.value else: while True: try: v = yield x except GeneratorExit: it.close() raise except: try: x = it.throw(*sys.exc_info()) except StopIteration: RETVAL = None; break except ReturnFromGenerator as e: RETVAL = e.value; break else: try: x = it.send(v) except StopIteration: RETVAL = None; break except ReturnFromGenerator as e: RETVAL = e.value; break 7. In stage 7 we finally ask ourselves what should happen if it is not a generator (but some other iterator). The best answer seems subtle: send() should degenerator to next(), and all exceptions should simply be re-raised. We can conceptually specify this by simply re-using the for-loop expansion: it = iter(EXPR) if <it is not a generator>: for x in it: yield next(x) RETVAL = None else: try: x = next(it) except StopIteration: RETVAL = None except ReturnFromGenerator as e: RETVAL = e.value else: while True: try: v = yield x except GeneratorExit: it.close() raise except: try: x = it.throw(*sys.exc_info()) except StopIteration: RETVAL = None; break except ReturnFromGenerator as e: RETVAL = e.value; break else: try: x = it.send(v) except StopIteration: RETVAL = None; break except ReturnFromGenerator as e: RETVAL = e.value; break Note: I don't mean that we literally should have a separate code path for non-generators. But writing it this way adds the generator test to one place in the spec, which helps understanding why I am choosing these semantics. The entire code of stage 6 degenerates to stage 1 if we make the following substitutions: it.send(v) -> next(v) it.throw(sys.exc_info()) -> raise it.close() -> pass (Except for some edge cases if the incoming exception is StopIteration or ReturnFromgenerator, so we'd have to do the test before entering the try/except block around the throw() or send() call.) We could do this based on the presence or absence of the send/throw/close attributes: this would be duck typing. Or we could use isinstance(it, types.GeneratorType). I'm not sure there are strong arguments for either interpretation. The type check might be a little faster. We could even check for an exact type, since GeneratorType is final. Perhaps the most important consideration is that if EXPR produces a file stream object (which has a close() method), it would not consistently be closed: it would be closed if the outer generator was closed before reaching the end, but not if the loop was allowed to run until the end of the file. So I'm leaning towards only making the generator-specific method calls if it is really a generator. -- --Guido van Rossum (home page: http://www.python.org/~guido/<http://www.python.org/%7Eguido/> )
Guido van Rossum wrote:
Oh, and "yield from" competes with @couroutine over when the initial next() call is made, which again suggests the two styles (yield-from and coroutines) are incompatible.
It is a serious problem, because one of the major points of the PEP is
[Guido] that
it should be useful for refactoring coroutines. As a matter of fact, I started another thread on this specific issue earlier today which only Nick has so far responded to. I think it is solvable, but requires some more work.
I think that's the thread where I asked you and Nick to stop making more proposals.I a worried that a solution would become too complex, and I want to keep the "naive" interpretation of "yield from EXPR" to be as close as possible to "for x in EXPR: yield x". I think the @coroutine generator (whether built-in or not) or explicit "priming" by a next() call is fine.
The trick is that if the definition of "yield from" *includes* the priming step, then we are saying that coroutines *shouldn't* be primed in a decorator. I don't actually have a problem with that, so long as we realise that existing coroutines that are automatically primed when created won't work unmodified with "yield from" (since they would get primed twice - once by the wrapper function and once by the "yield from" expression). To be honest, I see that "auto-priming" behaviour as similar to merging creation of threading.Thread instances with calling t.start() on them - while it is sometimes convenient to do that, making it impossible to separate the creation from the activation the way a @coroutine decorator does actually seems like an undesirable thing to do. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------
Guido van Rossum wrote:
We could do this based on the presence or absence of the send/throw/close attributes: this would be duck typing. Or we could use isinstance(it, types.GeneratorType).
I don't like the idea of switching the entire behaviour based on a blanket generator/non-generator distinction. Failure to duck-type is unpythonic unless there's a very good reason for it, and I don't see any strong reason here. Why shouldn't an iterator be able to emulate a generator by providing all the necessary methods? On the other hand, if we're looking at the presence of methods, what happens if e.g. it has a throw() method but not a send() method? Do we treat it as though the throw() method didn't exist just because it doesn't have the full complement of generator methods? That doesn't seem very pythonic either.
if EXPR produces a file stream object (which has a close() method), it would not consistently be closed: it would be closed if the outer generator was closed before reaching the end, but not if the loop was allowed to run until the end of the file.
I don't think this is a serious problem, for the following reasons: 1. We've already more or less decided that yield-from is not going to address the case of shared subiterators, so if anything else would care about the file being closed unexpectedly, you shouldn't be using yield-from on it. 2. It's well known that you can't rely on automatic closing of files in non-refcounting implementations, so code wanting to ensure the file is closed will need to do so explicitly using a finally clause or something equivalent, which will get triggered by closing the outer generator. -- Greg
I haven't been following this discussion too much (as I would have no time for anything else if I did, it seems), but I think I understand the problem with priming a coroutine, and then trying to use it in yield from, and I may have a solution. I don't understand what it means to 'yield from' a coroutine, but I'll here's my proposed fix: Give all generators/coroutines a 'prime' (or better named) function. This prime function can set some 'is_primed' internal variable so that it never primes more than once. Now, yield from and @coroutine can (and this is the hazy part because I don't know what yield from is really doing under the hood) both use prime(), so yielding from a non-decorated coroutine will have the same effect as yielding from a decorated coroutine. -- Cheers, Leif
Guido van Rossum wrote:
So now let me develop my full thoughts on yield-from. This is unfortunately long, because I want to show some intermediate stages.
I found this extremely helpful. Whatever expansion you finally decide on (pun intended ;-), an explanation like this in the PEP would be nice. tjr
Hi Guido I like the way you are building the description up from the simple case, but I think you are missing a few details along the way. Those details are what has been driving the discussion, so I think it is important to get them handled. I'll comment on each point as I get to it. Guido van Rossum wrote:
I want to name the new exception ReturnFromGenerator to minimize the similarity with GeneratorExit [...]
Fine with me, assuming we can't get rid of it altogether. [Snipped description of close() and __del__(), which I intend to comment on in the other thread]
[Guido]
Oh, and "yield from" competes with @couroutine over when the initial next() call is made, which again suggests the two styles (yield-from and coroutines) are incompatible.
It is a serious problem, because one of the major points of the PEP is that it should be useful for refactoring coroutines. As a matter of fact, I started another thread on this specific issue earlier today which only Nick has so far responded to. I think it is solvable, but requires some more work.
I think that's the thread where I asked you and Nick to stop making more proposals.I a worried that a solution would become too complex, and I want to keep the "naive" interpretation of "yield from EXPR" to be as close as possible to "for x in EXPR: yield x". I think the @coroutine generator (whether built-in or not) or explicit "priming" by a next() call is fine.
I think it is important to be able to use yield-from with a @coroutine, but I'll wait a bit before I do more on that front (except for a few more comments in this mail). There are plenty of other issues to tackle.
So now let me develop my full thoughts on yield-from. This is unfortunately long, because I want to show some intermediate stages. I am using a green font for new code. I am using stages, where each stage provides a better approximation of the desired semantics. Note that each stage *adds* some semantics for corner cases that weren't handled the same way in the previous stage. Each stage proposes an expansion for "RETVAL = yield from EXPR". I am using Py3k syntax. [snip stage 1-3] 4. Stage four adds handling for ReturnFromGenerator, in both places where next() is called:
it = iter(EXPR) try: x = next(it) except StopIteration: RETVAL = e.value except ReturnFromGenerator as e: RETVAL = e.value; break else: while True: yield x try: x = next(it) except StopIteration: RETVAL = None; break except ReturnFromGenerator as e: RETVAL = e.value; break yield x
(There are two cut'n'paste errors here. The first "break" and the second "yield x" shouldn't be there. Just wanted to point it out in case this derivation makes it to the PEP)
5. Stage five shows what should happen if "yield x" above returns a value: it is passed into the subgenerator using send(). I am ignoring for now what happens if it is not a generator; this will be cleared up later. Note that the initial next() call does not change into a send() call, because there is no value to send before before we have yielded:
[snipped code for stage 5] The argument that we have no value to send before we have yielded is wrong. The generator containing the "yield-from" could easily have a value to send (or throw), and if iter(EXPR) returns a coroutine or a non-generator it could easily be ready to accept it. That is the idea behind my attempted fixes to the @coroutine issue.
6. Stage six adds more refined semantics for when "yield x" raises an exception: it is thrown into the generator, except if it is GeneratorExit, in which case we close() the generator and re-raise it (in this case the loop cannot continue so we do not set RETVAL):
[snipped code for stage 6] This is where the fun begins. In an earlier thread we concluded that if the thrown exception is a StopIteration and the *same* StopIteration instance escapes the throw() call, it should be reraised rather than caught and turned into a RETVAL. The reasoning was the following example: def inner(): for i in xrange(10): yield i def outer(): yield from inner() print "if StopIteration is thrown in we shouldn't get here" Which we wanted to be equivalent to: def outer(): for i in xrange(10): yield i print "if StopIteration is thrown in we shouldn't get here" The same argument goes for ReturnFromGenerator, so the expansion at this stage should be more like: it = iter(EXPR) try: x = next(it) except StopIteration: RETVAL = None except ReturnFromGenerator as e: RETVAL = e.value else: while True: try: v = yield x except GeneratorExit: it.close() raise except BaseException as e: try: x = it.throw(e) # IIRC this includes the correct traceback in 3.x so we don't need to use sys.exc_info except StopIteration as r: if r is e: raise RETVAL = None; break except ReturnFromGenerator as r: if r is e: raise RETVAL = r.value; break else: try: x = it.send(v) except StopIteration: RETVAL = None; break except ReturnFromGenerator as e: RETVAL = e.value; break Next issue is that the value returned by it.close() is thrown away by yield-from. Here is a silly example: def inner(): i = 0 while True try: yield except GeneratorExit: return i i += 1 def outer(): try: yield from inner() except GeneratorExit: # nothing I can write here will get me the value returned from inner() Also the trivial: def outer(): return yield from inner() Would swallow the return value as well. I have previously suggested attaching the return value to the (re)raised GeneratorExit, and/or saving the return value on the generator and making close return the value each time it is called. We could also choose to define this as broken behavior and raise a RuntimeError, although it seems a bit strange to have yield-from treat it as an error when close doesn't. Silently having the yield-from construct swallow the returned value is my least favored option.
7. In stage 7 we finally ask ourselves what should happen if it is not a generator (but some other iterator). The best answer seems subtle: send() should degenerator to next(), and all exceptions should simply be re-raised. We can conceptually specify this by simply re-using the for-loop expansion:
it = iter(EXPR) if <it is not a generator>: for x in it: yield next(x) RETVAL = None else: try: x = next(it) except StopIteration: RETVAL = None except ReturnFromGenerator as e: RETVAL = e.value else: while True: try: v = yield x except GeneratorExit: it.close() raise except: try: x = it.throw(*sys.exc_info()) except StopIteration: RETVAL = None; break except ReturnFromGenerator as e: RETVAL = e.value; break else: try: x = it.send(v) except StopIteration: RETVAL = None; break except ReturnFromGenerator as e: RETVAL = e.value; break
Note: I don't mean that we literally should have a separate code path for non-generators. But writing it this way adds the generator test to one place in the spec, which helps understanding why I am choosing these semantics. The entire code of stage 6 degenerates to stage 1 if we make the following substitutions:
it.send(v) -> next(v) it.throw(sys.exc_info()) -> raise it.close() -> pass
(Except for some edge cases if the incoming exception is StopIteration or ReturnFromgenerator, so we'd have to do the test before entering the try/except block around the throw() or send() call.)
We could do this based on the presence or absence of the send/throw/close attributes: this would be duck typing. Or we could use isinstance(it, types.GeneratorType). I'm not sure there are strong arguments for either interpretation. The type check might be a little faster. We could even check for an exact type, since GeneratorType is final. Perhaps the most important consideration is that if EXPR produces a file stream object (which has a close() method), it would not consistently be closed: it would be closed if the outer generator was closed before reaching the end, but not if the loop was allowed to run until the end of the file. So I'm leaning towards only making the generator-specific method calls if it is really a generator.
Like Greg, I am in favor of duck-typing this as closely as possible. My preferred treatment for converting stage 6 to stage 7 goes like this: x = it.close() --> m = getattr(it, 'close', None) if m is not None: x = it.close() else: x = None x = it.send(v) --> if v is None: x = next(it) else: try: m = it.send except AttributeError: m = getattr(it, 'close', None) if m is not None: it.close() # in this case I think it is ok to ignore the return value raise else: x = m(v) x = throw(e) --> m = getattr(it, 'throw', None) if m is not None: x = m() else: m = getattr(it, 'close', None) if m is not None: it.close() # in this case I think it is ok to ignore the return value raise e In this version it is easy enough to wrap the final iterator if you want different behavior. With your version it becomes difficult to replace a generator that is used in a yield-from with an iterator. (You would have to wrap the iterator in a generator that mostly consisted of the expansion from this PEP with the above substitution). I don't think we need to worry about performance at this stage. AFAICT from the patch I was working on, the cost of a few extra checks is negligible compared to the savings you get from using yield-from in the first place. Best regards - Jacob
Jacob Holm wrote:
The argument that we have no value to send before we have yielded is wrong. The generator containing the "yield-from" could easily have a value to send (or throw)
No, Guido is right here. You *can't* send a value (other than None) into a generator that hasn't reached its first yield (try it and you'll get an exception). The first call has to be next().
and if iter(EXPR) returns a coroutine or a non-generator it could easily be ready to accept it.
If it's ready to accept a send, it must have already yielded a value, which has been lost, when it should have been yielded to the caller of the delegating generator.
In an earlier thread we concluded that if the thrown exception is a StopIteration and the *same* StopIteration instance escapes the throw() call, it should be reraised rather than caught and turned into a RETVAL.
That part is right.
Next issue is that the value returned by it.close() is thrown away by yield-from.
Since I don't believe that close() should be expected to return a useful value anyway, that's not a problem. -- Greg
Greg Ewing wrote:
Jacob Holm wrote:
The argument that we have no value to send before we have yielded is wrong. The generator containing the "yield-from" could easily have a value to send (or throw)
No, Guido is right here. You *can't* send a value (other than None) into a generator that hasn't reached its first yield (try it and you'll get an exception). The first call has to be next().
The whole idea of the coroutine pattern is to replace this restriction on the caller with a restriction about the first yield in the coroutine. This would probably be a lot clearer if the coroutine decorator was written as: def coroutine(func): def start(*args,**kwargs): cr = func(*args,**kwargs) v = cr.next() if v is not None: raise RuntimeError('first yield in coroutine was not None') return cr return start The first call *from user code* to a generator decorated with @coroutine *can* be a send() or throw(), and in most cases probably should be.
and if iter(EXPR) returns a coroutine or a non-generator it could easily be ready to accept it.
If it's ready to accept a send, it must have already yielded a value, which has been lost, when it should have been yielded to the caller of the delegating generator.
No, in the coroutine pattern it absolutely should not. The first value yielded by the generator of every coroutine is None and should be thrown away.
Next issue is that the value returned by it.close() is thrown away by yield-from.
Since I don't believe that close() should be expected to return a useful value anyway, that's not a problem.
It is a problem in the sense that it is surprising behavior. If close() doesn't return the value, it should at least raise some exception. I am warming to the idea of reraising the StopIteration if it has a non-None value. This matches Guidos suggestion for how to retrieve the value after a yield-from that was thrown a GeneratorExit. If you insist it is an error to return a value as response to GeneratorExit, raise RuntimeError. But *please* don't just swallow the value. - Jacob
Jacob Holm wrote:
If you insist it is an error to return a value as response to GeneratorExit, raise RuntimeError. But *please* don't just swallow the value.
As I asked in the other thread (but buried in a longer message): why do you see it as OK for close() to throw away every later value a generator may have yielded, but not OK for it to throw away the return value? close() is for finalisation so that generators can have a __del__ method and hence we can allow yield inside try-finally. That's it. Don't break that by trying to turn close() into something it isn't. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------
Jacob Holm wrote:
No, in the coroutine pattern it absolutely should not. The first value yielded by the generator of every coroutine is None and should be thrown away.
That only applies to the *top level* of a coroutine. If you factor some code out of a coroutine and call it using yield-from, the first value yielded by the factored-out code is needed and mustn't be thrown away. So I stand by what I said before. If you're using such a decorator, you only apply it to the top level generator of the coroutine, and you don't call the top level using yield-from. -- Greg
Greg Ewing wrote:
Jacob Holm wrote:
No, in the coroutine pattern it absolutely should not. The first value yielded by the generator of every coroutine is None and should be thrown away.
That only applies to the *top level* of a coroutine. If you factor some code out of a coroutine and call it using yield-from, the first value yielded by the factored-out code is needed and mustn't be thrown away.
One major reason for factoring something out of a coroutine would be if the factored-out code was independently useful as a coroutine. But I cannot actually *make* it a coroutine if I want to call it using yield-from because it is not always at the "top level".
So I stand by what I said before. If you're using such a decorator, you only apply it to the top level generator of the coroutine, and you don't call the top level using yield-from.
So one @coroutine can't call another using yield-from. Why shouldn't it be possible? All we need is a way to avoid the first next() call and substitute some other value. Here is a silly example of two coroutines calling each other using one of the syntax-based ideas I have for handling this. (I don't care about the actual syntax, just about the ability to do this) @coroutine def avg2(): a = yield b = yield return (a+b)/2 @coroutine def avg_diff(): a = yield from avg2() start None # "start EXPR" means use EXPR for first value to yield instead of next() b = yield from avg2() start a # "start EXPR" means use EXPR for first value to yield instead of next() yield b return a-b a = avg2() a.send(41) a.send(43) # raises StopIteration(42) d = avg_diff() d.send(1.0) d.send(2.0) # returns from first yield-from, yields 1.5 as part of starting second yield-from d.send(3.0) d.send(4.0) # returns from second yield-from. yields 3.5 d.next() # returns from avg_diff. raises StopIteration(-2.0) The important things to note here are that both avg2 and avg_diff are independently useful coroutines (yeah ok, not that useful), and that the "natural" value to yield from the "d.send(2.0)" line does not come from calling next() on the subgenerator, but rather from the outer generator. I don't think there is any way to achieve this without some way of substituting the initial next() call in yield-from. Regards - Jacob
Jacob Holm wrote:
One major reason for factoring something out of a coroutine would be if the factored-out code was independently useful as a coroutine.
So provide another entry point for using that part as a top level, or manually apply the wrapper when it's appropriate. I find it extroardinary that people seem to have latched onto David Beazley's idiosyncratic definition of a "coroutine" and decided that it's written on a stone tablet from Mt. Sinai that we must always wrap them in his decorator.
@coroutine def avg_diff(): a = yield from avg2() start None # "start EXPR" means use EXPR for first value to yield instead of next() b = yield from avg2() start a # "start EXPR" means use EXPR for first value to yield instead of next()
I'm going to need a less abstract example to see why you might want to do something like that. -- Greg
Greg Ewing wrote:
Jacob Holm wrote:
One major reason for factoring something out of a coroutine would be if the factored-out code was independently useful as a coroutine.
So provide another entry point for using that part as a top level, or manually apply the wrapper when it's appropriate.
It is inconvenient to have to keep separate versions around, and it doesn't solve the problem. See example below.
I find it extroardinary that people seem to have latched onto David Beazley's idiosyncratic definition of a "coroutine" and decided that it's written on a stone tablet from Mt. Sinai that we must always wrap them in his decorator.
I find it extraordinary that my critical perspective on this issue makes you think I am taking his tutorial as dogma. There is a real issue here IMNSHO, and the @coroutine examples just happens to be the easiest way I can see of explaining it.
@coroutine def avg_diff(): a = yield from avg2() start None # "start EXPR" means use EXPR for first value to yield instead of next() b = yield from avg2() start a # "start EXPR" means use EXPR for first value to yield instead of next()
I'm going to need a less abstract example to see why you might want to do something like that.
Ok, below you will find a modified version of your parser example taken from http://www.cosc.canterbury.ac.nz/greg.ewing/python/yield-from/parser.txt The modification consists of applying the @coroutine decorator to parse_items and parse_elem and changing them to yield a stream of 2-tuples describing how each token sent to the coroutine was interpreted. The expected output is: Feeding: '<foo>' Yielding: ('Open', 'foo') Feeding: 'This' Yielding: ('Data', 'This') Feeding: 'is' Yielding: ('Data', 'is') Feeding: 'a' Yielding: ('Data', 'a') Feeding: '<b>' Yielding: ('Open', 'b') Feeding: 'foo' Yielding: ('Data', 'foo') Feeding: 'file' Yielding: ('Data', 'file') Feeding: '</b>' Yielding: ('Close', 'b') Feeding: 'you' Yielding: ('Data', 'you') Feeding: 'know.' Yielding: ('Data', 'know.') Feeding: '</foo>' Yielding: ('Close', 'foo') [('foo', ['This', 'is', 'a', ('b', ['foo', 'file']), 'you', 'know.'])] I can't see a nice way to get the same sequence of send() calls to yield the same values without the ability to override the way the value for the initial yield in yield-from is computed. Even avoiding the use of @coroutine, I will still need to pass extra arguments to the generator functions to control the initial yield. - Jacob ------------------------------------------------------------------------ # Support code from example at http://www.cosc.canterbury.ac.nz/greg.ewing/python/yield-from/parser.txt import re pat = re.compile(r"(\S+)|(<[^>]*>)") def scanner(text): for m in pat.finditer(text): token = m.group(0) print "Feeding:", repr(token) yield token yield None # to signal EOF text = "<foo> This is a <b> foo file </b> you know. </foo>" token_stream = scanner(text) def is_opening_tag(token): return token.startswith("<") and not token.startswith("</") # Coroutine decorator copied from earlier mail, based on the one in http://dabeaz.com/coroutines/ def coroutine(func): def start(*args, **kwargs): cr = func(*args, **kwargs) v = cr.next() if v is not None: raise RuntimeError('first yield from coroutine was not None') return cr return start # Runner modified from example at http://www.cosc.canterbury.ac.nz/greg.ewing/python/yield-from/parser.txt # to also print the yielded values. def run(): parser = parse_items() # The original forgot to call next() here. That is not necessary in this version since parse_items uses # the @coroutine decorator. try: for m in pat.finditer(text): token = m.group(0) print "Feeding:", repr(token) v = parser.send(token) print "Yielded:", v parser.send(None) # to signal EOF except StopIteration, e: tree = e.args[0] print tree # parse_elem modified from example at http://www.cosc.canterbury.ac.nz/greg.ewing/python/yield-from/parser.txt # to make it a coroutine and to yield a sequence of 2-tuples describing how the recieved data was interpreted. # (Does not yield the final ('Close', <tagname>) because it returns the tree instead). @coroutine def parse_elem(): opening_tag = yield name = opening_tag[1:-1] closing_tag = "</%s>" % name items = yield from parse_items(closing_tag) start ('Open', name) return (name, items) # parse_items modified from example at http://www.cosc.canterbury.ac.nz/greg.ewing/python/yield-from/parser.txt # to make it a coroutine and to yield a sequence of 2-tuples describing how the recieved data was interpreted. @coroutine def parse_items(closing_tag = None): elems = [] token = yield while token != closing_tag: if is_opening_tag(token): subtree = yield from parse_elem() as p start p.send(token) elems.append(subtree) out = ('Close', subtree[0]) else: elems.append(token) out = ('Data', token) token = yield out return elems
Jacob Holm wrote:
So one @coroutine can't call another using yield-from. Why shouldn't it be possible? All we need is a way to avoid the first next() call and substitute some other value.
Here is a silly example of two coroutines calling each other using one of the syntax-based ideas I have for handling this. (I don't care about the actual syntax, just about the ability to do this)
@coroutine def avg2(): a = yield b = yield return (a+b)/2
@coroutine def avg_diff(): a = yield from avg2() start None # "start EXPR" means use EXPR for first value to yield instead of next() b = yield from avg2() start a # "start EXPR" means use EXPR for first value to yield instead of next() yield b return a-b
You can fix this without syntax by changing the way avg2 is written. @coroutine def avg2(start=None): a = yield start b = yield return (a+b)/2 @coroutine def avg_diff(start=None): a = yield from avg2(start) b = yield from avg2(a) yield b return a-b a = avg2() a.send(41) a.send(43) # raises StopIteration(42) d = avg_diff() d.send(1.0) d.send(2.0) # returns from first yield-from, yields 1.5 as part of starting second yield-from d.send(3.0) d.send(4.0) # returns from second yield-from. yields 3.5 d.next() # returns from avg_diff. raises StopIteration(-2.0) So it just becomes a new rule of thumb for coroutines: a yield-from friendly coroutine will accept a "start" argument that defaults to None and is returned from the first yield call. And just like threading.Thread, it will leave the idea of defining the coroutine and starting the coroutine as separate activities. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------
Nick Coghlan wrote:
Jacob Holm wrote:
So one @coroutine can't call another using yield-from. Why shouldn't it be possible? All we need is a way to avoid the first next() call and substitute some other value.
[snip code] You can fix this without syntax by changing the way avg2 is written.
[snip code]
So it just becomes a new rule of thumb for coroutines: a yield-from friendly coroutine will accept a "start" argument that defaults to None and is returned from the first yield call.
That is not quite enough. Your suggested rule of thumb is to replace def example(*args, **kw): ... x = yield # first yield with def example(start=None, *args, **kw): ... x = yield start # first yield That would have been correct if my statement about what was needed wasn't missing a bit. What you actually need to replace with is something like: def example(start, *args, **kw): ... if 'throw' in start: raise start['throw'] # simulate a throw() on first next() elif 'send' in start: x = start['send'] # simulate a send() on first next() else: x = yield start.get('yield') # use specified value for first next() This allows you to set up so the first next() call skips the yield and acts like a send() or a throw() was called. Actually, I think that can be refactored to: def cr_init(start): if 'throw' in start: raise start['throw'] if 'send' in start: return start['send'] return yield start.get('yield') def example(start, *args, **kw): ... x = yield from cr_init(start) Which makes it almost bearable. It is also possible to write a @coroutine decorator that can be used with this, the trick is to make the undecorated function available as an attribute of the wrapper so it is available for use in yield-from. The wrapper can also hide the existence of the start argument from top-level users. def coroutine(func): def start(*args, **kwargs): cr = func({}, *args, **kwargs) v = cr.next() if v is not None: raise RuntimeError('first yield from coroutine was not None') return cr start.raw = func return start Using such a coroutine in yield-from then becomes: # Yield None as first value yield from example.raw({}, *args, **kwargs) # Yield 42 as first value yield from example.raw({'yield':42}, *args, **kwargs) # Skip the first yield and treat the first next() as a send(42) yield from example.raw({'send':42}, *args, **kwargs) # Skip the first yield and treat the first next() as a throw(ValueError(42)) yield from example.raw({'throw':ValueError(42)}, *args, **kwargs) While using it in other contexts is exactly like people are used to. So it turns out a couple of support routines and a simple convention can work around most of the problems with using @coroutines in yield-from. I still think it would be nice if yield-from didn't insist on treating its iterator as if it was new. - Jacob
Jacob Holm wrote:
That would have been correct if my statement about what was needed wasn't missing a bit. What you actually need to replace with is something like:
def example(start, *args, **kw): ... if 'throw' in start: raise start['throw'] # simulate a throw() on first next() elif 'send' in start: x = start['send'] # simulate a send() on first next() else: x = yield start.get('yield') # use specified value for first next()
This elaboration strikes me as completely unecessary, since next() or the equivalent send(None) are the only legitimate ways to start a generator:
def gen(): ... print "Generator started" ... yield ... g = gen() g.next() Generator started
A generator that receives a throw() first thing never executes at all:
g = gen() g.throw(AssertionError) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 1, in gen AssertionError
Similarly, sending a non-None value first thing triggers an exception since there is nowhere for the value to go:
g = gen() g.send(42) Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: can't send non-None value to a just-started generator
For the PEP, I think the solution to this issue is a couple of conventions: 1. Either don't implicitly call next() when creating coroutines or else make that behaviour easy to bypass). This is a bad idea for the same reason that implicitly calling start() on threads is a bad idea: sometimes the user will want to separate definition from activation, and it is a pain when the original author makes that darn near impossible in order to save one line in the normal case. 2. Coroutines intended for use with yield-from should take a "start" argument that is used for the value of their first yield. This allows coroutines to be nested without introducing spurious "None" values into the yield stream. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------
Nick Coghlan wrote:
Jacob Holm wrote:
That would have been correct if my statement about what was needed wasn't missing a bit. What you actually need to replace with is something like:
def example(start, *args, **kw): ... if 'throw' in start: raise start['throw'] # simulate a throw() on first next() elif 'send' in start: x = start['send'] # simulate a send() on first next() else: x = yield start.get('yield') # use specified value for first next()
This elaboration strikes me as completely unecessary, since next() or the equivalent send(None) are the only legitimate ways to start a generator:
Correct, but missing the point. Maybe I explained the "throw" and "send" parts badly. The point is that the following two examples have the same effect: g = example({'send':42}) g.next() g = example({}) g.next() g.send(42) Same effect meaning same last value returned and same internal state. You can't do this by just providing a value to yield on the first next(). The modified parser example I sent to Greg shows that there is a use case for it (although it is written using one of the syntax-based ideas).
For the PEP, I think the solution to this issue is a couple of conventions:
1. Either don't implicitly call next() when creating coroutines or else make that behaviour easy to bypass). This is a bad idea for the same reason that implicitly calling start() on threads is a bad idea: sometimes the user will want to separate definition from activation, and it is a pain when the original author makes that darn near impossible in order to save one line in the normal case.
I don't agree that it is a bad idea to call next automatically. I can see that it is necessary to keep a version around that doesn't do it, but that is because of limitations in yield-from.
2. Coroutines intended for use with yield-from should take a "start" argument that is used for the value of their first yield. This allows coroutines to be nested without introducing spurious "None" values into the yield stream.
For the coroutine writer, it is just as easy to write: x = yield start As it is to write: x = yield from cr_init(start) The difference is that the second version is much more useful in yield-from. Even assuming every relevant object implemented the pattern I suggest, it is still not possible to use yield-from to write something like itertools.dropwhile and have it delegate all send and throw calls correctly. To make that possible, you need exactly the same thing that you need for pre-started coroutines: The ability to replace the next() call made by the yield-from expression with something else. Give me that, and you will also have removed the need for a special pattern for coroutines that should be usable with yield-from. Still-hoping-to-avoid-the-need-for-a-special-pattern-ly yours - Jacob
Jacob Holm wrote:
Even assuming every relevant object implemented the pattern I suggest, it is still not possible to use yield-from to write something like itertools.dropwhile and have it delegate all send and throw calls correctly. To make that possible, you need exactly the same thing that you need for pre-started coroutines: The ability to replace the next() call made by the yield-from expression with something else. Give me that, and you will also have removed the need for a special pattern for coroutines that should be usable with yield-from.
To be clear, I think the best way of handling this is to add a read-only property to generator objects holding the latest value yielded, and let yield-from use that when present instead of calling next(). (This is not a new idea, I am just explaining the consequences as I see them). The property can be cleared when the frame is released, so there should be no issues with that. With that property, the dropwhile example becomes trivial: def dropwhile(predicate, iterable): it = iter(iterable) v = next(it) while predicate(v): v = next(it) return yield from it # Starts by yielding the last value checked, which is v. More interesting (to me) is that the following helpers allow you to call a pre-started generator using yield-from in the 3 special ways I mentioned *without* needing the generator constructor to take any magic arguments. def first_yielding(value, iterable): it = iter(iterable) try: s = yield value except GeneratorExit: it.close() except BaseException as e: it.throw(e) # sets the property so yield-from will use that first else: it.send(s) # sets the property so yield-from will use that first return yield from it def first_sending(value, iterable): it = iter(iterable) it.send(value) # sets the property so yield-from will use that first return yield from it def first_throwing(exc, iterable): it = iter(iterable) it.throw(exc) # sets the property so yield-from will use that first return yield from it # Yield None (first value yielded by a @coroutine) as first value yield from example(*args, **kwargs) # Yield 42 as first value yield from first_yielding(42, example(*args, **kwargs)) # Treat the first next() as a send(42) yield from first_sending(42, example(*args, **kwargs)) # Treat the first next() as a throw(ValueError(42)) yield from first_throwing(ValueError(42), example(*args, **kwargs)) So no new syntax needed, and coroutines are easily callable without the constructor needing extra magic arguments. Also, I am sure the property has other unrelated uses. What's not to like? - Jacob
On Wed, Apr 8, 2009 at 8:04 AM, Jacob Holm <jh@improva.dk> wrote:
Jacob Holm wrote:
Even assuming every relevant object implemented the pattern I suggest, it is still not possible to use yield-from to write something like itertools.dropwhile and have it delegate all send and throw calls correctly. To make that possible, you need exactly the same thing that you need for pre-started coroutines: The ability to replace the next() call made by the yield-from expression with something else. Give me that, and you will also have removed the need for a special pattern for coroutines that should be usable with yield-from.
To be clear, I think the best way of handling this is to add a read-only property to generator objects holding the latest value yielded, and let yield-from use that when present instead of calling next(). (This is not a new idea, I am just explaining the consequences as I see them). The property can be cleared when the frame is released, so there should be no issues with that.
Let me just respond with the recommendation that you stop pushing for features that require storing state on the generator object. Quite apart from implementation issues (which may well be non-existent) I think it's a really scary thing to add any "state" to a generator that isn't contained naturally in its stack frame. If this means that you can't use yield-from for some of your use cases, well, so be it. It has plenty of other use cases. And no, I am not prepared to defend this recommendation. But I feel very strongly about it. So don't challenge me -- it's just going to be a waste of everyone's time to continue this line of thought. -- --Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum wrote:
On Wed, Apr 8, 2009 at 8:04 AM, Jacob Holm <jh@improva.dk> wrote:
Jacob Holm wrote:
Even assuming every relevant object implemented the pattern I suggest, it is still not possible to use yield-from to write something like itertools.dropwhile and have it delegate all send and throw calls correctly. To make that possible, you need exactly the same thing that you need for pre-started coroutines: The ability to replace the next() call made by the yield-from expression with something else. Give me that, and you will also have removed the need for a special pattern for coroutines that should be usable with yield-from.
To be clear, I think the best way of handling this is to add a read-only property to generator objects holding the latest value yielded, and let yield-from use that when present instead of calling next(). (This is not a new idea, I am just explaining the consequences as I see them). The property can be cleared when the frame is released, so there should be no issues with that.
Let me just respond with the recommendation that you stop pushing for features that require storing state on the generator object. Quite apart from implementation issues (which may well be non-existent) I think it's a really scary thing to add any "state" to a generator that isn't contained naturally in its stack frame.
Does storing it as part of the frame object count as "naturally in the stack frame"? Because that is probably the most natural place to put this, implementation-wise. If storing on the frame object is also out, I will have to start thinking about new syntax again. Oh well. I was going to push for saving the final return value from a generator somewhere so that each StopIteration raised by an operation on the closed generator could have the same value as the StopIteration that closed it (or None if it was closed by another exception). If that value can't live on the generator object, I guess that idea is dead. Am I right?
If this means that you can't use yield-from for some of your use cases, well, so be it. It has plenty of other use cases.
That is exactly what I am worried about. I think the number of use cases will be severely limited if we don't have a way to replace the initial next() made by yield-from. This is different from the close() issues we debated earlier, where there is a relatively simple workaround. The full workaround for the "initial next()" issue is big, ugly, and slow. Anyway, I still hope the workaround won't be needed.
And no, I am not prepared to defend this recommendation. But I feel very strongly about it. So don't challenge me -- it's just going to be a waste of everyone's time to continue this line of thought.
No, I am not going to challenge you on this. Once I have your answer to the questions at the beginning of this mail, I will try to adjust my future proposals accordingly. Best regards - Jacob
On Wed, Apr 8, 2009 at 7:52 PM, Jacob Holm <jh@improva.dk> wrote:
Guido van Rossum wrote:
On Wed, Apr 8, 2009 at 8:04 AM, Jacob Holm <jh@improva.dk> wrote:
Jacob Holm wrote: To be clear, I think the best way of handling this is to add a read-only property to generator objects holding the latest value yielded, and let yield-from use that when present instead of calling next(). (This is not a new idea, I am just explaining the consequences as I see them). The property can be cleared when the frame is released, so there should be no issues with that.
Let me just respond with the recommendation that you stop pushing for features that require storing state on the generator object. Quite apart from implementation issues (which may well be non-existent) I think it's a really scary thing to add any "state" to a generator that isn't contained naturally in its stack frame.
Does storing it as part of the frame object count as "naturally in the stack frame"? Because that is probably the most natural place to put this, implementation-wise.
No, that's out too. My point is that I don't want to add *any* state beyond what the user thinks of as the "normal" state in the frame (i.e. local variables, where it is suspended, and the expression stack and try-except stack). Nothing else.
If storing on the frame object is also out, I will have to start thinking about new syntax again. Oh well.
Sorry, no go. New syntax is also out.
I was going to push for saving the final return value from a generator somewhere so that each StopIteration raised by an operation on the closed generator could have the same value as the StopIteration that closed it (or None if it was closed by another exception). If that value can't live on the generator object, I guess that idea is dead. Am I right?
Right.
If this means that you can't use yield-from for some of your use cases, well, so be it. It has plenty of other use cases.
That is exactly what I am worried about. I think the number of use cases will be severely limited if we don't have a way to replace the initial next() made by yield-from.
It doesn't matter if there is only one use case, as long as it is a common one. And we already have that: Greg Ewing's "refactoring". Remember, Dave Beazley in his coroutine tutorial expresses doubts about whether it is really that useful to use generators as coroutines. You are fighting a losing battle here, and it would be better if we stopped short of trying to attain perfection, and instead accepted an imperfect solution that may be sufficient, or may have to be extended in the future. If your hypothetical use cases really become important you can write another PEP.
This is different from the close() issues we debated earlier, where there is a relatively simple workaround. The full workaround for the "initial next()" issue is big, ugly, and slow.
Anyway, I still hope the workaround won't be needed.
Not if you give up now.
And no, I am not prepared to defend this recommendation. But I feel very strongly about it. So don't challenge me -- it's just going to be a waste of everyone's time to continue this line of thought.
No, I am not going to challenge you on this. Once I have your answer to the questions at the beginning of this mail, I will try to adjust my future proposals accordingly.
Great. -- --Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum wrote:
On Wed, Apr 8, 2009 at 7:52 PM, Jacob Holm <jh@improva.dk> wrote:
Does storing it as part of the frame object count as "naturally in the stack frame"? Because that is probably the most natural place to put this, implementation-wise.
No, that's out too. My point is that I don't want to add *any* state beyond what the user thinks of as the "normal" state in the frame (i.e. local variables, where it is suspended, and the expression stack and try-except stack). Nothing else.
That rules out Gregs and my patches as well. They both need extra state on the frame object to be able to implement yield-from in the first place.
If storing on the frame object is also out, I will have to start thinking about new syntax again. Oh well.
Sorry, no go. New syntax is also out.
In that case, I give up. There is no possible way to fix the "initial next()" issue of yield-from without one or the other.
I was going to push for saving the final return value from a generator somewhere so that each StopIteration raised by an operation on the closed generator could have the same value as the StopIteration that closed it (or None if it was closed by another exception). If that value can't live on the generator object, I guess that idea is dead. Am I right?
Right.
Then let me revisit my earlier statement that when close() catches a StopIteration with a non-None value, it should either return it or raise an exception. Since the value is not saved, a second close() will neither be able to return it, nor raise a StopIteration with it. Therefore I now think that raising a RuntimeError in that case is the only right thing to do.
If this means that you can't use yield-from for some of your use cases, well, so be it. It has plenty of other use cases.
That is exactly what I am worried about. I think the number of use cases will be severely limited if we don't have a way to replace the initial next() made by yield-from.
It doesn't matter if there is only one use case, as long as it is a common one. And we already have that: Greg Ewing's "refactoring".
I remain unconvinced that the "initial next()" issue isn't also a problem for that use case, but I am not going to argue about this.
Remember, Dave Beazley in his coroutine tutorial expresses doubts about whether it is really that useful to use generators as coroutines.
You are fighting a losing battle here, and it would be better if we stopped short of trying to attain perfection, and instead accepted an imperfect solution that may be sufficient, or may have to be extended in the future. If your hypothetical use cases really become important you can write another PEP.
Looks to me like the battle is not so much losing as already lost, and probably was from the start. I just wish I had understood that earlier. One final suggestion I have is to make yield-from raise a RuntimeError if used on a generator that already has a frame. That would a) open some optimization possibilities, b) make it clear that the only intended use is with a *fresh* generator or a non-generator iterable, and c) allow us to change our minds about the initial next() later without changing the semantics of working code.
This is different from the close() issues we debated earlier, where there is a relatively simple workaround. The full workaround for the "initial next()" issue is big, ugly, and slow.
Anyway, I still hope the workaround won't be needed.
Not if you give up now.
Well, since I am giving up on fixing the "initial next()" issue in the core, the workaround *will* be needed. Maybe not in the PEP, but certainly as a recipe somewhere. If you don't mind, I will continue the discussion about the details of such a workaround that Nick started a couple of mails back in this thread.
And no, I am not prepared to defend this recommendation. But I feel very strongly about it. So don't challenge me -- it's just going to be a waste of everyone's time to continue this line of thought.
No, I am not going to challenge you on this. Once I have your answer to the questions at the beginning of this mail, I will try to adjust my future proposals accordingly.
Great.
Frustrated-ly yours - Jacob
Jacob Holm wrote:
Then let me revisit my earlier statement that when close() catches a StopIteration with a non-None value, it should either return it or raise an exception. Since the value is not saved, a second close() will neither be able to return it, nor raise a StopIteration with it. Therefore I now think that raising a RuntimeError in that case is the only right thing to do.
Remember, close() is designed to be about finalization. So long as the generator indicates that it has finished (i.e. by reraising GeneratorExit or raising StopIteration with or without a value), the method has done its job. Raising a RuntimeError for a successfully closed generator doesn't make any sense. So if someone wants the return value, they'll need to either use next(), send() or throw() and catch the StopIteration themselves, or else use 'yield from'. That said, creating your own stateful wrapper that preserves the last yield value and the final return value of a generator iterator is also perfectly possible: class CaptureGen(object): """Capture and preserve the last yielded value and the final return value of a generator iterator instance""" NOT_SET = object() def __init__(self, geniter): self.geniter = geniter self._last_yield = self.NOT_SET self._return_value = self.NOT_SET @property def last_yield(self): if self._last_yield is self.NOT_SET: raise RuntimeError("Generator has not yielded") return self._last_yield @property def return_value(self): if self._return_value is self.NOT_SET: raise RuntimeError("Generator has not returned") return self._return_value def _delegate(self, meth, *args): try: val = meth(*args) except StopIteration, ex: if self._return_value is self.NOT_SET: self._return_value = ex.value raise raise StopIteration(self._return_value) self._last_yield = val return val def __next__(self): return self._delegate(self.geniter.next) next = __next__ def send(self, val): return self._delegate(self.geniter.send, val) def throw(self, et, ev=None, tb=None): return self._delegate(self.geniter.throw, et, ev, tb) def close(self): self.geniter.close() return self._return_value Something like that may actually turn out to be useful as the basis for an enhanced coroutine decorator, similar to the way one uses contextlib.contextmanager to turn a generator object into a context manager. The PEP is quite usable for refactoring without it though.
It doesn't matter if there is only one use case, as long as it is a common one. And we already have that: Greg Ewing's "refactoring".
I remain unconvinced that the "initial next()" issue isn't also a problem for that use case, but I am not going to argue about this.
For refactoring, the pattern of passing in a "start" value for use in the first yield expression in the subiterator should be adequate. That's enough to avoid injecting spurious "None" values into the yield sequence. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------
Nick Coghlan wrote:
Jacob Holm wrote:
Then let me revisit my earlier statement that when close() catches a StopIteration with a non-None value, it should either return it or raise an exception. Since the value is not saved, a second close() will neither be able to return it, nor raise a StopIteration with it. Therefore I now think that raising a RuntimeError in that case is the only right thing to do.
Remember, close() is designed to be about finalization. So long as the generator indicates that it has finished (i.e. by reraising GeneratorExit or raising StopIteration with or without a value), the method has done its job. Raising a RuntimeError for a successfully closed generator doesn't make any sense.
Returning a value in response to GeneratorExit is what doesn't make any sense when there is no possible way to access that value later. Raising a RuntimeError in this case will be a clear reminder of this. Think newbie protection if you will.
So if someone wants the return value, they'll need to either use next(), send() or throw() and catch the StopIteration themselves, or else use 'yield from'.
Yes. How is that an argument against making close raise a RuntimeError if it catches a StopIteration with a non-None value?
That said, creating your own stateful wrapper that preserves the last yield value and the final return value of a generator iterator is also perfectly possible:
class CaptureGen(object): """Capture and preserve the last yielded value and the final return value of a generator iterator instance""" NOT_SET = object()
def __init__(self, geniter): self.geniter = geniter self._last_yield = self.NOT_SET self._return_value = self.NOT_SET
@property def last_yield(self): if self._last_yield is self.NOT_SET: raise RuntimeError("Generator has not yielded") return self._last_yield
@property def return_value(self): if self._return_value is self.NOT_SET: raise RuntimeError("Generator has not returned") return self._return_value
def _delegate(self, meth, *args): try: val = meth(*args) except StopIteration, ex: if self._return_value is self.NOT_SET: self._return_value = ex.value raise raise StopIteration(self._return_value) self._last_yield = val return val
def __next__(self): return self._delegate(self.geniter.next) next = __next__
def send(self, val): return self._delegate(self.geniter.send, val)
def throw(self, et, ev=None, tb=None): return self._delegate(self.geniter.throw, et, ev, tb)
def close(self): self.geniter.close() return self._return_value
Something like that may actually turn out to be useful as the basis for an enhanced coroutine decorator, similar to the way one uses contextlib.contextmanager to turn a generator object into a context manager. The PEP is quite usable for refactoring without it though.
I know it is possible to create a complete workaround. The problem is that any complete workaround will break the chain of yield-from calls, causing a massive overhead on next(), send(), and throw() compared to a partial workaround that doesn't break the yield-from chain.
For refactoring, the pattern of passing in a "start" value for use in the first yield expression in the subiterator should be adequate. That's enough to avoid injecting spurious "None" values into the yield sequence.
If you define "refactoring" narrowly enough, you are probably right. Otherwise it depends on how you want the new subgenerator to work. If you want it to be a @coroutine (with or without the decorator) there are good reasons for wanting to throw away the value from the first next() and provide an initial value to send() or throw() immediately after that, so the first value yielded by the yield-from becomes the result of the send() or throw(). Taking just a simple "start" value in the constructor and making the first yield an "x = yield start" doesn't support that use. Taking a compound "start" value in the constructor and making the first yield an "x = yield from cr_init(start)" does, by skipping the first yield when neccessary and simulating the send() or throw(). - Jacob
On 4/9/09, Nick Coghlan <ncoghlan@gmail.com> wrote:
Jacob Holm wrote:
Then let me revisit my earlier statement that when close() catches a StopIteration with a non-None value, it should either return it or raise an exception.
This implies that the value is always important; I write plenty of functions that don't bother to return anything, and expect that would still be common if I were factoring out loops.
Since the value is not saved, a second close() will neither be able to return it, nor raise a StopIteration with it.
If close is called more than once, that suggests at least one of those calls is just freeing resources, and doesn't need a value.
It doesn't matter if there is only one use case, as long as it is a common one. And we already have that: Greg Ewing's "refactoring".
I remain unconvinced that the "initial next()" issue isn't also a problem for that use case, but I am not going to argue about this.
That does suggest that yield-from *should* accept pre-started generators, if only because the previous line (or a decorator) may have primed it.
For refactoring, the pattern of passing in a "start" value for use in the first yield expression in the subiterator should be adequate.
Only if you know what the first value should be. If you're using a sentinel that you plan to discard, then it would be way simpler to just prime the generator before the yield-from. -jJ
Jim Jewett wrote:
On 4/9/09, Nick Coghlan <ncoghlan@gmail.com> wrote:
Jacob Holm wrote:
Then let me revisit my earlier statement that when close() catches a StopIteration with a non-None value, it should either return it or raise an exception.
This implies that the value is always important; I write plenty of functions that don't bother to return anything, and expect that would still be common if I were factoring out loops.
If you return None (or no value) you won't get an exception with my proposal. Only if you handle a GeneratorExit by returning a non-None value. Since that value is not going to be visible to any other code, it doesn't make sense to return it. I am suggesting that you should get a RuntimeError so you become aware that returning the value doesn't make sense.
Since the value is not saved, a second close() will neither be able to return it, nor raise a StopIteration with it.
If close is called more than once, that suggests at least one of those calls is just freeing resources, and doesn't need a value.
It doesn't matter if there is only one use case, as long as it is a common one. And we already have that: Greg Ewing's "refactoring".
I remain unconvinced that the "initial next()" issue isn't also a problem for that use case, but I am not going to argue about this.
That does suggest that yield-from *should* accept pre-started generators, if only because the previous line (or a decorator) may have primed it.
That was my argument, but since there is no sane way of handling pre-primed generators without extending the PEP in a direction that Guido has forbidden, I suggest raising a RuntimeError in this case instead. That allows us to add the necessary features later if the need is recognized.
For refactoring, the pattern of passing in a "start" value for use in the first yield expression in the subiterator should be adequate.
Only if you know what the first value should be. If you're using a sentinel that you plan to discard, then it would be way simpler to just prime the generator before the yield-from.
Yeah, but yield from with pre-primed generators just ain't gonna happen. :( - Jacob
On 4/9/09, Jacob Holm <jh@improva.dk> wrote:
Jim Jewett wrote:
On 4/9/09, Nick Coghlan <ncoghlan@gmail.com> wrote:
Jacob Holm wrote:
... when close() catches a StopIteration with a non-None value, it should either return it or raise
This implies that the value is always important; ...
If you return None (or no value) you won't get an exception
Think of all the C functions that return a success status, or the number of bytes read/written. Checking the return code is good, but not checking it isn't always worth an Exception.
... does suggest that yield-from *should* accept pre-started generators, if only because the previous line (or a decorator) may have primed it.
That was my argument, but since there is no sane way of handling pre-primed generators without extending the PEP in a direction that Guido has forbidden,
Huh? All you need to do is to not care whether the generator is fresh or not (let alone primed vs half-used). If it won't be primed, *and* you can't afford to send back an extra junk yield, then you need to prime it yourself. That can be awkward, but so are all the syntax extensions that boil down to "and implicitly call next for me once". (And the "oops, just this once, don't prime it after all" extensions are even worse.) -jJ
Jim Jewett wrote:
On 4/9/09, Jacob Holm <jh@improva.dk> wrote:
... does suggest that yield-from *should* accept pre-started generators, if only because the previous line (or a decorator) may have primed it.
That was my argument, but since there is no sane way of handling pre-primed generators without extending the PEP in a direction that Guido has forbidden,
Huh? All you need to do is to not care whether the generator is fresh or not (let alone primed vs half-used).
By no sane way I mean that there is no way to avoid that the first call made by the yield-from construct is a next(). If the pre-primed generator is expecting a send() at that point you are screwed.
If it won't be primed, *and* you can't afford to send back an extra junk yield, then you need to prime it yourself.
And then the yield-from is still starting out by calling next() so you are still screwed.
That can be awkward, but so are all the syntax extensions that boil down to "and implicitly call next for me once". (And the "oops, just this once, don't prime it after all" extensions are even worse.)
The only syntax extension that was really interesting was intended to *avoid* this call to next() by providing a different value to yield the first time. Avoiding the call to next() allows you to create all kinds of wrappers that manipulate the start of the sequence, and covers all other cases I had considered syntax for. Anyway, this syntax discussion is moot since Guido has already ruled that we will have no syntax for this. - Jacob
Jim Jewett wrote:
That does suggest that yield-from *should* accept pre-started generators, if only because the previous line (or a decorator) may have primed it.
If the previous line has primed it, then it's not a fresh iterator, so all bets are off. Same if a decorator has primed it in such a way that it doesn't behave like a fresh iterator. -- Greg
Jacob Holm wrote:
That rules out Gregs and my patches as well. They both need extra state on the frame object to be able to implement yield-from in the first place.
But that state is obviously necessary in order to support yield-from, and it goes away as soon as the yield-from itself finishes. Your proposals add non-obvious extra state that persists longer than normally expected, to support obscure features that will rarely be used.
One final suggestion I have is to make yield-from raise a RuntimeError if used on a generator that already has a frame. That would ... b) make it clear that the only intended use is with a *fresh* generator or a non-generator iterable,
But there's no way of detecting a violation of that rule for a non-generator iterator, and I want to avoid having special cases for generators, since it goes against duck typing. -- Greg
On Thu, Apr 9, 2009 at 5:35 PM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
Jacob Holm wrote:
That rules out Gregs and my patches as well. They both need extra state on the frame object to be able to implement yield-from in the first place.
But that state is obviously necessary in order to support yield-from, and it goes away as soon as the yield-from itself finishes.
Another way to look at this is, "RETVAL = yield from EXPR" has an expansion into source code where all that state is kept as either a local variable or via the position in the code, and that's how we define the semantics. I don't believe Jacob's proposal (the one that doesn't require new syntax) works this way.
Your proposals add non-obvious extra state that persists longer than normally expected, to support obscure features that will rarely be used.
One final suggestion I have is to make yield-from raise a RuntimeError if used on a generator that already has a frame. That would ...
b) make it clear that the only intended
use is with a *fresh* generator or a non-generator iterable,
But there's no way of detecting a violation of that rule for a non-generator iterator, and I want to avoid having special cases for generators, since it goes against duck typing.
It also goes against the "for x in EXPR: yield x" expansion, which I would like to maintain as an anchor point for the semantics. To be precise: when the .send(), .throw() and .close() methods of the outer generator aren't used, *or* when iter(EXPR) returns a non-generator iterator, these semantics should be maintained precisely, and the extended semantics if .send()/.throw()/.close() are used on the outer generator and iter(EXPR) is a generator should be natural extensions of this, without semantic discontinuities. PS. On Jacob's complaint that he didn't realize earlier that his proposal was dead on arrival: I didn't either. It took me all this time to translate my gut feelings into rules. -- --Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum wrote:
On Thu, Apr 9, 2009 at 5:35 PM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
Jacob Holm wrote:
That rules out Gregs and my patches as well. They both need extra state on the frame object to be able to implement yield-from in the first place.
But that state is obviously necessary in order to support yield-from, and it goes away as soon as the yield-from itself finishes.
Another way to look at this is, "RETVAL = yield from EXPR" has an expansion into source code where all that state is kept as either a local variable or via the position in the code, and that's how we define the semantics. I don't believe Jacob's proposal (the one that doesn't require new syntax) works this way.
My proposal was to extend the lifetime of the yielded value until the yield expression returned, and to make that value available on the frame. The "keep the value alive" part can easily be written as an expansion based on the existing yield semantics, and the "make it available on the frame" is similar to the need that yield-from has. That doesn't look all that different to me, but I'll let you be the judge of that. - Jacob
Jacob Holm wrote:
I think the best way of handling this is to add a read-only property to generator objects holding the latest value yielded... The property can be cleared when the frame is released, so there should be no issues with that.
It will still keep the value alive longer than it would be otherwise. Some people might take issue with that.
def dropwhile(predicate, iterable): it = iter(iterable) v = next(it) while predicate(v): v = next(it) return yield from it # Starts by yielding the last value checked, which is v.
In my view this constitutes a shared iterator, and is therefore outside the scope of yield-from. I also don't think this generalizes well enough to be worth going out of our way to support. It only works because you're making a "tail call" to the iterator, which is a rather special case. Most itertools-style functions won't have that property.
What's not to like?
The fact that yield-from and/or generator behaviour is being complexified to support things that are outside the scope of my proposal. This is why I want to keep focused on refactoring, to prevent this kind of feature creep. -- Greg
Jacob Holm wrote:
You can't do this by just providing a value to yield on the first next().
I don't see how providing an initial yield value directly to the yield-from expression can give you any greater functionality, though. Can you post a complete example of that so I don't have to paste code from several messages together?
The modified parser example I sent to Greg shows that there is a use case for it
FWIW, that example doesn't fit David Beazley's definition of a coroutine, since it uses yields to both send and receive values. He doesn't think that's a sane thing to do.
I don't agree that it is a bad idea to call next automatically. I can see that it is necessary to keep a version around that doesn't do it, but that is because of limitations in yield-from.
An alternative viewpoint would be the idea that a coroutine should always start itself automatically is too simplistic. -- Greg
Greg Ewing wrote:
An alternative viewpoint would be the idea that a coroutine should always start itself automatically is too simplistic.
That's the angle I've been taking. I can see why it can be convenient to start a coroutine automatically, but only in the same way that a thread creation function that also starts the thread for you can be convenient. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------
Jim Jewett wrote:
On 4/3/09, Nick Coghlan <ncoghlan@gmail.com> wrote:
Greg tried to clarify this a bit already, but I think Jacob's averager example is an interesting case where it makes sense to both yield multiple times and also "return a value".
def averager(start=0): # averager that maintains a running average # and returns the final average when done count = 0 exc = None sum = start while 1: avg = sum / count try: val = yield avg except CalcAverage: return finally avg sum += val count += 1
It looks to me like it returns (or yields) the running average either way.
I see a reason to send in a sentinel value, saying "Don't update the average, just tell me the current value."
I don't see why that sentinel has to terminate the generator, nor do I see why that final average has to be returned rather than yielded.
It doesn't *have* to do anything - you could set it up that way if you wanted to. However, when the running average is only yielded, then the location providing the numbers (i.e. the top level code in my example) is also the only location which can receive the running average. That's why anyone using coroutines now *has* to have a top-level scheduler to handle the "coroutine stack" by knowing which coroutines are calling each other and detecting when one has "returned" (i.e. yielded a special sentinel value that the scheduler recognises) so the return value can be passed back to the calling coroutine. I hoped to show the advantage of the separate return value with the difference calculating coroutine: in that example, having the separate return value makes it easy to identify the values which need to be returned to a location *other* than the source of the numbers being averaged. The example goes through some distinct stages: - top level code sending numbers to first averager via send() and receiving running averages back via yield - top level code telling first averager to finish up (via either throw() or send() depending on implementation), first averager returning final average value to the difference calculator - top level code sending numbers to second averager and receiving running averages back - top level code telling second averager to finish up, second averager returning final average value to the difference calculator, difference calculator returning difference between the two averages to the top level code That is, yield, send() and throw() involve communication between the currently active subcoroutine and the client of the whole coroutine. They bypass the current stack in the coroutine itself. The return values, on the other hand, *do* involve unwinding the coroutine stack, just like they do with normal function calls. I'll have another go, this time comparing the "normal" calls to the coroutine version: def average(seq, start=0): if seq: return sum(seq, start) / len(seq) return start def average_diff(seq1, seq2): avg1 = average(seq1) avg2 = average(seq2) return avg2 - avg1 OK, simple and straightforward. The idea of 'yield from' is to allow "average_diff" to be turned into a coroutine that receives the values to be averaged one by one *without* having to inline the actual average calculation. I'll also go back to Jacob's original averaging example that *doesn't* yield a running average - it is more inline with examples where the final calculation is expensive, so you won't do it until you're told you have all the data. What might that look like as 'yield from' style coroutines? class CalcAverage(Exception): pass def average_cr(start=0): count = 0 exc = None sum = start while 1: try: # The yield surrenders control to the code # that is supplying the numbers to be # averaged val = yield except CalcAverage: # The return finally passes the final # average back to the code that was # asking for the average (which is NOT # necessarily the same code that was # supplying the numbers if count: return finally sum / count return finally start sum += val count += 1 # Note how similar this is to the normal version above def average_diff_cr(start): avg1 = yield from average_cr(start, seq1) avg2 = yield from average_cr(start, seq2) return avg2 - avg1 Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------
-1 on adding more methods to generators. +1 on adding this as a recipe to the docs. On Fri, Apr 3, 2009 at 6:13 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Jim Jewett wrote:
The times I did remember that (even) the expression form looped, I was still boggled that it would return something other than None after it was exhausted. Greg's answer was that it was for threading, and the final return was the real value. This seems like a different category of generator, but I could get my head around it -- so long as I forgot that the yield itself was returning anything useful.
Greg tried to clarify this a bit already, but I think Jacob's averager example is an interesting case where it makes sense to both yield multiple times and also "return a value".
While it is just a toy example, I believe it does a great job of illustrating the control flow expectations. (Writing this email has certainly clarified a lot of things about the PEP in my *own* mind).
The following reworking of Jacob's example assumes a couple of things that differ from the current PEP:
- the particular colour my bikeshed is painted when it comes to returning values from a generator is "return finally" (the idea being to emphasise that this represents a special "final" value for the generator that happens only after all of the normal yields are done).
- rather than trying to change the meaning of GeneratorExit and close(), 3 new generator methods would be added: next_return(), send_return() and throw_return(). The new methods have the same signatures as their existing counterparts, but if the generator raises GeneratorReturn, they trap it and return the associated value instead. Like close(), they complain with a RuntimeError if the generator doesn't finish. For example:
def throw_return(self, *exc_info): try: self.throw(*exc_info) raise RuntimeError("Generator did not terminate") except GeneratorReturn as gr: return gr.value
(Note that I've also removed the 'yield raise' idea from the example - if next() or send() triggers termination of the generator with an exception other than StopIteration, then that exception is already propagated into the calling scope by the existing generator machinery. I realise Jacob was trying to make it possible to "yield an exception" without terminating the coroutine, but that idea is well beyond the scope of the PEP)
You then get:
class CalcAverage(Exception): pass
def averager(start=0): # averager that maintains a running average # and returns the final average when done count = 0 exc = None sum = start while 1: avg = sum / count try: val = yield avg except CalcAverage: return finally avg sum += val count += 1
avg = averager() avg.next() # start coroutine avg.send(1.0) # yields 1.0 avg.send(2.0) # yields 1.5 print avg.throw_return(CalcAverage) # prints 1.5
Now, suppose I want to write another toy coroutine that calculates the averages of two sequences and then returns the difference:
def average_diff(start=0): avg1 = yield from averager(start) avg2 = yield from averager(start) return finally avg2 - avg1
diff = average_diff() diff.next() # start coroutine # yields 0.0 avg.send(1.0) # yields 1.0 avg.send(2.0) # yields 1.5 diff.throw(CalcAverage) # Starts calculation of second average # yields 0.0 diff.send(2.0) # yields 2.0 diff.send(3.0) # yields 2.5 print diff.throw_return(CalcAverage) # Prints 1.0 (from "2.5 - 1.5")
The same example could be rewritten to use "None" as a sentinel value instead of throwing in an exception (average_diff doesn't change, so I haven't rewritten that part):
def averager(start=0): count = 0 exc = None sum = start while 1: avg = sum / count val = yield avg if val is None: return finally avg sum += val count += 1
# yielded values are same as the throw_return() approach diff = average_diff() diff.next() # start coroutine diff.send(1.0) diff.send(2.0) diff.send(None) # Starts calculation of second average diff.send(2.0) diff.send(3.0) print diff.send_return(None) # Prints 1.0 (from "2.5 - 1.5")
Notice how the coroutines in this example can be thought of as simple state machines that the calling code needs to know how to drive. That state sequence is as much a part of the coroutine's signature as are the arguments to the constructor and the final return value.
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia --------------------------------------------------------------- _______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas
-- --Guido van Rossum (home page: http://www.python.org/~guido/)
participants (7)
-
Greg Ewing -
Guido van Rossum -
Jacob Holm -
Jim Jewett -
Leif Walsh -
Nick Coghlan -
Terry Reedy