Re: [Python-Dev] PEP 380 (yield from a subgenerator) comments
At 12:27 PM 3/26/2009 -0700, Guido van Rossum wrote:
There is some clear low-hanging fruit for Greg's proposal where no trampoline or helpers are needed -- but where currently refactoring complex code containing many yield statements is cumbersome due to the nee to write each "subroutine" call as "for x in subroutine(): yield x" -- being able to replace this with "yield from subroutine()" is a conceptual advantage to me that is not proportional to the number of characters saved.
Right - I don't object to the concept of "yield from" -- I'm -0 on that. What I don't like is the confusion of adding "return values" to generators, at least using the 'return' statement.
On Thu, Mar 26, 2009 at 4:19 PM, P.J. Eby
At 12:27 PM 3/26/2009 -0700, Guido van Rossum wrote:
There is some clear low-hanging fruit for Greg's proposal where no trampoline or helpers are needed -- but where currently refactoring complex code containing many yield statements is cumbersome due to the nee to write each "subroutine" call as "for x in subroutine(): yield x" -- being able to replace this with "yield from subroutine()" is a conceptual advantage to me that is not proportional to the number of characters saved.
Right - I don't object to the concept of "yield from" -- I'm -0 on that. What I don't like is the confusion of adding "return values" to generators, at least using the 'return' statement.
Well, after thinking about it some more I think I am going to have to take an executive decision and override your opposition. :-) I wasn't keen on this myself at the outset, but it's growing on me, and I am finding Greg's motivation for wanting to do it this way pretty strong. I'm +1 on "yield from" and +0 on return values in generators. That +0 could turn into a +1 if there was a way to flag this as an error (at runtime), at least if the return is actually executed: def g(): yield 42 return 43 for x in g(): print x # probably expected to print 42 and then 43 Perhaps the exception used in this case could be a different exception than StopIteration? Regular iteration could either just pass this exception through or explicitly check for it (a single pointer comparison could usually suffice), depending on whether it would be a subclass of StopIteration. Greg, what do you think? (There could be good reasons not to complexificate it this way too.) -- --Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum wrote:
That +0 could turn into a +1 if there was a way to flag this as an error (at runtime), at least if the return is actually executed:
def g(): yield 42 return 43
for x in g(): print x # probably expected to print 42 and then 43
Perhaps the exception used in this case could be a different exception than StopIteration?
Would checking that the value is None be sufficient? Or do you want to distinguish between 'return' and 'return None'? That would feel rather strange to me. I'm inclined to regard this as an unnecessary complication. People are already trained to think of 'return' and 'yield' as quite different things. I don't know why they would suddenly start using 'return' when they mean 'yield'. -- Greg
Greg Ewing wrote:
Guido van Rossum wrote:
That +0 could turn into a +1 if there was a way to flag this as an error (at runtime), at least if the return is actually executed:
def g(): yield 42 return 43
for x in g(): print x # probably expected to print 42 and then 43
Perhaps the exception used in this case could be a different exception than StopIteration?
Would checking that the value is None be sufficient? Or do you want to distinguish between 'return' and 'return None'? That would feel rather strange to me.
I'm inclined to regard this as an unnecessary complication. People are already trained to think of 'return' and 'yield' as quite different things. I don't know why they would suddenly start using 'return' when they mean 'yield'.
Probably because they just started to use Python and they just don't understand WT* you are talking about. Reviewing this thread reminds me of the story about the Princeton math professor who started his lecture by writing a complex three-line formula on the blackboard and saying "From this it is obvious that ..." and then proceeded to gaze at the blackboard in silence for three minutes. He then left the lecture theater, somewhat to his students' consternation, only to reappear with a smile on his face in ten minutes to say "Yes, it *is* obvious that ...". I am a *bit* concerned, without really being able to put my finger on it, that the "yield from" expression's value comes from inside (the "return" from the nested generator) while the "yield from" expression's value comes from "outside" (the value passed to a .send() method call). Please forgive this bear of very little brain. I suspect the documentation will need to make this asymmetry more obvious still. regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 Holden Web LLC http://www.holdenweb.com/ Want to know? Come to PyCon - soon! http://us.pycon.org/
Steve Holden wrote:
I am a *bit* concerned, without really being able to put my finger on it, that the "yield from" expression's value comes from inside (the "return" from the nested generator) while the "yield from" expression's value comes from "outside" (the value passed to a .send() method call).
The send() calls don't go to the yield-from expression, they go to the yields inside the subgenerator. Similarly, next() calls get their values from the yields inside the subgenerator. The only time the yield-from expression itself receives a value is right at the very end when the subgenerator terminates, and that's not a yielding operation, it's a returning operation. Yield-from is not a kind of yield. It's a kind of function call. -- Greg
On Fri, Mar 27, 2009 at 12:53 AM, Greg Ewing
Guido van Rossum wrote:
That +0 could turn into a +1 if there was a way to flag this as an error (at runtime), at least if the return is actually executed:
def g(): yield 42 return 43
for x in g(): print x # probably expected to print 42 and then 43
Perhaps the exception used in this case could be a different exception than StopIteration?
Would checking that the value is None be sufficient? Or do you want to distinguish between 'return' and 'return None'? That would feel rather strange to me.
Well currently there *is* a difference, because "return None" is a (phase 2) SyntaxError in a generator, while "return" isn't. But I don't really care if you cannot distinguish these two cases.
I'm inclined to regard this as an unnecessary complication. People are already trained to think of 'return' and 'yield' as quite different things. I don't know why they would suddenly start using 'return' when they mean 'yield'.
Because coroutines are mind-bendingly hard, and it's easy to confuse yield (pass a value back while keeping the stack frame) and return (pass a value back dropping the stack frame). The syntactic prohibition on "return <expr>" in generators currently helps people to be consistent. I'd like to keep this small crutch around so that people who are writing vanilla (== meant for iteration) generators are kept from accidentally introducing hard-to-debug problems. Perhaps the crux is that *if* you accidentally use "return <value>" in a vanilla generator expecting the value to show up somewhere, you are probably enough of a newbie that debugging this will be quite hard. I'd like not to have such a newbie trap lying around. -- --Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum wrote:
Perhaps the crux is that *if* you accidentally use "return <value>" in a vanilla generator expecting the value to show up somewhere, you are probably enough of a newbie that debugging this will be quite hard. I'd like not to have such a newbie trap lying around.
Okay, so would you be happy if the for-loop were to raise an exception if it gets a StopIteration whose value is not None? -- Greg
At 12:53 PM 3/28/2009 +1200, Greg Ewing wrote:
Guido van Rossum wrote:
Perhaps the crux is that *if* you accidentally use "return <value>" in a vanilla generator expecting the value to show up somewhere, you are probably enough of a newbie that debugging this will be quite hard. I'd like not to have such a newbie trap lying around.
Okay, so would you be happy if the for-loop were to raise an exception if it gets a StopIteration whose value is not None?
Wouldn't it have to be more than just for-loops? What about list(), map(), ...? It seems a lot simpler to just make it use a different exception, as nothing else has to change for that to work correctly; the new construct can just catch it, and everywhere else it's an error.
On Fri, Mar 27, 2009 at 8:45 PM, P.J. Eby
At 12:53 PM 3/28/2009 +1200, Greg Ewing wrote:
Guido van Rossum wrote:
Perhaps the crux is that *if* you accidentally use "return <value>" in a vanilla generator expecting the value to show up somewhere, you are probably enough of a newbie that debugging this will be quite hard. I'd like not to have such a newbie trap lying around.
Okay, so would you be happy if the for-loop were to raise an exception if it gets a StopIteration whose value is not None?
Wouldn't it have to be more than just for-loops? What about list(), map(), ...?
It seems a lot simpler to just make it use a different exception, as nothing else has to change for that to work correctly; the new construct can just catch it, and everywhere else it's an error.
Exactly. Making StopIteration() with a non-None value invalid is not backwards compatible. The new exception could either be a designated (built-in) subclass of StopIteration, or not; I haven't thought enough about which choice would be more useful or easier. I think in either case a check in PyIter_Next() would cover most cases -- though there are probable a few other places in the interpreter where tp_next is called that should also have this check added. -- --Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum wrote:
The new exception could either be a designated (built-in) subclass of StopIteration, or not;
I think it would have to not be; otherwise any existing code that catches StopIteration would catch the new exception as well without complaint. Using a different exception raises another question. Would you check whether the return value is None and raise an ordinary StopIteration in that case? Or would return with a value always raise the new exception? If the latter, then 'return' and 'return None' would no longer be equivalent in all cases, which would be rather strange.
I think in either case a check in PyIter_Next() would cover most cases
If that's acceptable, then the check might as well be for None as the StopIteration value, and there's no need for a new exception. -- Greg
On Sat, Mar 28, 2009 at 1:36 AM, Greg Ewing
Guido van Rossum wrote:
The new exception could either be a designated (built-in) subclass of StopIteration, or not;
I think it would have to not be; otherwise any existing code that catches StopIteration would catch the new exception as well without complaint.
OK.
Using a different exception raises another question. Would you check whether the return value is None and raise an ordinary StopIteration in that case? Or would return with a value always raise the new exception? I'm not sure it matters much, but let's sat the latter.
If the latter, then 'return' and 'return None' would no longer be equivalent in all cases, which would be rather strange.
They already aren't in generators, 'return' is allowed but 'return None' isn't.
I think in either case a check in PyIter_Next() would cover most cases
If that's acceptable, then the check might as well be for None as the StopIteration value, and there's no need for a new exception.
I don't understand this. -- --Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum wrote:
I think in either case a check in PyIter_Next() would cover most cases
If that's acceptable, then the check might as well be for None as the StopIteration value, and there's no need for a new exception.
I don't understand this.
Maybe I misunderstood what you were saying. What check were you suggesting to perform in PyIter_Next? -- Greg
On Sat, Mar 28, 2009 at 4:37 PM, Greg Ewing
Guido van Rossum wrote:
I think in either case a check in PyIter_Next() would cover most cases
If that's acceptable, then the check might as well be for None as the StopIteration value, and there's no need for a new exception.
I don't understand this.
Maybe I misunderstood what you were saying. What check were you suggesting to perform in PyIter_Next?
I now realize what you were saying. You said effectively "the check added to PyIter_Next() might as well check whether the value attribute of the StopIteration is not None", but due to PyCon tiredness last night my brain's English parser didn't come up with any meaningful parse of what you wrote. But it's been answered already -- we can't change the meaning of StopIteration() with a value unequal to None, so it has to be a separate exception, and it should not derive from StopIteration. -- --Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum wrote:
But it's been answered already -- we can't change the meaning of StopIteration() with a value unequal to None, so it has to be a separate exception, and it should not derive from StopIteration.
How about having StopIteration be a subclass of the new exception? Then things that want to get generator return values only have one exception to catch, and things that only know about StopIteration will fail to catch the new exception. -- Greg
Greg Ewing wrote:
Guido van Rossum wrote:
But it's been answered already -- we can't change the meaning of StopIteration() with a value unequal to None, so it has to be a separate exception, and it should not derive from StopIteration.
How about having StopIteration be a subclass of the new exception? Then things that want to get generator return values only have one exception to catch, and things that only know about StopIteration will fail to catch the new exception.
I actually like that, because it means that "coroutine-y" code could pretty much ignore the existence of StopIteration (despite it existing first). It would also mean that it wouldn't matter if "return" and "return None" both raised StopIteration, since they would still be intercepted by GeneratorReturn exception handlers. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------
On Sat, Mar 28, 2009 at 8:14 PM, Nick Coghlan
Greg Ewing wrote:
Guido van Rossum wrote:
But it's been answered already -- we can't change the meaning of StopIteration() with a value unequal to None, so it has to be a separate exception, and it should not derive from StopIteration.
How about having StopIteration be a subclass of the new exception? Then things that want to get generator return values only have one exception to catch, and things that only know about StopIteration will fail to catch the new exception.
I actually like that, because it means that "coroutine-y" code could pretty much ignore the existence of StopIteration (despite it existing first).
Okay.
It would also mean that it wouldn't matter if "return" and "return None" both raised StopIteration, since they would still be intercepted by GeneratorReturn exception handlers.
Well I already didn't care about that. :-) -- --Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum wrote: [...] (There could be good reasons not to
complexificate it this way too.)
You've been 'Mercanized. This is the most worstest English I have ever seen you write. regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 Holden Web LLC http://www.holdenweb.com/ Want to know? Come to PyCon - soon! http://us.pycon.org/
participants (5)
-
Greg Ewing -
Guido van Rossum -
Nick Coghlan -
P.J. Eby -
Steve Holden