[Python-ideas] Revised**5 PEP on yield-from

[Jacob Holm]

Greg Ewing wrote:
> Jacob Holm wrote:
>> After doing
>> "yield from R" in generator A, you can go ahead and do "yield from R" 
>> in generator B as well. If R is also using "yield from" you will have 
>> the following situation:
>>
>> A
>>  \
>>    R --- (whatever R is waiting for)
>>  /
>> B
>
> Unless you're doing something extremely unusual, this situation
> wouldn't arise. The yield-from statement is intended to run the
> iterator to exhaustion, and normally you'd create a fresh iterator
> for each yield-from that you want to do. So A and B would really
> be yielding from different iterators, even if they were both
> iterating over the same underlying object.
Unusual, yes.  Extremely?  I'm not sure.  If you/we allow this, someone 
will find a use for it.
>
> If you did try to share iterators between yield-froms like that,
> you would have to arrange things so that at least all but one
> of them broke out of the loop early, otherwise something is
> going to get an exception due to trying to resume an exhausted
> iterator.
Did you read the spelled-out version at the bottom?  No need to "break 
out" of anything. That happens automatically because of the "yield 
from".  Just a few well-placed calls to next...

>
> But in any case, I think you can still model this as two separate
> stacks, with R appearing in both stacks: [A, R] and [B, R]. Whichever
> one of them finishes yielding from R first pops it from its stack,
> and when the other one tries to resume R it gets an exception. Either
> that or it breaks out of its yield-from early and discards its
> version of R.
I am not worried about R running out, each of A and B would find out 
about that next time they tried to get a value. I *am* worried about R 
doing a yield-from to X (the xrange in this example) which then needs to 
appear in both stacks to get the expected behavior from the PEP.

>
> > As long as that scenario is possible I can construct
>> an example where treating it as a simple stack will either do the 
>> wrong thing, or do the right thing but slower than a standard "for v 
>> in it: yield v".
>
> That depends on what you think the "right thing" is. If you
> think that somehow A needs to notice that B has finished yielding
> from R and gracefully stop doing so itself, then that's not something
> I intended and it's not the way the current prototype implementation
> would behave.
The right thing is whatever the PEP specifies :)  You are the author, so 
you get to decide...

I am saying that what the PEP currently specifies is not quite so simple 
to speed up as you and Arnaud seem to think.
(Even with a simple stack, handling 'close' and 'StopIteration' 
correctly is not exactly trivial)

>
> So IMO you're worrying about a problem that doesn't exist.
>
No, I am worrying about a problem that so far has only appeared in 
contrived examples designed to expose it.  Any real-life examples I have 
seen of the "yield from" feature would work perfectly well with a simple 
stack-based approach.  However, I have seen several ideas for speeding 
up long chains of "yield from"s beyond the current C implementation, and 
most of them fail either by giving wrong results (bad) or by slowing 
things down in admittedly unusual cases (not so bad, but not good).

Anyway...  it is 3 in the morning.  As I told Arnaud, I will try to find 
some time this week to write some more of these crazy examples. 

Regards

   Jacob