[Python-ideas] x=(yield from) confusion [was:Yet another alternative name for yield-from]

[Jacob Holm]

Hi Guido

Thank you for taking the time.  I'll try to be brief so I don't spend 
more of it than necesary...  (I'll probably fail though)

Guido van Rossum wrote:
> Hm. I am beginning to see what you are asking for. Your averager
> example is somewhat convincing.
>
> Interestingly, in a sense it seems unrelated to yield-from: the
> averager doesn't seem to need yield-from, it receives values sent to
> it using send(). 

It is related to yield-from only because the "return value from 
generator" concept is introduced there.  I was trying to work out the 
details of how I would like that to work in the context of GeneratorExit 
and needed a near-trivial example.


[snip "summer" example using existing features and comparison with 
version using suggested new features]
> At this point, I admin that I am not yet convinced. On the one hand,
> my support code melts away, except for the @coroutine decorator. On
> the other hand:
>
> - the overloading of GeneratorExit and StopIteration reduces
> diagnostics for common beginner's mistakes when writing regular
> (iterator-style) generator code
> - the usage example isn't much simpler
> - the support code isn't rocket science
> - the coroutine use case is specialized enough that a little support
> seems okay (you still need @coroutine anyway)
>
> The last three points don't sway me either way: they pit minor
> conveniences against minor inconveniences.
>
> However, the first point worries me a lot. The concern over
> StopIteration can be dealt with by introducing a new exception that is
> raised only by "return value" inside a generator.
>   

I am still trying to get a clear picture of what kind of mistakes you 
are trying to protect against. 

If it is people accidently writing return in a generator when they 
really mean yield, that is what I thought the proposal for an alternate 
syntax was for.  That sounds like a good idea to me, especially if we 
could also ban or discourage the use of normal return.  But the 
alternate syntax doesn't have to mean a different exception.

Since you are no longer pushing an alternative syntax for return but 
still want a different exception, I'll assume there is some other 
beginner mistake you are worried about.  My guess is it is some mistake 
at the places where the generator is used, but I am having a hard time 
figuring out where the mistake could be in ignoring the returned value.  
Perhaps you (or someone who has more time) can provide an example where 
this is a bad thing?

> But I'm also worried that the mere need to catch GeneratorExit for a
> purpose other than resource cleanup will cause examples using it to
> pop up on the web, which will then be copied and modified by clueless
> beginners, and *increase* the probability of bad code being written.
> (That's why I introduce *new* exceptions in my support code -- they
> don't have predefined meanings in other contexts.)
>   

That worry I can understand.

> Finally, I am not sure of the connection with "yield from". I don't
> see a way to exploit it for this example. As an exercise, I
> constructed an "averager" generator out of the above "summer" and a
> similar "counter", and I didn't see a way to exploit "yield from". The
> only connection seems to be PEP 380's proposal to turn "return value"
> inside a generator into "raise StopIteration(value)", and that's the
> one part of the PEP with which I have a problem anyway (the beginner's
> issues above). 

Yes, the only connection is that this is where "return value" is 
introduced.   I could easily see "return value" as a separate PEP, 
except PEP 380 provides one of the better reasons for its inclusion.  It 
might be good to figure out how this feature should work by itself 
before complicating things by integrating it in the yield-from semantics.

> Oh, and "yield from" competes with @couroutine over
> when the initial next() call is made, which again suggests the two
> styles (yield-from and coroutines) are incompatible.
>   

It is a serious problem, because one of the major points of the PEP is 
that it should be useful for refactoring coroutines.  As a matter of 
fact, I started another thread on this specific issue earlier today 
which only Nick has so far responded to.  I think it is solvable, but 
requires some more work.

> All in all, I think I would be okay with turning "return value" inside
> a generator into raising *some* exception, as long as that exception
> is not StopIteration (nor derives from it, nor from GeneratorExit).
> PEP 380 and its implementation would become just a tad more complex,
> but I think that's worth it. Generators used as iterators would raise
> a (normally) uncaught exception if they returned a value, and that's
> my main requirement. I'm still not convince that more is needed, in
> particular I'm still -0 on catching this value in gen_close() and
> returning the value attribute from there.
>   

As long as close is not catching it without also returning the value.  
That would be *really* annoying.

> As I've said before, I don't care whether "return None" would be
> treated more like "return" or more like "return value" -- for
> beginners' code I don't think it matters, and for advanced code they
> should be equivalent.
>
> I'll stop arguing for new syntax to return a value from a generator
> (like Phillip Eby's proposed "return from yield with <value>"): I
> don't think it adds enough to overcome the pain for the parser and
> other tools.
>   

And here I was starting to think that a new syntax for return could 
solve the problem of beginner mistakes without needing a new exception.

> Finally, as far as a name for the new exception, I think something
> long like ReturnFromGenerator would be fine, since most of the time it
> is handled implicitly by coroutine support code (whether this is user
> code or gen_close()) or the yield-from implementation.
>   

GeneratorReturn is the name we have been using for this beast so far, 
but I really don't care what it is called.

> I'm sorry for being so long winded and yet somewhat inconclusive. I
> wouldn't have bothered if I didn't think there was something worth
> pursuing. But it sure seems elusive.
>   

And I thank you again for your time.

Best regards
- Jacob