[Python-ideas] Yield-From: GeneratorExit?

[Bruce Frederiksen]

Greg Ewing wrote:
>
> My feeling is that GeneratorExit is a peculiarity of
> generators that other kinds of iterators shouldn't have
> to know about. So, if you close() a generator, that
> shouldn't imply throwing GeneratorExit into the
> subiterator -- [...]
It can only be "thrown into the subiterator" if the subiterator is a 
generator (i.e., has a throw method) -- in which case, it knows about 
GeneratorExit.  So the hasattr(_i, 'throw') test already covers this case.
>
>
> If the subiterator happens to be another generator,
> dropping the last reference to it will cause it to
> be closed, [...] 
NO, NO, NO.  Unless you are prepared to say that programs written to 
this spec are *not* expected to run on any other version of Python other 
than CPython.  CPython is the *only* version with a reference counting 
collector.  And encouraging Python programmers to rely on this invites 
trouble when they try to port to any other version of Python.  I know.  
I've been there, and have the T-shirt.  And it's not pretty.  The errors 
that you get when your finally clauses and context managers aren't run 
can be quite mysterious.  And God help that person if they haven't slept 
with PEP 342 under their pillow!
> Other kinds of iterators can finalize
> themselves however they see fit, and don't need to
> pretend they're generators and understand
> GeneratorExit.
Your PEP currently does not demand that other iterators "pretend they're 
generators and understand GeneratorExit".  Non-generator iterators don't 
have throw or close methods and will remain blissfully ignorant of these 
finer points as the PEP stands now.  So this is not a problem.
>
> For consistency, this implies that a GeneratorExit
> explicitly thrown in using throw() shouldn't be
> forwarded to the subiterator either, even if it has
> a throw() method.
>
> To do otherwise would require making a distinction that
> can't be expressed in the Python expansion. Also, it
> seems elegant to preserve the property that if g is a
> generator then g.close() and g.throw(GeneratorExit) are
> exactly equivalent.
Yes, g.close and g.throw(GeneratorExit) are equivalent.  So you should 
be able to translate a close into a throwing GeneratorExit or vice 
versa.  But if the subiterator doesn't have the first method that you 
look for (let's say you pick throw), then you should call the other 
method (if it has that one instead).

Finally, on your previous post, you say:

> It would also avoid the problem of a partially exhausted
> iterator that's still in use by something else getting
> prematurely finalized, which is another thing that's been
> bothering me. 
This is a valid point.  But consider:

1.  The delegating generator has no way to stop the subgenerator 
prematurely when it uses the yield from.  So the yield from can only be 
stopped prematurely by the delegating generator's caller.  And then the 
subgenerator would have to be communicated between the caller to the 
delegating generator somehow (e.g, passed in as a parameter) so that the 
caller could continue to use it.  (And the subgenerator has to be a 
generator, not a plain iterator).  Though possible, this kind of a use 
case would be used very rarely compared to the use case of the yield 
from being the final place the subgenerator is used.

2.  If finalization of the subgenerator needs to be prevented, it can be 
wrapped in a plain iterator wrapper that doesn't define throw or close.

class no_finalize:
    def __init__(self, gen):
        self.gen = gen
    def __iter__(self):
        return self
    def __next__(self):
        return next(self.gen)
    def send(self, x):
        return self.gen.send(x)

g = subgen(...)
yield from no_finalize(g)
... use g


As I see it, you are faced with two options:

1. Define "yield from" in a way that it will work the same in all 
implementations of Python and will work for the 98% use case without any 
extra boilerplate code, and only require extra boilerplate (as above) 
for the 2% use case.  or

2. Define "yield from" in a way that will have quite different behavior 
(for reasons very obscure to most programmers) on the different 
implementations of Python (due to the different implementation of 
garbage collectors), require boilerplate code to be portable for the 98% 
use case (e.g., adding a "with closing(subgen())" around the yield 
from); but not require any boilerplate code for portability in the 2% 
use case.

The only argument I can think in favor of option 2, is that's what the 
"for" statement ended up with.  But that was only because changing the 
"for" statement to option 1 would break the legacy 2% use cases...

IMHO option 1 is the better choice.

-bruce frederiksen