[Python-ideas] Yield-From: GeneratorExit?
dangyogi at gmail.com
Mon Mar 23 03:40:21 CET 2009
Greg Ewing wrote:
> My feeling is that GeneratorExit is a peculiarity of
> generators that other kinds of iterators shouldn't have
> to know about. So, if you close() a generator, that
> shouldn't imply throwing GeneratorExit into the
> subiterator -- [...]
It can only be "thrown into the subiterator" if the subiterator is a
generator (i.e., has a throw method) -- in which case, it knows about
GeneratorExit. So the hasattr(_i, 'throw') test already covers this case.
> If the subiterator happens to be another generator,
> dropping the last reference to it will cause it to
> be closed, [...]
NO, NO, NO. Unless you are prepared to say that programs written to
this spec are *not* expected to run on any other version of Python other
than CPython. CPython is the *only* version with a reference counting
collector. And encouraging Python programmers to rely on this invites
trouble when they try to port to any other version of Python. I know.
I've been there, and have the T-shirt. And it's not pretty. The errors
that you get when your finally clauses and context managers aren't run
can be quite mysterious. And God help that person if they haven't slept
with PEP 342 under their pillow!
> Other kinds of iterators can finalize
> themselves however they see fit, and don't need to
> pretend they're generators and understand
Your PEP currently does not demand that other iterators "pretend they're
generators and understand GeneratorExit". Non-generator iterators don't
have throw or close methods and will remain blissfully ignorant of these
finer points as the PEP stands now. So this is not a problem.
> For consistency, this implies that a GeneratorExit
> explicitly thrown in using throw() shouldn't be
> forwarded to the subiterator either, even if it has
> a throw() method.
> To do otherwise would require making a distinction that
> can't be expressed in the Python expansion. Also, it
> seems elegant to preserve the property that if g is a
> generator then g.close() and g.throw(GeneratorExit) are
> exactly equivalent.
Yes, g.close and g.throw(GeneratorExit) are equivalent. So you should
be able to translate a close into a throwing GeneratorExit or vice
versa. But if the subiterator doesn't have the first method that you
look for (let's say you pick throw), then you should call the other
method (if it has that one instead).
Finally, on your previous post, you say:
> It would also avoid the problem of a partially exhausted
> iterator that's still in use by something else getting
> prematurely finalized, which is another thing that's been
> bothering me.
This is a valid point. But consider:
1. The delegating generator has no way to stop the subgenerator
prematurely when it uses the yield from. So the yield from can only be
stopped prematurely by the delegating generator's caller. And then the
subgenerator would have to be communicated between the caller to the
delegating generator somehow (e.g, passed in as a parameter) so that the
caller could continue to use it. (And the subgenerator has to be a
generator, not a plain iterator). Though possible, this kind of a use
case would be used very rarely compared to the use case of the yield
from being the final place the subgenerator is used.
2. If finalization of the subgenerator needs to be prevented, it can be
wrapped in a plain iterator wrapper that doesn't define throw or close.
def __init__(self, gen):
self.gen = gen
def send(self, x):
g = subgen(...)
yield from no_finalize(g)
... use g
As I see it, you are faced with two options:
1. Define "yield from" in a way that it will work the same in all
implementations of Python and will work for the 98% use case without any
extra boilerplate code, and only require extra boilerplate (as above)
for the 2% use case. or
2. Define "yield from" in a way that will have quite different behavior
(for reasons very obscure to most programmers) on the different
implementations of Python (due to the different implementation of
garbage collectors), require boilerplate code to be portable for the 98%
use case (e.g., adding a "with closing(subgen())" around the yield
from); but not require any boilerplate code for portability in the 2%
The only argument I can think in favor of option 2, is that's what the
"for" statement ended up with. But that was only because changing the
"for" statement to option 1 would break the legacy 2% use cases...
IMHO option 1 is the better choice.
More information about the Python-ideas