Nick Coghlan wrote:
[snip arguments for modelling on basic iteration] That is, I now believe the 'normal' case for 'yield from' should be modelled on basic iteration, which means no implicit finalisation.
Now, keep in mind that in parallel with this I am now saying that *all* exceptions, *including GeneratorExit* should be passed down to the subiterator if it has a throw() method.
I still think that is less useful than catching it and just dropping the reference, see below.
So even without implicit finalisation you can use "yield from" to nest generators to your heart's content and an explicit close on the outermost generator will be passed down to the innermost generator and unwind the generator stack from there.
The same would happen with the *implicit* close caused by the last reference to the outermost generator going away. Delegating the GeneratorExit is a sure way to premature finalization when using shared generators, but only in a refcounting implementation like C-Python. That makes this the only feature I know of that would be *more* useful in a non-refcounting implementation.
Using your "no finally clause" version from earlier in this thread as the base for the exact semantic description:
_i = iter(EXPR) try: _u = _i.next() except StopIteration, _e: _r = _e.value else: while 1: try: _v = yield _u except BaseException, _e: _m = getattr(_i, 'throw', None) if _m is not None: _u = _m(_e) else: raise else: try: if _v is None: _u = _i.next() else: _u = _i.send(_v) except StopIteration, _e: _r = _e.value break RESULT = _r
I know I didn't comment on that expansion earlier, but should have. It fails to handle the case where the throw raises a StopIteration (or there is no throw method and the thrown exception is a StopIteration). You need something like: _i = iter(EXPR) try: _u = _i.next() while 1: try: _v = yield _u # except GeneratorExit: # raise except BaseException: _m = getattr(_i, 'throw', None) if _m is not None: _u = _m(*sys.exc_info()) else: raise else: if _v is None: _u = _i.next() else: _u = _i.send(_v) except StopIteration, _e: RESULT = _e.value finally: _i = _u = _v = _e = _m = None del _i, _u, _v, _e, _m This is independent of the GeneratorExit issue, but I put it in there as a comment just to make it clear what *I* think it should be if we are not putting a close in the finally clause. If we *do* put a call to close in the finally clause, the premature finalization of shared generators is guaranteed anyway, so there is not much point in specialcasing GeneratorExit.
With an expansion of that form, you can easily make arbitrary iterators (including generators) shareable by wrapping them in an iterator with no throw or send methods:
class ShareableIterator(object): def __init__(self, itr): self.itr = itr def __iter__(self): return self def __next__(self): return self.itr.next() next = __next__ # Be 2.x friendly def close(self): # Still support explicit finalisation of the # shared iterator, just not throw() or send() try: close_itr = self.itr.close except AttributeError: pass else: close_itr()
# Decorator to use the above on a generator function def shareable(g): @functools.wraps(g) def wrapper(*args, **kwds): return ShareableIterator(g(*args, **kwds)) return wrapper
With this wrapper, you will not be able to throw *any* exceptions to the shared iterator. Even if you fix the wrapper to pass through all other exceptions than GeneratorExit, you will still completely lose the speed benefits of yield-from when doing so. (For next, send, and throw it is possible to completely bypass all the intervening generators, so the call overhead becomes independent of the number of generators in the yield-from chain. I have a patch that does exactly this, working except for details related to this discussion). It is not possible to write such a wrapper efficiently without making it a builtin and special-casing it in the yield-from implementation, and I don't think that is a good idea.
Iterators that need finalisation can either make themselves implicitly closable in yield from expressions by defining a throw() method that delegates to close() and then reraises the exception appropriately, or else they can recommend explicit closure regardless of the means of iteration (be it a for loop, a generator expression or container comprehension, manual iteration or the new yield from expression).
A generator or iterator that needs closing should recommend explicit closing *anyway* to work correctly in other contexts on platforms other than C-Python. Not delegating GeneratorExit just happens to make it much simpler and faster to use shared generators/iterators that *don't* need immediate finalization. In C-Python you even get the finalization for free due to the refcounting, but of course relying on that is generally considered a bad idea. - Jacob