Greg Ewing wrote:
Nick Coghlan wrote:
Greg Ewing wrote:
(1) In non-refcounting implementations, subiterators are finalized promptly when the delegating generator is explicitly closed.
(2) Subiterators are not prematurely finalized when other references to them exist.
If you choose (2), then (1) is trivial to implement
with contextlib.closing(make_subiter()) as subiter: yield from subiter
That's a fairly horrendous thing to expect people to write around all their yield-froms, though. It also means we would have to say that the inlining principle only holds for refcounting implementations.
Maybe we should just give up trying to accommodate shared subiterators. Is it worth complicating everything for the sake of something that's not really part of the intended set of use cases?
Consider what happens if you replace the 'yield from' with the basic form of iterator delegation that exists now: for x in make_subiter(): yield x Is such code wrong in any way? No it isn't. Failing to finalise the object of iteration is the *normal* case. If for some reason it is important in a given application to finalise it properly (e.g. the subiter opens a database connection or file and we want to ensure they are closed promptly no matter what else happens), only *then* does deterministic finalisation come into play: with closing(make_subiter()) as subiter: for x in subiter: yield x That is, I now believe the 'normal' case for 'yield from' should be modelled on basic iteration, which means no implicit finalisation. Now, keep in mind that in parallel with this I am now saying that *all* exceptions, *including GeneratorExit* should be passed down to the subiterator if it has a throw() method. So even without implicit finalisation you can use "yield from" to nest generators to your heart's content and an explicit close on the outermost generator will be passed down to the innermost generator and unwind the generator stack from there. Using your "no finally clause" version from earlier in this thread as the base for the exact semantic description: _i = iter(EXPR) try: _u = _i.next() except StopIteration, _e: _r = _e.value else: while 1: try: _v = yield _u except BaseException, _e: _m = getattr(_i, 'throw', None) if _m is not None: _u = _m(_e) else: raise else: try: if _v is None: _u = _i.next() else: _u = _i.send(_v) except StopIteration, _e: _r = _e.value break RESULT = _r With an expansion of that form, you can easily make arbitrary iterators (including generators) shareable by wrapping them in an iterator with no throw or send methods: class ShareableIterator(object): def __init__(self, itr): self.itr = itr def __iter__(self): return self def __next__(self): return self.itr.next() next = __next__ # Be 2.x friendly def close(self): # Still support explicit finalisation of the # shared iterator, just not throw() or send() try: close_itr = self.itr.close except AttributeError: pass else: close_itr() # Decorator to use the above on a generator function def shareable(g): @functools.wraps(g) def wrapper(*args, **kwds): return ShareableIterator(g(*args, **kwds)) return wrapper Iterators that need finalisation can either make themselves implicitly closable in yield from expressions by defining a throw() method that delegates to close() and then reraises the exception appropriately, or else they can recommend explicit closure regardless of the means of iteration (be it a for loop, a generator expression or container comprehension, manual iteration or the new yield from expression). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------