[Python-ideas] Yield-From: Finalization guarantees

[Jacob Holm]

Nick Coghlan wrote:
> [snip arguments for modelling on basic iteration]
> That is, I now believe the 'normal' case for 'yield from' should be
> modelled on basic iteration, which means no implicit finalisation.
>
> Now, keep in mind that in parallel with this I am now saying that *all*
> exceptions, *including GeneratorExit* should be passed down to the
> subiterator if it has a throw() method.
>   

I still think that is less useful than catching it and just dropping the 
reference, see below.

> So even without implicit finalisation you can use "yield from" to nest
> generators to your heart's content and an explicit close on the
> outermost generator will be passed down to the innermost generator and
> unwind the generator stack from there.
>   

The same would happen with the *implicit* close caused by the last 
reference to the outermost generator going away.

Delegating the GeneratorExit is a sure way to premature finalization 
when using shared generators, but only in a refcounting implementation 
like C-Python. That makes this the only feature I know of that would be 
*more* useful in a non-refcounting implementation.

> Using your "no finally clause" version from earlier in this thread as
> the base for the exact semantic description:
>
>     _i = iter(EXPR)
>     try:
>         _u = _i.next()
>     except StopIteration, _e:
>         _r = _e.value
>     else:
>         while 1:
>             try:
>                 _v = yield _u
>             except BaseException, _e:
>                 _m = getattr(_i, 'throw', None)
>                 if _m is not None:
>                     _u = _m(_e)
>                 else:
>                     raise
>             else:
>                 try:
>                     if _v is None:
>                         _u = _i.next()
>                     else:
>                         _u = _i.send(_v)
>                 except StopIteration, _e:
>                     _r = _e.value
>                     break
>     RESULT = _r
>
>   

I know I didn't comment on that expansion earlier, but should have. It 
fails to handle the case where the throw raises a StopIteration (or 
there is no throw method and the thrown exception is a StopIteration). 
You need something like:

    _i = iter(EXPR)
    try:
        _u = _i.next()
        while 1:
            try:
                _v = yield _u
    #        except GeneratorExit:
    #            raise
            except BaseException:
                _m = getattr(_i, 'throw', None)
                if _m is not None:
                    _u = _m(*sys.exc_info())
                else:
                    raise
            else:
                if _v is None:
                    _u = _i.next()
                else:
                    _u = _i.send(_v)
    except StopIteration, _e:
        RESULT = _e.value
    finally:
        _i = _u = _v = _e = _m = None
        del _i, _u, _v, _e, _m

This is independent of the GeneratorExit issue, but I put it in there as 
a comment just to make it clear what *I* think it should be if we are 
not putting a close in the finally clause. If we *do* put a call to 
close in the finally clause, the premature finalization of shared 
generators is guaranteed anyway, so there is not much point in 
specialcasing GeneratorExit.
> With an expansion of that form, you can easily make arbitrary iterators
> (including generators) shareable by wrapping them in an iterator with no
> throw or send methods:
>
>   class ShareableIterator(object):
>     def __init__(self, itr):
>       self.itr = itr
>     def __iter__(self):
>       return self
>     def __next__(self):
>       return self.itr.next()
>     next = __next__ # Be 2.x friendly
>     def close(self):
>       # Still support explicit finalisation of the
>       # shared iterator, just not throw() or send()
>       try:
>          close_itr = self.itr.close
>       except AttributeError:
>          pass
>       else:
>          close_itr()
>
>   # Decorator to use the above on a generator function
>   def shareable(g):
>     @functools.wraps(g)
>     def wrapper(*args, **kwds):
>       return ShareableIterator(g(*args, **kwds))
>     return wrapper
>   

With this wrapper, you will not be able to throw *any* exceptions to the 
shared iterator. Even if you fix the wrapper to pass through all other 
exceptions than GeneratorExit, you will still completely lose the speed 
benefits of yield-from when doing so. (For next, send, and throw it is 
possible to completely bypass all the intervening generators, so the 
call overhead becomes independent of the number of generators in the 
yield-from chain. I have a patch that does exactly this, working except 
for details related to this discussion). It is not possible to write 
such a wrapper efficiently without making it a builtin and 
special-casing it in the yield-from implementation, and I don't think 
that is a good idea.

> Iterators that need finalisation can either make themselves implicitly
> closable in yield from expressions by defining a throw() method that
> delegates to close() and then reraises the exception appropriately, or
> else they can recommend explicit closure regardless of the means of
> iteration (be it a for loop, a generator expression or container
> comprehension, manual iteration or the new yield from expression).
>   

A generator or iterator that needs closing should recommend explicit 
closing *anyway* to work correctly in other contexts on platforms other 
than C-Python. Not delegating GeneratorExit just happens to make it much 
simpler and faster to use shared generators/iterators that *don't* need 
immediate finalization. In C-Python you even get the finalization for 
free due to the refcounting, but of course relying on that is generally 
considered a bad idea.

- Jacob