[Python-ideas] use context managers for new-style "for" statement

[Bruce Frederiksen]

Raymond Hettinger wrote:
> ## untested recipe
>
> def closeme(iterable):
> it = iter(iterable)
> try:
>  for i in it:
>   yield i
> finally:
>   it.close()
>
>
> # doesn't this do this same thing without any interpreter magic?
> for i in closeme(gen(x)):
>   ...
>
> for i in chain.from_iterable(map(closeme, [it1, it2, it3, it4])):
>   ...
No, this only works on CPython because of the referencing counting 
collector and the fact that PEP 342 specifies that __del__ calls close 
on generators.  This will not work reliably on Jython, IronPython or 
Pypy because none of these have reference counting collectors.

The closeme generator really adds nothing here, because it is just 
another generator that relies on either running off the end of the 
generator, or its close or throw methods to be called to activate the 
finally clause.  This is identical to the generators that it is being 
mapped over.  *Nothing* in python is defined to call the close or throw 
methods on generators, except for the generator __del__ method -- and 
that is *only* called reliably in CPython, and not in any of the other 
python implementations which may never garbage collect the generator if 
it's allocated near the end of the program run!

I had generators with try/finally, and these fail on Jython and 
IronPython.  What I ended up doing to get my program working on Jython 
was to convert all of my generators to return context managers.  That 
way I could not accidentally forget to use a with statement with them.  
Thus:

def gen(x):
    return itertools.chain.from_iterable(...)

for i in gen(x):
    ...

becomes the following hack:

class chain_context(object):
    def __init__(self, outer_it):
        self.outer_it = outer_iterable(outer_it)
    def __enter__(self):
        return itertools.chain.from_iterable(self.outer_it)
    def __exit__(self, type, value, tb): self.outer_it.close()

class outer_iterable(object):
    def __init__(self, outer_it):
        self.outer_it = iter(outer_it)
        self.inner_it = None
    def __iter__(self): return self
    def close(self):
        if hasattr(self.inner_it, '__exit__'):
            self.inner_it.__exit__(None, None, None)
        elif hasattr(self.inner_it, 'close'): self.inner_it.close()
        if hasattr(self.outer_it, 'close'): self.outer_it.close()
    def next(self):
        ans = self.outer_it.next()
        if hasattr(ans, '__enter__'):
            self.inner_it = ans
            return ans.__enter__()
        ans = iter(ans)
        self.inner_it = ans
        return ans

def gen(x):
    return chain_context(...)

with gen(x) as it:
    for i in it:
        ...

Most of my generators used chain.  Those that didn't went from:

def gen(x):
    ...

to:

def gen(x):
    def gen2(x):
        ...
    return contextlib.closing(gen2(x))

This got the program working on Jython in a way that future maintenance 
on the program can't screw up, but it sure doesn't feel "pythonic"...

-bruce frederiksen