[Tutor] reading lines from a list of files

[Oscar Benjamin]

On 12 May 2015 at 11:46, Peter Otten <__peter__ at web.de> wrote:
>
> > although the reason
> > for my inquiry was more to diminish levels of indentation
> > than number of lines.
>
> You usually do that by factoring out the loops into a generator:
>
> def lines(files):
>     for file in files:
>         with open(files) as f:
>             yield from f  # before python 3.3: for line in f: yield line
>
>
> for line in lines(files):
>     ...
>
>
> Also possible, but sloppy as files are closed on garbage collection rather
> than explicitly:
>
> lines = (line for file in files for line in open(file))
> for line in lines:
>    ...

The lines generator function above relies on __del__ as well if the
loop exits from break, return or an exception in the loop body. This
happens whenever you yield or yield-from from inside a with block. The
chain of events that calls your context manager if I break from the
loop is:

1) Once the for loop discards the generator it has a zero reference count.
2) generator.__del__() called.
3) __del__() calls close()
4) close() throws GeneratorExit into the generator frame.
5) The GeneratorExit triggers the __exit__() methods of any context
managers active in the generator frame.

Try the following:

$ cat test_gen_cm.py
#!/usr/bin/env python3

class cleanup():
    def __enter__(self):
        pass
    def __exit__(self, *args):
        print("__exit__ called")__del__.

def generator_with_cm():
    with cleanup():
        yield 1
        yield 2
        yield 3

g = generator_with_cm()
for x in g:
    break

print('Deleting g')
del g
print('g is now deleted')

$ ./test_gen_cm.py
Deleting g
__exit__ called
g is now deleted

A generator cannot guarantee that execution continues after a yield so
any context manager used around a yield is dependent on __del__. I
think a good rule of thumb is "don't yield from a with block".

Alex I apologise if what I've written here is confusing but really
what you started with is just fine. It is not important to fully
understand what I wrote above.


--
Oscar