[Tutor] Yielding from a with block

[Peter Otten]

Oscar Benjamin wrote:

> On 23 May 2015 at 10:52, Peter Otten <__peter__ at web.de> wrote:

> I'm just wondering what other people think about this. Should code
> like make_lines below be discouraged?

>> def make_lines():
>>     with open('co2-sample.csv') as co2:
>>         yield htbegin
>>         for linelist in csv.reader(co2):
>>             yield from fprint(linelist)
>>         yield htend

In my opinion, yes. 

As I've used the pattern since Python 2.5 when try...finally in generators 
became legal I will need some time to unlearn.

> There's a fundamental contradiction between the with and yield
> statements. With means "don't exit this block without running my
> finalisation routine". Yield means "immediately exit this block
> without doing anything (including any finalisation routines) and then
> allow anything else to occur".
> 
> Using yield inside a with statement like this renders the with
> statement pointless: either way we're really depending on __del__ to
> execute the finalisation code. So it with or without the with
> statement it will work fine on CPython. On other implementations it
> may or may not close the file with or without the with statement. IOW
> the with statement is not able to provide the guarantees normally
> expected of it when used this way.

Even if you limit yourself to CPython there is another effect: the order of 
execution may not meet one's expectations/requirements:

$ cat with_in_generator.py
import contextlib

@contextlib.contextmanager
def demo():
    print("before")
    try:
        yield
    finally:
        print("after")

def gen(items="abc"):
    with demo():
        yield from items

if __name__ == "__main__":
    g = gen()
    for item in g:
        print(item)
        if item == "b":
            break
    print("bye")
$ python3 with_in_generator.py 
before
a
b
bye
after
$ 

(in case you don't spot it: "after" should be printed before "bye")
To get a well-defined order of execution you can close the generator 
explicitly

$ cat with_in_generator2.py
[...]
if __name__ == "__main__":
    g = gen()
    with contextlib.closing(g) as h:
        for item in h:
            print(item)
            if item == "b":
                break
    print("bye")
$ python3 with_in_generator2.py 
before
a
b
after
bye

...but the obvious route is of course

> It's usually fairly trivial to rearrange things so that this doesn't
> happen:
> 
> def wrap_header_footer(fin):
>     yield htbegin
>     for linelist in csv.reader(fin):
>         yield from fprint(linelist)
>     yield htend

which in this case also has the advantage of better separation of concerns 
(I'd probably move the csv.reader() out of the generator, too).

PS: I'm still looking for a fairly elegant rewrite of the problematic

def lines(files):
    for file in files:
        with open(files) as f:
            yield from f

(see Oscar's comment in 
<https://mail.python.org/pipermail/tutor/2015-May/105448.html>)