[Tutor] Yielding from a with block

[Cameron Simpson]

On 27May2015 23:27, Oscar Benjamin <oscar.j.benjamin at gmail.com> wrote:
>On 23 May 2015 at 10:52, Peter Otten <__peter__ at web.de> wrote:
>> import csv
>
>I'm just wondering what other people think about this. Should code
>like make_lines below be discouraged?

In my opinion, no.

>> def make_lines():
>>     with open('co2-sample.csv') as co2:
>>         yield htbegin
>>         for linelist in csv.reader(co2):
>>             yield from fprint(linelist)
>>         yield htend
>>
>>
>> def fprint(linelist):
>>     yield ("<td class=l>{}</td><td>{}</td><td>{}</td>"
>>            "<td>{}</td><td>{}</td><td>{}</td>\n").format(*linelist)
>>     yield '</tr>\n'
>
>There's a fundamental contradiction between the with and yield
>statements. With means "don't exit this block without running my
>finalisation routine". Yield means "immediately exit this block
>without doing anything (including any finalisation routines) and then
>allow anything else to occur".

Not really. Yield passes a return value back to the iterator user and _pauses_ 
execution of the generator. The generator's execution context is _still_ inside 
the "with".

Certainly if the user/consumer of the generator does not consume all the way to 
the end then the "with" exit code won't fire. At least in a timely fashion; 
might it fire during the cleanup phase? I'd think so.

However, the common circumstance is that the consumer does run to the end of 
the iterator, and therefore the "with" exit process _will_ run when because 
that is a standard part of the generator's execution.

BTW, I think including the fprint() function here is a furphy; it doesn't 
affect the issue discussed.

>Using yield inside a with statement like this renders the with
>statement pointless:

This is not true.

>either way we're really depending on __del__ to
>execute the finalisation code. So it with or without the with
>statement it will work fine on CPython. On other implementations it
>may or may not close the file with or without the with statement. IOW
>the with statement is not able to provide the guarantees normally
>expected of it when used this way.

If make_lines() above runs to completion then the "with" exit process will also 
run and everything will be fine.

It is only when make_lines() is not completely consumed that the file may not 
be closed in a timely fashion.

Why? Because the consumer does not get to see end-of-iterator until 
make_lines() actually returns, and that directly implies completion of the 
"with" exit phase.

>It's usually fairly trivial to rearrange things so that this doesn't happen:
>
>def wrap_header_footer(fin):
>    yield htbegin
>    for linelist in csv.reader(fin):
>        yield from fprint(linelist)
>    yield htend

I would not encourage this kind of thing except in quite special circumstance.  
For your example, failure to completely consume make_ines() will result in 
invalid HTML output (missing closing "htend", for example). So the result of 
the entire process will be invalid anyway, with the leaked open file just 
another side effect.

Normally (ignoring unbounded iterators), the same scenario will apply: you 
always iterate to the end of somehting like your example because to do 
otherwise will usually result in an invalid/incorrect final result. So while 
technically the risk you describe exists, in practice it will almost never 
occur unless the larger program outcome also results in failure of some kind.

Cheers,
Cameron Simpson <cs at zip.com.au>

I object to doing things that computers can do. - Olin Shivers