[Tutor] Yielding from a with block

[Oscar Benjamin]

On 28 May 2015 at 03:12, Steven D'Aprano <steve at pearwood.info> wrote:
> On Wed, May 27, 2015 at 11:27:46PM +0100, Oscar Benjamin wrote:
>
>> I'm just wondering what other people think about this. Should code
>> like make_lines below be discouraged?
>>
>> > def make_lines():
>> >     with open('co2-sample.csv') as co2:
>> >         yield htbegin
>> >         for linelist in csv.reader(co2):
>> >             yield from fprint(linelist)
>> >         yield htend
> [...]
>> There's a fundamental contradiction between the with and yield
>> statements. With means "don't exit this block without running my
>> finalisation routine". Yield means "immediately exit this block
>> without doing anything (including any finalisation routines) and then
>> allow anything else to occur".
>
> No, I think you have misunderstood the situation, and perhaps been
> fooled by two different usages of the English word "exit".
>
>
> Let me put it this way:
>
> with open(somefile) as f:
>     text = f.write()  # here
>     ...
>
> In the line marked "here", execution exits the current routine and
> passes to the write method. Or perhaps it is better to say that
> execution of the current routine pauses, because once the write method
> has finished, execution will return to the current routine, or continue,
> however you want to put it. Either way, there is a sense in which
> execution has left the with-block and is running elsewhere.
>
> I trust that you wouldn't argue that one should avoid calling functions
> or methods inside a with-block?
>
> In this case, there is another sense in which we have not left the
> with-block, just paused it, and in a sense the call to write() occurs
> "inside" the current routine. Nevertheless, at the assembly language
> level, the interpreter has to remember where it is, and jump to the
> write() routine, which is outside of the current routine.
>
> Now let us continue the block:
>
> with open(somefile) as f:
>     text = f.write()
>     yield text  # here
>
>
> Just like the case of transferring execution to the write() method, the
> yield pauses the currently executing code (a coroutine), and transfers
> execution to something else. That something else is the caller. So in
> once sense, we have exited the current block, but in another sense we've
> just paused it, waiting to resume, no different from the case of
> transferring execution to the write() method.
>
> In this case, the with block is not deemed to have been exited until
> execution *within the coroutine* leaves the with-block. Temporarily
> pausing it by yielding leaves the coroutine alive (although in a paused
> state), so we haven't really exited the with-block.

I'm sure you understand the fundamental difference between calling a
function and yielding inside a with statement: when calling a function
the new frame is *appended* to the call stack keeping the current
frame and all of its exception traps and context managers in place
ready for unwinding. When yielding the current frame is *removed* from
the call stack (along with its exception traps and context managers).

When a with block is used without a yield it is not possible (barring
unrecoverable failure of the runtime itself) to leave the block
without calling its finaliser. It doesn't matter if we call a function
that in turn calls a generator. As long as there is no yield inside
the with block in the current frame then the finalisation guarantee is
maintained. When using yield we don't have this guarantee and in fact
there are common non-exceptional cases (break/return) where the
undesirable occurs.

>> Using yield inside a with statement like this renders the with
>> statement pointless: either way we're really depending on __del__ to
>> execute the finalisation code.
>
> No, that's not correct. __del__ may not enter into it. When you run the
> generator to completion, or possibly even earlier, the with-statement
> exits and the file is closed before __del__ gets a chance to run.
[snip]

Obviously there is this case in which the with achieves its purpose
but I mean this in a more general context than the specific example. I
consider that it is generally best to avoid doing this. In the same
way that I would advise someone to always use with when working with
files (even though it often doesn't matter) I would advise someone
generally not to yield from a with statement.


--
Oscar