[Tutor] Yielding from a with block

[Steven D'Aprano]

On Wed, May 27, 2015 at 11:27:46PM +0100, Oscar Benjamin wrote:

> I'm just wondering what other people think about this. Should code
> like make_lines below be discouraged?
> 
> > def make_lines():
> >     with open('co2-sample.csv') as co2:
> >         yield htbegin
> >         for linelist in csv.reader(co2):
> >             yield from fprint(linelist)
> >         yield htend
[...]
> There's a fundamental contradiction between the with and yield
> statements. With means "don't exit this block without running my
> finalisation routine". Yield means "immediately exit this block
> without doing anything (including any finalisation routines) and then
> allow anything else to occur".

No, I think you have misunderstood the situation, and perhaps been 
fooled by two different usages of the English word "exit".


Let me put it this way:

with open(somefile) as f:
    text = f.write()  # here 
    ...

In the line marked "here", execution exits the current routine and 
passes to the write method. Or perhaps it is better to say that 
execution of the current routine pauses, because once the write method 
has finished, execution will return to the current routine, or continue, 
however you want to put it. Either way, there is a sense in which 
execution has left the with-block and is running elsewhere.

I trust that you wouldn't argue that one should avoid calling functions 
or methods inside a with-block?

In this case, there is another sense in which we have not left the 
with-block, just paused it, and in a sense the call to write() occurs 
"inside" the current routine. Nevertheless, at the assembly language 
level, the interpreter has to remember where it is, and jump to the 
write() routine, which is outside of the current routine.

Now let us continue the block:

with open(somefile) as f:
    text = f.write()
    yield text  # here 


Just like the case of transferring execution to the write() method, the 
yield pauses the currently executing code (a coroutine), and transfers 
execution to something else. That something else is the caller. So in 
once sense, we have exited the current block, but in another sense we've 
just paused it, waiting to resume, no different from the case of 
transferring execution to the write() method.

In this case, the with block is not deemed to have been exited until 
execution *within the coroutine* leaves the with-block. Temporarily 
pausing it by yielding leaves the coroutine alive (although in a paused 
state), so we haven't really exited the with-block.


> Using yield inside a with statement like this renders the with
> statement pointless: either way we're really depending on __del__ to
> execute the finalisation code. 

No, that's not correct. __del__ may not enter into it. When you run the 
generator to completion, or possibly even earlier, the with-statement 
exits and the file is closed before __del__ gets a chance to run. 
__del__ may not run until Python exits, but the file will be closed the 
moment execution inside the generator leaves the with-statement. 
Consider:

def gen():
    with open(somefile) as f:
        yield "Ready..."
    yield "Set..."
    yield "Go!"

it = gen()
next(it)

At this point, the generator is paused at the first yield statement, and 
the file is held open, possibly indefinitely. If I never return control 
to the generator, eventually the interpreter will shut down and call 
__del__. But if I continue:

next(it)  # file closed here

the with-block exits, the file is closed, and execution halts after the 
next yield. At this point, the generator is paused, but the file object 
is closed. The garbage collector cannot clean up the file (not that it 
needs to!) because f is still alive. If I never return control to the 
generator, then the garbage collector will never clean up the reference 
to f, __del__ will never run, but it doesn't matter because the file is 
already closed. But if I continue:

next(it)
next(it)

we get one more value from the generator, and on the final call to 
next() we exit the generator and f goes out of scope and __del__ may be 
called.



-- 
Steve