[Tutor] Yielding from a with block

[Steven D'Aprano]

On Fri, May 29, 2015 at 03:01:50PM +0100, Oscar Benjamin wrote:
> On 29 May 2015 at 14:29, Steven D'Aprano <steve at pearwood.info> wrote:
> > On Thu, May 28, 2015 at 10:16:00AM +0200, Peter Otten wrote:
> >
> >> Even if you limit yourself to CPython there is another effect: the order of
> >> execution may not meet one's expectations/requirements:
> >
> > [snip example]
> >
> > That's an interesting example, and I can't tell if that's a
> > problem with your (and my) expectations, or a bug in the context
> > manager implementation.
> 
> I think that this behaviour is consistent with the generally
> unspecified language-level behaviour of deallocation and __del__.

Yes it is. There's nothing unusual going on in this example. The 
generator is still paused *inside the with block*, and it doesn't exit 
the with block until the generator is closed on deallocation, which 
might not be until interpreter exit.

[...]
> It is not guaranteed that __del__() methods are called for
> objects that still exist when the interpreter exits.

Okay, well that gives pypy a "Get Out Of Jail Free" card for the 
behaviour you demonstrated. I guess it's not necessarily a bug.


[...]
> But the with statement is not "exited" as the suite is unfinished.

Exactly! So Peter's expectation that it should be finalised is wrong. He 
managed to convince me for a few moments that there was something 
mysterious going on, but on further thought I realised that the observed 
behaviour is correct and should be expected. Escaping from the "for x in 
generator" loop early leaves the generator paused inside the with 
block.


> The
> generator has suspended itself and removed itself from the call-stack
> so that it is no longer permitted to catch exceptions etc. Also note
> that the removal from the call stack is not an implementation detail.
> The interpreter can implement the call stack how it likes but it must
> be semantically equivalent to a call stack with exception unwinding in
> order to meet the general definition of the language.

The documention of with statements and context managers says nothing 
about the call stack, and nor should it, since the call stack is part of 
the implementation of the Python virtual machine, not part of the Python 
runtime. What's important is not *how* a paused generator ceases to 
catch exceptions etc. but the fact that it no longer does. That's what I 
mean by calling it an implementation detail.

-- 
Steve