[ python-Bugs-1542308 ] Nested finally in generators don't follow PEP 342

[SourceForge.net]

Bugs item #1542308, was opened at 2006-08-17 22:56
Message generated for change (Comment added) made by pje
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1542308&group_id=5470

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Python Interpreter Core
Group: Python 2.5
Status: Open
Resolution: None
Priority: 8
Submitted By: Bob Ippolito (etrepum)
Assigned to: Nobody/Anonymous (nobody)
Summary: Nested finally in generators don't follow PEP 342

Initial Comment:
The close() and GC interaction of generators that use yield inside of finally 
blocks doesn't execute correctly when nested. See the attached example.

More information about the issue is in the Mozilla bug tracker (they found 
a similar bug in their implementation for JS 1.7):
https://bugzilla.mozilla.org/show_bug.cgi?id=349012

----------------------------------------------------------------------

>Comment By: Phillip J. Eby (pje)
Date: 2006-08-22 22:51

Message:
Logged In: YES 
user_id=56214

You'll need to propose a patch to PEP 342 to alter the
specification, then, and get it past Python-Dev. 
Personally, I don't think that changing the spec is the
right thing to do for 2.5.

But irrespective of those procedural matters, your __del__
analogy is flawed on two points.  First, we do not re-run a
__del__ method to handle an error that was raised by that
__del__ method!  Second, if a generator contains an
infinite, non-yielding loop, then of course it will loop
forever.  So __del__ does not provide any actually useful
guidance here.


----------------------------------------------------------------------

Comment By: Bob Ippolito (etrepum)
Date: 2006-08-22 22:30

Message:
Logged In: YES 
user_id=139309

It seems that the infinite loop would be the right thing to do, given that's what 
would happen in a similarly broken __del__ (or anywhere else).

----------------------------------------------------------------------

Comment By: Phillip J. Eby (pje)
Date: 2006-08-22 22:23

Message:
Logged In: YES 
user_id=56214

Bob, the problem Guido is pointing out is that to run the
finally clauses, we have to resume the generator with the
RuntimeError thus generated.  So it doesn't terminate the
loop, because the RuntimeError is effectively raised at the
point where the yield occurs.  So, in order to run finally
clauses sanely after a close() attempt, we would have to
prevent such loops.  

That doesn't mean Guido's example is valid as such; but I
think it's possible to accidentally create quasi-indefinite
loops using infinite iterators written prior to Python 2.5,
if you had a try/except block that was expecting to catch
something *other* than GeneratorExit or RuntimeError.  So,
the net effect would be an unintended hang, if we tried to
support retrying until the generator can be closed.

Regarding the GC second attempt, this behavior is actually
exactly as-specified by the PEP's pseudo-code explanation of
how close() is handled.  A RuntimeError raised from close()
does *not* close the generator; it specifically indicates
that the generator has not in fact closed.

At this point, after re-reading the PEP and looking at what
the code is doing, it's clear that the behavior is "as
specified", so the title of the bug is incorrect: the
behavior is exactly as specified by PEP 342.  So, the rest
of my comments below are regarding whether PEP 342 should be
changed.

And really, the question there is whether it's sane to
attempt heroic measures in order to run finally clauses in a
generator whose code is *known* to be buggy.  The
RuntimeError basically says, "this generator is broken", and
the GC also tries a second time to close it.  If two close
attempts don't close it, it's a dead duck.

What other measures can we sanely add?  We could try forcing
the RuntimeError into the generator itself, but then we have
to deal with the infinite loop possibility, and the oddities
involved in getting to a language specification that can be
implemented for Jython, IronPython, etc.  On the whole, I
think that we probably reached the right balance the first
time around: a broken generator should not be allowed to
handle the error of its own brokenness.


----------------------------------------------------------------------

Comment By: Bob Ippolito (etrepum)
Date: 2006-08-22 21:00

Message:
Logged In: YES 
user_id=139309

Uh no, that wouldn't reach an infinite loop because any attempt to yield during 
close will raise RuntimeError and terminate the loop.

The problem is that finally doesn't execute. Finally clauses always must execute. 
If they don't, then they're worthless.

The real strange issue is that if close is called, then GC makes a second attempt, 
and it *does* execute the outer finally clause. There are definitely bugs here.

----------------------------------------------------------------------

Comment By: Guido van Rossum (gvanrossum)
Date: 2006-08-22 20:47

Message:
Logged In: YES 
user_id=6380

I lost my patience halfway through reading the Mozilla bug
tracker, but IMO this works as designed.  The philosophical
question is, would you rather see the outer finally clause
reached in your example, or would you rather see the
following generator terminated upon close? (If you "fix" the
former, the latter ends up in an infinite loop when you
attempt to close() or GC it.)

def gen():
  while True:
    try:
      yield
    except:
      pass

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1542308&group_id=5470