Combining the best of PEP 288 and PEP 325: generator exceptions and cleanup
I believe that in the discussion about PEP 343 vs. Nick's PEP 3XX (http://members.iinet.net.au/~ncoghlan/public/pep-3XX.html, still awaiting PEP moderator approval I believe?) the main difference is that Nick proposes a way to inject an exception into a generator; and I've said a few times that I like that idea. I'd like to propose to make that a separate PEP, which can combine elements of PEP 288 and PEP 325. Summary: - g.throw(type, value, traceback) causes the specified exception to be thrown at the place where the generator g is currently suspended. If the generator catches the exception and yields another value, that is the return value of g.throw(). If it doesn't catch the exception, the throw() appears to raise the same exception passed it (it "falls through"). If the generator raises another exception (this includes the StopIteration produced when it returns) that exception is raised by the throw. In summary, throw() behaves like next() except it raises an exception at the place of the yield. If the generator is already in the closed state, throw() just raises the exception it was passed without going through the generator. - There's a new exception, GeneratorExit, which can be thrown to cause a generator to clean up. A generator should not yield a value in response to this exception. - g.close() throws a GeneratorExit exception in the generator, and catches it (so g.close() itself does not raise an exception). g.close() is idempotent -- if the generator is already closed, it is a no-op. If the generator, against the rules, yields another value, it is nevertheless marked closed. - When a generator is GC'ed, its close() method is called (which is a no-op if it is already closed). That's it! With this, we can write the decorator from Nick's PEP 3XX and the generator examples in PEP 343 can be rewritten to have a try/finally clause around the yield statement. Oh, somewhere it should be stated that yield without an expression is equivalent to yield None. PEP 342 ("continue EXPR") already implies that, so we don't have to write a separate PEP for it. I also propose to go with the alternative in PEP 342 of using next() rather than __next__() -- generators will have methods next(), throw(), and close(). -- --Guido van Rossum (home page: http://www.python.org/~guido/)
On Wednesday 18 May 2005 17:39, Guido van Rossum wrote:
I believe that in the discussion about PEP 343 vs. Nick's PEP 3XX (http://members.iinet.net.au/~ncoghlan/public/pep-3XX.html, still awaiting PEP moderator approval I believe?) the main difference is that Nick proposes a way to inject an exception into a generator; and I've said a few times that I like that idea.
I'd like to propose to make that a separate PEP, which can combine elements of PEP 288 and PEP 325. Summary:
- g.throw(type, value, traceback) causes the specified exception to be thrown at the place where the generator g is currently suspended. .. - There's a new exception, GeneratorExit, which can be thrown to cause a generator to clean up. A generator should not yield a value in response to this exception.
- g.close() throws a GeneratorExit exception in the generator, and catches it (so g.close() itself does not raise an exception). g.close() is idempotent -- if the generator is already closed, it is a no-op. If the generator, against the rules, yields another value, it is nevertheless marked closed.
+1 We're currently using python generators to handle concurrency in a single thread, and this allows a simple scheduler to have a clean way of sending generators a generator shutdown message. (Currently we have to do it another way that is specific to the style of wrapped generator) If you need a volunteer to code this - should it go through, I'm willing to have a go at this. (I can't do this though for 3 weeks or so though due to a crunch at work, and I might be in over my head in offering this.) Michael. -- Michael Sparks, Senior R&D Engineer, Digital Media Group Michael.Sparks@rd.bbc.co.uk, http://kamaelia.sourceforge.net/ British Broadcasting Corporation, Research and Development Kingswood Warren, Surrey KT20 6NP This e-mail may contain personal views which are not the views of the BBC.
At 09:39 AM 5/18/2005 -0700, Guido van Rossum wrote:
- g.throw(type, value, traceback) causes the specified exception to be thrown at the place where the generator g is currently suspended. If the generator catches the exception and yields another value, that is the return value of g.throw(). If it doesn't catch the exception, the throw() appears to raise the same exception passed it (it "falls through"). If the generator raises another exception (this includes the StopIteration produced when it returns) that exception is raised by the throw. In summary, throw() behaves like next() except it raises an exception at the place of the yield. If the generator is already in the closed state, throw() just raises the exception it was passed without going through the generator.
- There's a new exception, GeneratorExit, which can be thrown to cause a generator to clean up. A generator should not yield a value in response to this exception.
- g.close() throws a GeneratorExit exception in the generator, and catches it (so g.close() itself does not raise an exception). g.close() is idempotent -- if the generator is already closed, it is a no-op. If the generator, against the rules, yields another value, it is nevertheless marked closed.
- When a generator is GC'ed, its close() method is called (which is a no-op if it is already closed).
That's it! With this, we can write the decorator from Nick's PEP 3XX and the generator examples in PEP 343 can be rewritten to have a try/finally clause around the yield statement.
Oh, somewhere it should be stated that yield without an expression is equivalent to yield None. PEP 342 ("continue EXPR") already implies that, so we don't have to write a separate PEP for it. I also propose to go with the alternative in PEP 342 of using next() rather than __next__() -- generators will have methods next(), throw(), and close().
And there was much rejoicing in the land of the co-routiney people. :) +1000. Should this maybe just be added to PEP 342? To me, PEP 342 has always seemed incomplete without ways to throw() and close(), but that could easily be just me. In any case I'd expect the implementation of 'next(arg)' to have some overlap with the implementation of 'throw()'. Also, if the generator yields a value upon close(), shouldn't that throw a runtime error? Otherwise, you have no way to know the generator's exception handling is broken.
[Phillip J. Eby]
And there was much rejoicing in the land of the co-routiney people. :) +1000.
Should this maybe just be added to PEP 342? To me, PEP 342 has always seemed incomplete without ways to throw() and close(), but that could easily be just me. In any case I'd expect the implementation of 'next(arg)' to have some overlap with the implementation of 'throw()'.
Maybe, but on the other hand this idea can be done independently from PEP 342. After the "monster-PEP" 340, I'd rather break proposals up in small parts.
Also, if the generator yields a value upon close(), shouldn't that throw a runtime error? Otherwise, you have no way to know the generator's exception handling is broken.
Maybe. But then what should happen when this happens to close() invoked by the GC? I guess the same as when a __del__() method raises an exception -- print a traceback and go on. OK, works for me. -- --Guido van Rossum (home page: http://www.python.org/~guido/)
At 09:55 AM 5/18/2005 -0700, Guido van Rossum wrote:
And there was much rejoicing in the land of the co-routiney
[Phillip J. Eby] people. :) +1000.
Should this maybe just be added to PEP 342? To me, PEP 342 has always seemed incomplete without ways to throw() and close(), but that could easily be just me. In any case I'd expect the implementation of 'next(arg)' to have some overlap with the implementation of 'throw()'.
Maybe, but on the other hand this idea can be done independently from PEP 342. After the "monster-PEP" 340, I'd rather break proposals up in small parts.
Okay. Maybe we should just update PEP 325, then? It has much of the stuff that we'd want in the new PEP, such as the rationale. Your new proposal, AFAICT, is just a simple extension of the PEP 325 protocol (i.e., adding 'throw()'), along with some decisions to resolve its open issues. Even the addition of 'throw()' seems tacitly approved by this bit at the end: """Were PEP 288 implemented, Exceptions Semantics for close could be layered on top of it""" So at this point it seems your proposal is just nailing down specifics for the open parts of PEP 325.
[Phillip J. Eby]
Okay. Maybe we should just update PEP 325, then? It has much of the stuff that we'd want in the new PEP, such as the rationale. Your new proposal, AFAICT, is just a simple extension of the PEP 325 protocol (i.e., adding 'throw()'), along with some decisions to resolve its open issues. Even the addition of 'throw()' seems tacitly approved by this bit at the end:
"""Were PEP 288 implemented, Exceptions Semantics for close could be layered on top of it"""
So at this point it seems your proposal is just nailing down specifics for the open parts of PEP 325.
Or PEP 288? That has throw() (albeit with a different signature). I could do without the attributes though (PEP 342 provides a much better solution IMO). If either of those PEP authors feels like updating their PEP, they have my blessings! I probably won't get to writing my own for a few more days. -- --Guido van Rossum (home page: http://www.python.org/~guido/)
So at this point it seems your proposal is just nailing down specifics for the open parts of PEP 325.
Or PEP 288? That has throw() (albeit with a different signature). I could do without the attributes though (PEP 342 provides a much better solution IMO).
If either of those PEP authors feels like updating their PEP, they have my blessings! I probably won't get to writing my own for a few more days.
Okay, I volunteer to recast PEP 288 to encompass your combined proposal. Will tackle it in the morning. Raymond
At 01:28 PM 5/18/2005 -0400, Raymond Hettinger wrote:
Okay. Maybe we should just update PEP 325, then?
-1.
Keep this separate.
Have you read PEP 325 lately? Mostly the change would consist of deleting rejected options or moving them to a rejected options section. The only other change would be adding a short section stating how throw() would work and that it's being made public to support the future use of generators as flow-control templates. A new PEP would have to copy, reinvent, or reference large chunks of PEP 325, resulting in either redundancy or excess complexity. Or are you suggesting a new PEP for throw(), containing *only* an explanation of its semantics, and then modifying PEP 325 to indicate that it will be implemented using the new PEP's 'throw()'? That's about the only scenario that makes sense to me for adding a new PEP, because PEP 325 is already pretty darn complete with respect to close() and GC.
Should this maybe just be added to PEP 342? To me, PEP 342 has always seemed incomplete without ways to throw() and close(), but that could easily be just me. In any case I'd expect the implementation of 'next(arg)' to have some overlap with the implementation of 'throw()'.
Maybe, but on the other hand this idea can be done independently from PEP 342. After the "monster-PEP" 340, I'd rather break proposals up in small parts.
+1 I want this as a separate PEP. It is a straight-forward solution to long standing issues. I would rather not have it contaminated with distracting issues and co-routine dreams. Raymond
I'd like to propose to make that a separate PEP, which can combine elements of PEP 288 and PEP 325.
+1 Overall, the combined PEP proposal looks pretty good.
- g.throw(type, value, traceback) causes the specified exception to be thrown at the place where the generator g is currently suspended.
Are the value and traceback arguments optional as they are with the current raise statement? If they are optional, what would the default be? I think the preferred choice is to have the call to the throw method be the anchor point. That makes sense in a traceback so you can see who threw the exception. The alternative is to have the generator resumption point be the anchor. That is closer to the notion that throw(ex) is equivalent to a "raise ex" following the last yield. This probably isn't the way to go but the PEP should address it explicitly.
If the generator raises another exception (this includes the StopIteration produced when it returns) that exception is raised by the throw. In summary, throw() behaves like next() except it raises an exception at the place of the yield.
The parallel to next() makes this easy to understand, learn, and implement. However, there are some disadvantages to passing through a StopIteration. It means that throw() calls usually need to be wrapped in a try/except or that a generator's exception handler would terminate with a "yield None" where a "return" would be more natural. As a example, it is a bit painful to simulate the effects of g.close() using g.throw(GeneratorExit).
That's it! With this, we can write the decorator from Nick's PEP 3XX and the generator examples in PEP 343 can be rewritten to have a try/finally clause around the yield statement.
Yea! This is very nice.
Oh, somewhere it should be stated that yield without an expression is equivalent to yield None. PEP 342 ("continue EXPR") already implies that, so we don't have to write a separate PEP for it.
Right.
I also propose to go with the alternative in PEP 342 of using next() rather than __next__() -- generators will have methods next(), throw(), and close().
+0 The switch from __next__() to next() is attractive but not essential to the proposal. Besides a small cost to backwards compatability, it introduces yet another new/old style distinction where we have to keep both forms in perpetuity. Raymond
[Raymond Hettinger]
Are the value and traceback arguments optional as they are with the current raise statement? If they are optional, what would the default be? I think the preferred choice is to have the call to the throw method be the anchor point. That makes sense in a traceback so you can see who threw the exception.
AFAI throw() is concerned, the defaults are None. The raise statement does something sane when the second and/or third arg are None (the first can't be though).
The alternative is to have the generator resumption point be the anchor. That is closer to the notion that throw(ex) is equivalent to a "raise ex" following the last yield. This probably isn't the way to go but the PEP should address it explicitly.
It's actually kind of tricky since the exception will come *back* to the throw point anyway. I think the traceback ought to start at the resumption point by default.
If the generator raises another exception (this includes the StopIteration produced when it returns) that exception is raised by the throw. In summary, throw() behaves like next() except it raises an exception at the place of the yield.
The parallel to next() makes this easy to understand, learn, and implement. However, there are some disadvantages to passing through a StopIteration. It means that throw() calls usually need to be wrapped in a try/except or that a generator's exception handler would terminate with a "yield None" where a "return" would be more natural. As a example, it is a bit painful to simulate the effects of g.close() using g.throw(GeneratorExit).
Doesn't bother me; the main use case is in the do_template (or with_template) decorator. Since it must support both raising an exception and returning a value, we're pretty much forced to catch the exception (unless we just want to pass it through, which is actually a reasonable use case).
I also propose to go with the alternative in PEP 342 of using next() rather than __next__() -- generators will have methods next(), throw(), and close().
+0 The switch from __next__() to next() is attractive but not essential to the proposal. Besides a small cost to backwards compatability, it introduces yet another new/old style distinction where we have to keep both forms in perpetuity.
Right. PBP. :-) -- --Guido van Rossum (home page: http://www.python.org/~guido/)
[Raymond Hettinger]
Are the value and traceback arguments optional as they are with the current raise statement? If they are optional, what would the default be? I think the preferred choice is to have the call to the throw method be the anchor point. That makes sense in a traceback so you can see who threw the exception.
AFAI throw() is concerned, the defaults are None. The raise statement does something sane when the second and/or third arg are None (the first can't be though).
The alternative is to have the generator resumption point be the anchor. That is closer to the notion that throw(ex) is equivalent to a "raise ex" following the last yield. This probably isn't the way to go but the PEP should address it explicitly.
It's actually kind of tricky since the exception will come *back* to the throw point anyway. I think the traceback ought to start at the resumption point by default.
If the generator raises another exception (this includes the StopIteration produced when it returns) that exception is raised by the throw. In summary, throw() behaves like next() except it raises an exception at the place of the yield.
The parallel to next() makes this easy to understand, learn, and implement. However, there are some disadvantages to passing through a StopIteration. It means that throw() calls usually need to be wrapped in a try/except or that a generator's exception handler would terminate with a "yield None" where a "return" would be more natural. As a example, it is a bit painful to simulate the effects of g.close() using g.throw(GeneratorExit).
Doesn't bother me; the main use case is in the do_template (or with_template) decorator. Since it must support both raising an exception and returning a value, we're pretty much forced to catch the exception (unless we just want to pass it through, which is actually a reasonable use case).
I also propose to go with the alternative in PEP 342 of using next() rather than __next__() -- generators will have methods next(), throw(), and close().
+0 The switch from __next__() to next() is attractive but not essential to the proposal. Besides a small cost to backwards compatability, it introduces yet another new/old style distinction where we have to keep both forms in perpetuity.
Right. PBP. :-)
FWIW, I'm in agreement with everything. I hope this one gets accepted. Please do put it in a separate PEP. Raymond
At 01:24 PM 5/18/2005 -0400, Raymond Hettinger wrote:
- g.throw(type, value, traceback) causes the specified exception to be thrown at the place where the generator g is currently suspended.
Are the value and traceback arguments optional as they are with the current raise statement? If they are optional, what would the default be? I think the preferred choice is to have the call to the throw method be the anchor point. That makes sense in a traceback so you can see who threw the exception.
The alternative is to have the generator resumption point be the anchor. That is closer to the notion that throw(ex) is equivalent to a "raise ex" following the last yield. This probably isn't the way to go but the PEP should address it explicitly.
My use case for throw() calls for the latter option; i.e., the exception is raised by the yield expression at the resumption point. Keep in mind that if the exception passes out of the generator, the throw() call will show in the traceback anyway. It's unlikely the generator itself will inspect the traceback and need to see the throw() call as if it were nested.
If the generator raises another exception (this includes the StopIteration produced when it returns) that exception is raised by the throw. In summary, throw() behaves like next() except it raises an exception at the place of the yield.
The parallel to next() makes this easy to understand, learn, and implement. However, there are some disadvantages to passing through a StopIteration. It means that throw() calls usually need to be wrapped in a try/except or that a generator's exception handler would terminate with a "yield None" where a "return" would be more natural. As a example, it is a bit painful to simulate the effects of g.close() using g.throw(GeneratorExit).
I don't see this as a big problem, personally, but that's because all of my use cases for throw() will be using only one piece of code that calls throw(), and that code will be overall simplified by the availability of throw(). It's also easy to write a wrapper for control-flow "signals" of the kind you used in PEP 288: def send(gen, *exc): try: return gen.throw(*exc) except StopIteration: pass
Phillip J. Eby wrote:
My use case for throw() calls for the latter option; i.e., the exception is raised by the yield expression at the resumption point. Keep in mind that if the exception passes out of the generator, the throw() call will show in the traceback anyway. It's unlikely the generator itself will inspect the traceback and need to see the throw() call as if it were nested.
There mightn't be much choice anyway. If the frame making the call to throw() were to be made the starting point for the traceback, and the exception propagated back to the throw, something would try to put the same frame in the traceback twice, which can't work since it's a linked list. -- Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg.ewing@canterbury.ac.nz +--------------------------------------+
[Guido van Rossum]
I believe that in the discussion about PEP 343 vs. Nick's PEP 3XX (http://members.iinet.net.au/~ncoghlan/public/pep-3XX.html, still awaiting PEP moderator approval I believe?) ...
Nick hasn't submitted it for a PEP number yet. -- David Goodger <http://python.net/~goodger>
Guido van Rossum wrote:
- When a generator is GC'ed, its close() method is called (which is a no-op if it is already closed).
Does this mean that all generators will be ineligible for cyclic garbage collection (since they implicitly have something equivalent to a __del__ method)? Other than that, all this looks good. -- Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg.ewing@canterbury.ac.nz +--------------------------------------+
At 06:09 PM 5/19/2005 +1200, Greg Ewing wrote:
Guido van Rossum wrote:
- When a generator is GC'ed, its close() method is called (which is a no-op if it is already closed).
Does this mean that all generators will be ineligible for cyclic garbage collection (since they implicitly have something equivalent to a __del__ method)?
No, since it's implemented in C. (The C equivalent to __del__ does not interfere with cyclic GC.)
On May 19, 2005, at 8:43 AM, Phillip J. Eby wrote:
At 06:09 PM 5/19/2005 +1200, Greg Ewing wrote:
Guido van Rossum wrote:
- When a generator is GC'ed, its close() method is called (which is a no-op if it is already closed).
Does this mean that all generators will be ineligible for cyclic garbage collection (since they implicitly have something equivalent to a __del__ method)?
No, since it's implemented in C. (The C equivalent to __del__ does not interfere with cyclic GC.)
But you're missing the point -- there's a *reason* that __del__ interferes with cyclic GC. It doesn't just do it for the heck of it! You can't simply have the C delete call into python code...the objects the generator has references to may be invalid objects already because they've been cleared to break a cycle. If you want to execute python code on collection of a generator, it must be done via __del__, or else it'll be horribly, horribly broken. James
At 12:36 PM 5/19/2005 -0400, James Y Knight wrote:
On May 19, 2005, at 8:43 AM, Phillip J. Eby wrote:
At 06:09 PM 5/19/2005 +1200, Greg Ewing wrote:
Guido van Rossum wrote:
- When a generator is GC'ed, its close() method is called (which is a no-op if it is already closed).
Does this mean that all generators will be ineligible for cyclic garbage collection (since they implicitly have something equivalent to a __del__ method)?
No, since it's implemented in C. (The C equivalent to __del__ does not interfere with cyclic GC.)
But you're missing the point -- there's a *reason* that __del__ interferes with cyclic GC. It doesn't just do it for the heck of it! You can't simply have the C delete call into python code...the objects the generator has references to may be invalid objects already because they've been cleared to break a cycle. If you want to execute python code on collection of a generator, it must be done via __del__, or else it'll be horribly, horribly broken.
Eeyowch. Good point. OTOH, the only way a generator-iterator can become part of a cycle is via an action taken outside the generator. (E.g. passing it into itself via 'continue', creating a link from one of its arguments to it, etc.) So, it's probably not a terrible limitation in practice.
[Phillip J. Eby]the only way a generator-iterator can become
part of a cycle is via an action taken outside the generator. (E.g. passing it into itself via 'continue', creating a link from one of its arguments to it, etc.) So, it's probably not a terrible limitation in practice.
It's enough to store a reference to the generator in a global (or in anything that's reachable from a global). The generator's frame's f_globals pointer then ensures the cycle. Throwing an exception also provides ample opportunity for creating cycles, since the frame hold a reference to the most recent traceback. Ironically, throwing an exception with a traceback into a generator is likely to cause a cycle because the traceback likely references the throwing frame, which certainly has a reference to the generator... -- --Guido van Rossum (home page: http://www.python.org/~guido/)
At 10:09 AM 5/19/2005 -0700, Guido van Rossum wrote:
[Phillip J. Eby]the only way a generator-iterator can become
part of a cycle is via an action taken outside the generator. (E.g. passing it into itself via 'continue', creating a link from one of its arguments to it, etc.) So, it's probably not a terrible limitation in practice.
It's enough to store a reference to the generator in a global (or in anything that's reachable from a global). The generator's frame's f_globals pointer then ensures the cycle.
Well, if the generator was defined in the same module, I suppose. This would probably only happen with short scripts, where the lack of GC is unlikely to be an issue. However, at least it would also be possible to explicitly close the generator, which wasn't possible before.
Throwing an exception also provides ample opportunity for creating cycles, since the frame hold a reference to the most recent traceback. Ironically, throwing an exception with a traceback into a generator is likely to cause a cycle because the traceback likely references the throwing frame, which certainly has a reference to the generator...
*head exploding* Double ouch. Wait a minute... those cycles don't include the generator, do they? Let me think. Frame A has a reference to the generator iterator, and invokes throw() (directly or indirectly) on it. Frame B, the generator frame, gets its f_back set to point to Frame A, but presumably that link is cleared on exit? (If it isn't, it probably should be). Anyway, frame B throws an exception, and the traceback is created. The traceback has a reference to frame B. We return to frame A, and add it to the traceback as well, and a reference to the traceback goes into the frame too. Hm. Still no cycle passing through the *generator iterator*, unless the generator's frame's f_back is still pointing to the frame that it was last called from. This holds even if the generator's frame holds the traceback that was current at the time of the error, because that traceback only includes the generator's frame, not the caller's frame. So, as long as the generator guarantees its frame's f_back is empty while the generator is not actually executing, the cycle should not include the generator, so it will still be GC'able. (Note, by the way, that this property is probably another good reason for using the yield expression as the source location for a throw()!)
[Phillip J. Eby]
Throwing an exception also provides ample opportunity for creating cycles, since the frame hold a reference to the most recent traceback. Ironically, throwing an exception with a traceback into a generator is likely to cause a cycle because the traceback likely references the throwing frame, which certainly has a reference to the generator...
*head exploding* Double ouch.
Wait a minute... those cycles don't include the generator, do they? Let me think. Frame A has a reference to the generator iterator, and invokes throw() (directly or indirectly) on it. Frame B, the generator frame, gets its f_back set to point to Frame A, but presumably that link is cleared on exit? (If it isn't, it probably should be).
Anyway, frame B throws an exception, and the traceback is created. The traceback has a reference to frame B. We return to frame A, and add it to the traceback as well, and a reference to the traceback goes into the frame too. Hm. Still no cycle passing through the *generator iterator*, unless the generator's frame's f_back is still pointing to the frame that it was last called from. This holds even if the generator's frame holds the traceback that was current at the time of the error, because that traceback only includes the generator's frame, not the caller's frame.
So, as long as the generator guarantees its frame's f_back is empty while the generator is not actually executing, the cycle should not include the generator, so it will still be GC'able. (Note, by the way, that this property is probably another good reason for using the yield expression as the source location for a throw()!)
Hm. The way I see it, as soon as a generator raises an exception, its frame is part of a cycle: the frame's f_exc_traceback points to the traceback object, and the traceback object's tb_frame points back to the frame. So that's a cycle right there. -- --Guido van Rossum (home page: http://www.python.org/~guido/)
At 10:48 AM 5/19/2005 -0700, Guido van Rossum wrote:
Hm. The way I see it, as soon as a generator raises an exception, its frame is part of a cycle: the frame's f_exc_traceback points to the traceback object, and the traceback object's tb_frame points back to the frame. So that's a cycle right there.
But that cycle doesn't include the generator-iterator object, and it's not a collectable cycle while the iterator still lives. Once the iterator itself goes away, that frame cycle is collectable. However, Tim's new post brings up a different issue: if the collector can't tell the difference between a cycle participant and an object that's only reachable from a cycle, then the mere existence of a generator __del__ will prevent the cycle collection of the entire traceback/frame system that includes a generator-iterator reference anywhere! And that's a pretty serious problem.
"Phillip J. Eby" <pje@telecommunity.com> writes:
However, Tim's new post brings up a different issue: if the collector can't tell the difference between a cycle participant and an object that's only reachable from a cycle,
Uh, that's not what he meant: />> class C: |.. def __del__(self): |.. print 'bye' \__ ->> a = [C()] ->> a.append(a) ->> del a ->> gc.collect() bye 1 Cheers, mwh -- Now this is what I don't get. Nobody said absolutely anything bad about anything. Yet it is always possible to just pull random flames out of ones ass. -- http://www.advogato.org/person/vicious/diary.html?start=60
At 07:18 PM 5/19/2005 +0100, Michael Hudson wrote:
"Phillip J. Eby" <pje@telecommunity.com> writes:
However, Tim's new post brings up a different issue: if the collector can't tell the difference between a cycle participant and an object that's only reachable from a cycle,
Uh, that's not what he meant:
/>> class C: |.. def __del__(self): |.. print 'bye' \__ ->> a = [C()] ->> a.append(a) ->> del a ->> gc.collect() bye 1
Now you've shaken my faith in Uncle Timmy. :) Seriously, he did *say*: """For example, it doesn't know the difference between an object that's in a trash cycle, and an object that's not in a trash cycle but is reachable only from trash cycles.""" So now I wonder what he *did* mean.
[Phillip J. Eby]
Now you've shaken my faith in Uncle Timmy. :)
Now, now, a mere technical matter is no cause for soul-damning heresy!
Seriously, he did *say*:
"""For example, it doesn't know the difference between an object that's in a trash cycle, and an object that's not in a trash cycle but is reachable only from trash cycles."""
So now I wonder what he *did* mean.
What I said, of course ;-) I hope my later email clarified it. gc knows which trash objects have __del__ methods, and which don't. That's all it needs to know so that a __del__ method on an object that's not in a trash cycle but is reachable only from trash cycles will get reclaimed (provided that no __del__ method on a predecessor object that's not in a trash cycle but is reachable only from trash cycles resurrects it). gc doesn't know whether the set of objects it _intends_ to call tp_clear on are or aren't in cycles, but all objects directly in __del__ free cycles are included in that set. That's enough to ensure that trash "hanging off of them" sees its refcounts fall to 0 as gc applies tp_clear to the objects in that set. Note that this set can mutate as gc goes along: calling tp_clear on one object can (also as a side effect of refcounts falling to 0) remove any number of other objects from that set (that's why I said "intends" above: there's no guarantee that gc will end up calling tp_clear on any object other than "the first" one in the set, where "the first" is utterly arbitrary now). If an object in a trash cycle has a __del__ method, this is why the cycle won't be reclaimed: all trash objects with __del__ methods, and everything transitively reachable from them, are moved to the "finalizers" list early on. If that happens to include a trash cycle C, then all of C ends up in the "finalizers" list, and no amount of tp_clear'ing on the objects that remain can cause the refcount on any object in C to fall to 0. gc has no direct knowledge of cycles in this case either.
[Phillip J. Eby]
... However, Tim's new post brings up a different issue: if the collector can't tell the difference between a cycle participant and an object that's only reachable from a cycle, then the mere existence of a generator __del__ will prevent the cycle collection of the entire traceback/frame system that includes a generator-iterator reference anywhere! And that's a pretty serious problem.
It's not that simple <wink>. If an object with a __del__ is not part of a cycle, but is reachable only from trash cycles, that __del__ does not inhibit garbage collection. Like: A<->B -> C -> D -> E where C, D and E have __del__, but A and B don't, and all are trash. Relatively early on, gc "moves" C, D, and E into a special "finalizers" list, and doesn't look at this list again until near the end. Then A.tp_clear() and B.tp_clear() are called in some order. As a *side effect* of calling B.tp_clear(), C's refcount falls to 0, and Python's normal refcount-based reclamation (probably) recovers all of C, D and E, and runs their __del__ methods. Note that refcount-based reclamation necessarily follows a DAG order: E is still intact when D.__del__ is called, and likewise D is still intact when C.__del__ is called. It's possible that C.__del__ will resurrect D and/or E, and that D.__del__ will resurrect E. In such cases, D and/or E's refcounts don't fall to 0, and their __del__ methods won't be called then. Cyclic gc doesn't force any of that, though -- it's all a side effect of the clear() in gcmodule.c's: if ((clear = op->ob_type->tp_clear) != NULL) { Py_INCREF(op); clear(op); Py_DECREF(op); In turn, one of A and B get reclaimed as a side effect of the Py_DECREF there -- it's one of the delights of gcmodule.c that if you don't know the trick, you can stare at it for hours and never discover where exactly it is anything gets released <0.9 wink>. In fact, it doesn't release anything directly -- "all it really does" now is break reference cycles, so that Py_DECREF can do its end-of-life thing.
[James Y Knigh]
But you're missing the point -- there's a *reason* that __del__ interferes with cyclic GC. It doesn't just do it for the heck of it! You can't simply have the C delete call into python code...the objects the generator has references to may be invalid objects already because they've been cleared to break a cycle. If you want to execute python code on collection of a generator, it must be done via __del__, or else it'll be horribly, horribly broken.
Thank you for reminding me -- that's indeed the reason, and it applies here. I think in the past I've unsuccessfully tried to argue that if a cycle contains exactly one object with a Python-invoking finalizer, that finalizer could be invoked before breaking the cycle. I still think that's a sensible proposal, and generators may be the use case to finally implement it. All this suggests that generators should indeed have a __del__() method which is synonymous with close() (I want close() to be the user-facing API). BTW I think that close() and __del__() should raise an exception when the throw(GeneratorExit) call doesn't end up either re-raising GeneratorExit or raising StopIteration. The framework for calling __del__() takes care of handling this exception (by printing and then ignoring it). Raymond take notice if you're still working on the PEP. -- --Guido van Rossum (home page: http://www.python.org/~guido/)
BTW I think that close() and __del__() should raise an exception when the throw(GeneratorExit) call doesn't end up either re-raising GeneratorExit or raising StopIteration. The framework for calling __del__() takes care of handling this exception (by printing and then ignoring it). Raymond take notice if you're still working on the PEP.
Got it. Raymond
[Guido]
... I think in the past I've unsuccessfully tried to argue that if a cycle contains exactly one object with a Python-invoking finalizer, that finalizer could be invoked before breaking the cycle. I still think that's a sensible proposal, and generators may be the use case to finally implement it.
You have argued it, and I've agreed with it. The primary hangup is that there's currently no code capable of doing it. gc currently determines the set of objects that must be part of cyclic trash, or reachable only from cyclic trash, but has no relevant knowledge beyond that. For example, it doesn't know the difference between an object that's in a trash cycle, and an object that's not in a trash cycle but is reachable only from trash cycles. In fact, it doesn't know anything about the cycle structure. That would require some sort of new SCC (strongly connected component) analysis. The graph derived from an arbitrary object graph by considering each SCC to be "a node" is necessarily a DAG (contains no cycles), and the general way to approach what you want here is to clear trash in a topological sort of the SCC DAG: so long as an SCC contains only one object that may execute Python code, it's safe to run that object's cleanup code first (for a meaning of "safe" that may not always coincide with "explainable" or "predictable" <0.9 wink>). gc would probably need to give up after the first such thingie is run, if any SCCs are reachable from the SCC X containing that thingie (gc can no longer be sure that successor SCCs _are_ still trash: they too are reachable from X, so may have been resurrected by the Python code X ran). There's currently no forcing at all of the order in which tp_clear gets called, and currently no analysis done sufficient to support forcing a relevant ordering.
Guido van Rossum wrote:
I believe that in the discussion about PEP 343 vs. Nick's PEP 3XX (http://members.iinet.net.au/~ncoghlan/public/pep-3XX.html, still awaiting PEP moderator approval I believe?)
It turns out my submission email took the scenic route, and I wasn't using a proper text editor so the formatting of the raw text version is messed up. Given that I need to clean that formatting up, I figure I might as well update it in light of recent discussions before I resubmit it.
the main difference is that Nick proposes a way to inject an exception into a generator; and I've said a few times that I like that idea.
If the current generator integration is dropped from PEP 343, I can rewrite my PEP to be *just* about the combination of PEP 288 and PEP 325 you describe, and use PEP 343 integration as a motivating use case. The alternative would be to just submit and immediately withdraw it (since the user defined statement semantics now match PEP 343, and I basically like the generator interface you are proposing here, there wouldn't be much left of my PEP except for the big 'Rejected Options' section giving my understanding of the reasons we didn't take up various options). <snip parts of the proposal I agree with completely>
- g.close() throws a GeneratorExit exception in the generator, and catches it (so g.close() itself does not raise an exception). g.close() is idempotent -- if the generator is already closed, it is a no-op. If the generator, against the rules, yields another value, it is nevertheless marked closed.
Can't we make the method raise a RuntimeError when the generator breaks the rules? Or should we just print a warning instead (like the deletion code does if __del__ raises an exception)?
- When a generator is GC'ed, its close() method is called (which is a no-op if it is already closed).
This is like giving it a __del__ method though, since it can now resurrect arbitrary objects from a cycle it is involved in. I think you made the right call elsewhere, when you convinced me that generator's shouldn't have a __del__ method any more than files should. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia --------------------------------------------------------------- http://boredomandlaziness.blogspot.com
participants (10)
-
David Goodger -
Greg Ewing -
Guido van Rossum -
James Y Knight -
Michael Hudson -
Michael Sparks -
Nick Coghlan -
Phillip J. Eby -
Raymond Hettinger -
Tim Peters