At 10:15 PM 6/18/2005 -0400, Phillip J. Eby wrote:
Okay, I think I see why you can't do it. You could guarantee that all relevant __del__ methods get called, but it's bloody difficult to end up with only unreachable items in gc.garbage afterwards. I think gc would have to keep a new list for items reachable from finalizers, that don't themselves have finalizers. Then, before creating gc.garbage, you walk the finalizers and call their finalization (__del__) methods. Then, you put any remaining items that are in either the finalizer list or the reachable-from-finalizers list into gc.garbage.
This approach might need a new type slot, but it seems like it would let us guarantee that finalizers get called, even if the object ends up in garbage as a result. In the case of generators, however, close() guarantees that the generator releases all its references, and so can no longer be part of a cycle. Thus, it would guarantee eventual cleanup of all generators. And, it would lift the general limitation on __del__ methods.
Hm. Sounds too good to be true. Surely if this were possible, Uncle Timmy would've thought of it already, no? Guess we'll have to wait and see what he thinks.
Or maybe not. After sleeping on it, I realized that the problems are all in when and how often __del__ is called. The idea I had above would end up calling __del__ twice on non-generator objects. For generators it's not a problem because the first call ends up ensuring that the second call is a no-op. However, the *order* of __del__ calls makes a difference, even for generators. What good is a finally: clause if all the objects reachable from it have been finalized already, anyway? Ultimately, I'm thinking that maybe we were right not to allow try-finally to cross yield boundaries in generators. It doesn't seem like you can guarantee anything about the behavior in the presence of cycles, so what's the point? For a while I played around with the idea that maybe we could still support 'with:' in generators, though, because to implement that we could make frames call __exit__ on any pending 'with' blocks as part of their tp_clear operation. This would only work, however, if the objects with __exit__ methods don't have any references back to the frame. In essence, you'd need a way to put the __exit__ objects on a GC-managed list that wouldn't run until after all the tp_clear calls had finished. But even that is tough to make guarantees about. For example, can you guarantee in that case that a generator's 'with:' blocks are __exit__-ed in the proper order? Really, if we do allow 'with' and 'try-finally' to surround yield, I think we're going to have to tell people that it only works if you use a with or try-finally in some non-generator code to ensure that the generator.close() gets called, and that if you end up creating a garbage cycle, we either have to let it end up in gc.garbage, or just not execute its finally clause or __exit__ methods. Of course, this sort of happens right now for other things with __del__; if it's part of a cycle the __del__ method never gets called. The only difference is that it hangs around in gc.garbage, doing nothing useful. If it's garbage, it's not reachable from anywhere else, so it does nobody any good to have it around. So, maybe we should just say, "sucks to be you" and tp_clear anything that we'd otherwise have put in gc.garbage. :) In other words, since we're not going to call those __del__ methods anyway, maybe it just needs to be part of the language semantics that __del__ isn't guaranteed to be called, and a garbage collector that can't find a safe way to call it, doesn't have to. tp_dealloc for classic classes and heap types could then just skip calling __del__ if they've already been cleared... oh wait, how do you know you've been cleared? Argh. Another nice idea runs up on the rocks of reality. On the other hand, if you go ahead and run __del__ after tp_clear, the __del__ method will quickly run afoul of an AttributeError and die with only a minor spew to sys.stderr, thus encouraging people to get rid of their silly useless __del__ methods on objects that normally end up in cycles. :)