[Python-Dev] reference leaks, __del__, and annotations

[Phillip J. Eby]

At 08:14 AM 3/31/2006, Tim Peters wrote:
>[Phillip J. Eby]
> > ...
> > As Tim suggested, it'd be better to have the code be generator-specific, at
> > least for now.  That had actually been my original plan, to make it
> > generator-specific, but I was afraid of breaking encapsulation in the
> > garbage collector by having it know about generators.
>
>It sucks in a way, but so would adding yet another new slot just for
>(at present, and possibly forever) making gc and generators play nicer
>together.  "Practicality beats purity" here.

I'm trying to figure out how to implement this now, and running into 
a bit of a snag.  It's easy enough for gcmodule.c to check if an 
object is a generator, but I'm not sure how safe the dynamic check 
actually is, since it depends on the generator's state.  In 
principle, running other finalizers could cause the generator's state 
to change from a finalizer being required to not being required, or 
vice versa.  Could this mess up the GC process?  It seems to me that 
it's safe for a generator to say, "yes, I need finalization", because 
if it later turns out not to, it's just a waste. But if the generator 
says, "no, I don't need finalization", and then later turns out to 
need it, doesn't that leave an opportunity to screw things up if the 
GC does anything other than immediately clear the generator?

As best I can tell, the only things that could cause arbitrary Python 
code to run, that could possibly result in generator state changes, 
are structure traversal and weakref callback handling.  It's probably 
not sane for anybody to have structure traversal run arbitrary Python 
code, so I'm going to ignore that for the sake of my own 
sanity.  :)  Weakref callbacks are tougher, though; it seems possible 
that you could have one of those cause a generator to be advanced to 
a point where it now needs finalization.

OTOH, if such a generator could be advanced by the callback, then 
wouldn't that mean the generator is reachable, and ergo, not 
garbage?  That is, since only reachable weakref callbacks are run, 
they must by definition be unable to access any generator that 
declared itself finalizer-free.

It does seem possible you could end up with a situation where an 
object with a finalizer is called after a generator it references is 
torn down, but that circumstance can occur in earlier versions of 
Python anyway, and in fact this behavior would be consistent.

Okay, I *think* I've convinced myself that a dynamic state check is 
OK, but I'm hoping somebody with more GC experience can check my 
reasoning here for holes.