Weakref.ref callbacks and eliminating __del__ methods

[Mike C. Fletcher]

Tim Peters wrote:

>[Mike C. Fletcher]
>  
>
>>I'm looking at rewriting parts of Twisted and TwistedSNMP to eliminate
>>__del__ methods (and the memory leaks they create).
>>    
>>
>A worthy goal!
>  
>
Well, as of now it seems to have eliminated the last leaks in 
TwistedSNMP, and that's likely going to eliminate the last of our leaks 
in our products, so yes, I suppose it was :) .

>A callback is strongly referenced from the weakref(s) containing it.
>  
>
Thanks.

>It would really help to give a minimal example of self-contained
>executable code.  I'm not sure how you're using this code, and
>guessing takes too long.
>  
>
Sorry about that.  I was expecting a theoretical problem, not a 
practical one, so didn't think to provide a practical example.

>The "2" there is important, just because this is an interactive
>session.  The point of it was to stop `_` from holding a reference to
>the created weakref.  In your code
>
>        weakref.ref( self, self.close )
>
>the weakref itself becomes trash immediately after it's created, so
>the weakref is thrown away entirely (including its strong reference to
>self.close, and its weak reference to self) right after that line
>ends.  Of course callbacks aren't *supposed* to be called when the
>weakref itself goes away, they're supposed to be called when the thing
>being weakly referenced goes away.  So this all looks vanilla and
>inevitable to me -- the callback is never invoked because the weakref
>itself goes away long before self goes away.
>  
>
Alex already scooped you on this "smack yourself on the head, Mikey".  I 
was assuming the weakref object was a proxy for a reference in an 
internal structure; I was expecting a list of weak references that was 
nothing but a series of functions to call after finalising the object.  
My bad.

>I'm pretty sure it's just because there's nothing here to keep the
>weakref itself alive after it's created.  You could try, e.g.,
>
>    self.wr = weakref.ref( self, self.close )
>
>to keep it alive, but things get fuzzy then if self becomes part of
>cyclic trash.
>  
>
Yes, so it requires an external storage mechanism and code to clean that 
up... ick.  __del__ looks cleaner and cleaner by comparison.  You'd 
think I would have noticed the huge tables of weakref objects in 
PyDispatcher as I was working in there...

>>I can work around it in this particular case by defining a __del__ on
>>the Closer, but that just fixes this particular instance (and leaves
>>just as many __del__'s hanging around).  I'm wondering if there's a
>>ready recipe that can *always* replace a __del__'s operation?
>>    
>>
>
>In the presence of cycles it gets tricky; there's a lot of
>more-or-less recent words (less than a year old <wink>) about that on
>python-dev.  It gets complicated quickly, and it seems nobody can make
>more time to think about it.
>  
>
Sigh, guess I should stop listening to rumours just because they are 
saying something about that for which I yearn.

>>I know I heard a rumour somewhere about Uncle Timmy wanting to eliminate
>>__del__ in 2.5 or thereabouts,
>>    
>>
>
>Python 3 at the earliest.  That's the earliest everything nobody can
>make time for lands <0.5 wink>.
>  
>
Well, we darn well better solve it by then!  Don't want memory leaks 
when we're hard-wired into someone's brain.

>There are unresolved issues about how to get all this stuff to work
>sanely.  The driving principle behind cyclic-trash weakref endcases
>right now is "do anything defensible that won't segfault".
>
>I'll note that one fairly obvious pattern works very well for weakrefs
>and __del__ methods (mutatis mutandis):  don't put the __del__ method
>in self, put it in a dead-simple object hanging *off* of self.  Like
>the simple:
>
>class BTreeCloser:
>    def __init__(self, btree):
>        self.btree = btree
>
>    def __del__(self):
>        if self.btree:
>            self.btree.close()
>            self.btree = None
>  
>
Yes, this was the "work around" I'd implemented (well, with __call__ 
instead and del just calling it).  Of course, that didn't actually 
eliminate any __del__ methods, but oh well, if we're not losing those 
until 3.x it doesn't really matter :) .

The fix for the Deferred in Twisted got a little hideous (there the 
__del__ is poking around into lots of different attributes of the 
Deferred.  To hack that I created a descriptor class that forwards a 
variable to the held object and wrapped each of the needed variables in 
one of those... I'm going to have to find something a little less 
baroque if I'm going to get it into Twisted... really, it doesn't look 
like any of it's necessary, it's just logging an error if the Deferred 
ended on a Failure.  Something should be done though, leaking memory 
from a core object in the framework is just a PITA.

>When a weakref and its weak referent are both in cyclic trash, Python
>currently says "the order in which they die is undefined, so we'll
>pretend the weakref dies first", and then, as at the start of this
>msg, the callback is never invoked.  
>
So external storage of a weakref is basically a requirement to make it 
useful. Hmm.

>The problem this avoids is that
>the callback may itself be part of cyclic trash too, in which case
>it's possible for the callback to resurrect anything else in cyclic
>trash, including the weakly-referenced object associated with the
>callback.  But by hypothesis in this case, that object is also in
>cyclic trash, and horrible things can happen if a partially destructed
>object gets resurrected.
>  
>
Reasonable I suppose.

>So the real rule right now is that, if you want a guarantee that a
>callback on a weakly referenced object gets invoked, you have to write
>your code to guarantee that the referent becomes trash before its
>weakref becomes trash.  (And note that if you do, the callback
>necessarily will be alive too, because a weakref does in fact hold a
>strong reference to its callback.)
>  
>
Guess I'll go back to the sub-object approach for all things and merely 
dream of a far off weakref utopia free of all __del__ methods.

Thanks, Tim,
Mike

________________________________________________
  Mike C. Fletcher
  Designer, VR Plumber, Coder
  http://www.vrplumber.com
  http://blog.vrplumber.com