Confusion with weakref, __del__ and threading

[Rhamphoryncus]

On Jun 10, 8:15 pm, George Sakkis <george.sak... at gmail.com> wrote:
> I'm baffled with a situation that involves:
> 1) an instance of some class that defines __del__,
> 2) a thread which is created, started and referenced by that instance,
> and
> 3) a weakref proxy to the instance that is passed to the thread
> instead of 'self', to prevent a cyclic reference.
>
> This probably sounds like gibberish so here's a simplified example:
>
> ==========================================
>
> import time
> import weakref
> import threading
>
> num_main = num_other = 0
> main_thread = threading.currentThread()
>
> class Mystery(object):
>
>     def __init__(self):
>         proxy = weakref.proxy(self)
>         self._thread = threading.Thread(target=target, args=(proxy,))
>         self._thread.start()
>
>     def __del__(self):
>         global num_main, num_other
>         if threading.currentThread() is main_thread:
>             num_main += 1
>         else:
>             num_other += 1
>
>     def sleep(self, t):
>         time.sleep(t)
>
> def target(proxy):
>     try: proxy.sleep(0.01)
>     except weakref.ReferenceError: pass
>
> if __name__ == '__main__':
>     for i in xrange(1000):
>         Mystery()
>     time.sleep(0.1)
>     print '%d __del__ from main thread' % num_main
>     print '%d __del__ from other threads' % num_other
>
> ==========================================
>
> When I run it, I get around 950 __del__ from the main thread and the
> rest from non-main threads. I discovered this accidentally when I
> noticed some ignored AssertionErrors caused by a __del__ that was
> doing "self._thread.join()", assuming that the current thread is not
> self._thread, but as it turns out that's not always the case.
>
> So what is happening here for these ~50 minority cases ? Is __del__
> invoked through the proxy ?

The trick here is that calling proxy.sleep(0.01) first gets a strong
reference to the Mystery instance, then holds that strong reference
until it returns.

If the child thread gets the GIL before __init__ returns it will enter
Mystery.sleep, then the main thread will return from Mystery.__init__
and release its strong reference, followed by the child thread
returning from Mystery.sleep, releasing its strong reference, and (as
it just released the last strong reference) calling Mystery.__del__.

If the main thread returns from __init__ before the child thread gets
the GIL, it will release the only strong reference to the Mystery
instance, causing it to clear the weakref proxy and call __del__
before the child thread ever gets a chance.  If you added counters to
the target function you should see them match the counters of the
__del__ function.

Incidentally, += 1 isn't atomic in Python.  It is possible for updates
to be missed.