[New-bugs-announce] [issue40312] Weakref callbacks running before finalizers in GC collection

[Allan Feldman]

New submission from Allan Feldman <allan.d.feldman at gmail.com>:

Our team is making use of a weakref.WeakValueDictionary() that is accessed through the finalizer of a class. We observed that in Python 3 occasionally values that are directly referenced by an object being finalized were missing from the WeakValueDictionary.


Example:

    import weakref
    cache = weakref.WeakValueDictionary()

    class Foo(object):
        pass


    class Bar(object):
        def __init__(self, foo):
            self.foo = foo
            cache['foo'] = foo

        def __del__(self):
            del cache['foo']

    bar = Bar(Foo())
    del bar


Upon further investigation, we realized that this had to do with the weakref callback within WeakValueDictionary being called (removing the key from the dict) before the finalizer for Foo was called.

Reproduction:

    import gc
    import weakref

    cache = weakref.WeakValueDictionary()


    class Foo(object):
        pass


    class Bar(object):
        def __init__(self, foo):
            # Force a reference cycle to run del only on gc.collect
            self._self = self
            self.foo = foo
            cache["foo"] = foo

        def __del__(self):
            # foo is missing from the cache because the weakref callback has
            # already run. KeyError is raised.
            del cache["foo"]


    bar = Bar(Foo())
    del bar

    gc.collect()


Expected behavior:

The weakref callback should only be called when the object is known to be deleted (after the finalizer runs). Running weakref callbacks before then means that the weakref callback can run on objects being ressurected by the finalizer.

Example:

    import gc
    import weakref


    class ForeverObject(object):
        def __init__(self, circular):
            # Introduce a circular reference so that gc must collect the object
            if circular:
                self._self = self

        def __del__(self):
            global o
            o = self


    def callback(wr):
        print("callback running", wr)


    for circular in (True, False):
        print("------- Circular reference:", circular, "-------")
        o = ForeverObject(circular)
        wr = weakref.ref(o, callback)
        del o
        gc.collect()
        print("--------------")


Note: Python 2.7 appears to have the opposite behavior - weakref callbacks are only invoked when dealloc occurs outside of gc. The Python 2.7 behavior hasn't yet been investigated.


If the expected behavior above is confirmed, I would be happy to submit a patch for this issue!

----------
components: Interpreter Core
messages: 366675
nosy: a-feld
priority: normal
severity: normal
status: open
title: Weakref callbacks running before finalizers in GC collection
type: behavior
versions: Python 2.7, Python 3.5, Python 3.6, Python 3.7, Python 3.8, Python 3.9

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue40312>
_______________________________________