__del__ is not called after creating a new reference
Hello, I already tried to ask on python-list, see https://mail.python.org/pipermail/python-list/2017-March/720037.html but it seems that this list is not for technical questions. Let me resend my question to python-dev. Please tell me if I should not spam this list with newbiesh questions, and thanks in advance. --------------------------------------------------------------------------- I started to learn python a few days ago and I am trying to understand what __del__() actually does. https://docs.python.org/3/reference/datamodel.html says: object.__del__(self) ... Note that it is possible (though not recommended!) for the __del__() method to postpone destruction of the instance by creating a new reference to it. It may then be called at a later time when this new reference is deleted. However, this trivial test-case class C: def __del__(self): print("DEL") global X X = self C() print(X) X = 0 print(X) shows that __del__ is called only once, it is not called again after "X = 0": DEL <__main__.C object at 0x7f067695f4a8> 0 (Just in case, I verified later that this object actually goes away and its memory is freed, so the problem is not that it still has a reference). I've cloned https://github.com/python/cpython.git and everything looks clear at first glance (but let me repeat that I am very new to python): PyObject_CallFinalizerFromDealloc() calls PyObject_CallFinalizer() which finally calls "__del__" method in slot_tp_finalize(), then it notices that "X = self" creates the new reference and does: /* tp_finalize resurrected it! Make it look like the original Py_DECREF * never happened. */ refcnt = self->ob_refcnt; _Py_NewReference(self); self->ob_refcnt = refcnt; However, PyObject_CallFinalizer() also does _PyGC_SET_FINALIZED(self, 1) and that is why __del__ is not called again after "X = 0": /* tp_finalize should only be called once. */ if (PyType_IS_GC(tp) && _PyGC_FINALIZED(self)) return; The comment and the code are very explicit, so this does nt look like a bug in cpython. Probably the docs should be fixed? Or this code is actually wrong? The test-case works as documented if I remove _PyGC_SET_FINALIZED() in PyObject_CallFinalizer() or add another _PyGC_SET_FINALIZED(self, 0) into PyObject_CallFinalizerFromDealloc() after _Py_NewReference(self), but yes, yes, I understand that this is not correct and won't really help. Oleg.
Hello Oleg, On Mon, 20 Mar 2017 18:28:29 +0100 Oleg Nesterov <oleg@redhat.com> wrote:
I started to learn python a few days ago and I am trying to understand what __del__() actually does. https://docs.python.org/3/reference/datamodel.html says:
object.__del__(self) ... Note that it is possible (though not recommended!) for the __del__() method to postpone destruction of the instance by creating a new reference to it. It may then be called at a later time when this new reference is deleted.
This sentence is not technically wrong, but it can easily be misleading. It says "it *may* then be called at a later time" and probably it should say "it may or may not be called at a later time, depending on the Python implementation you are using". Indeed CPython, the reference implementation, only calls __del__ once and doesn't call it again on resurrected objects. It is an implementation detail, though, and other implementations are free to behave otherwise, as garbage collectors are delicate beasts, difficult to tame. Regards Antoine.
On 2017-03-20 20:23, Antoine Pitrou wrote:
Hello Oleg,
On Mon, 20 Mar 2017 18:28:29 +0100 Oleg Nesterov <oleg@redhat.com> wrote:
I started to learn python a few days ago and I am trying to understand what __del__() actually does. https://docs.python.org/3/reference/datamodel.html says:
object.__del__(self) ... Note that it is possible (though not recommended!) for the __del__() method to postpone destruction of the instance by creating a new reference to it. It may then be called at a later time when this new reference is deleted.
This sentence is not technically wrong, but it can easily be misleading. It says "it *may* then be called at a later time" and probably it should say "it may or may not be called at a later time, depending on the Python implementation you are using".
[snip] I don't think I'd say it's misleading, but only that it might be misunderstood.
On Mar 20, 2017 1:26 PM, "Antoine Pitrou" <solipsis@pitrou.net> wrote: Hello Oleg, On Mon, 20 Mar 2017 18:28:29 +0100 Oleg Nesterov <oleg@redhat.com> wrote:
I started to learn python a few days ago and I am trying to understand what __del__() actually does. https://docs.python.org/3/ reference/datamodel.html says:
object.__del__(self) ... Note that it is possible (though not recommended!) for the __del__() method to postpone destruction of the instance by creating a new reference to it. It may then be called at a later time when this new reference is deleted.
This sentence is not technically wrong, but it can easily be misleading. It says "it *may* then be called at a later time" and probably it should say "it may or may not be called at a later time, depending on the Python implementation you are using". Modern CPython, and all extant versions of PyPy and Jython, guarantee that __del__ is called at most once. MicroPython doesn't support user-defined __del__ methods. It's fine if the text wants to leave that open, but the current phrasing is pretty misleading IMO. I also read it as saying that __del__ would be called again if the object is collected again (which may or may not happen). But AFAICT there are actually zero implementations where this is true. Probably worth a small edit :-) -n
On 03/20, Nathaniel Smith wrote:
Modern CPython, and all extant versions of PyPy and Jython, guarantee that __del__ is called at most once. MicroPython doesn't support user-defined __del__ methods.
It's fine if the text wants to leave that open, but the current phrasing is pretty misleading IMO. I also read it as saying that __del__ would be called again if the object is collected again (which may or may not happen).
Yes, that is why I was confused. Just I could not believe nobody else noticed this "bug" so I decided to check the sources and yes, the code looks very clear.
But AFAICT there are actually zero implementations where this is true.
Probably this was mostly true until the commit 796564c2 ("Issue #18112: PEP 442 implementation (safe object finalization)."), python2 calls __del__ again.
Probably worth a small edit :-)
Agreed. And it seems that not only me was confused, http://doc.pypy.org/en/latest/cpython_differences.html says: There are a few extra implications from the difference in the GC. Most notably, if an object has a __del__, the __del__ is never called more than once in PyPy; but CPython will call the same __del__ several times if the object is resurrected and dies again. Thanks to all! Oleg.
Hi all, On 20 March 2017 at 22:28, Nathaniel Smith <njs@pobox.com> wrote:
Modern CPython, and all extant versions of PyPy and Jython, guarantee that __del__ is called at most once.
Just a note, if someone actually depends on this: it is not true in all cases. For example, in CPython 3.5.3:
class X: ... __slots__=() # <= note this! ... def __del__(self): ... print("DEL") ... global resurrect ... resurrect = self ... print(X()) <__main__.X object at 0x7f5d1ad600d0> DEL resurrect=None DEL resurrect=None DEL resurrect=None DEL
A bientôt, Armin.
On 04/02, Armin Rigo wrote:
Hi all,
On 20 March 2017 at 22:28, Nathaniel Smith <njs@pobox.com> wrote:
Modern CPython, and all extant versions of PyPy and Jython, guarantee that __del__ is called at most once.
Just a note, if someone actually depends on this: it is not true in all cases. For example, in CPython 3.5.3:
class X: ... __slots__=() # <= note this! ... def __del__(self): ... print("DEL") ... global resurrect ... resurrect = self ... print(X()) <__main__.X object at 0x7f5d1ad600d0> DEL resurrect=None DEL
Objects/typeobject.c:type_new() /* Enable GC unless this class is not adding new instance variables and the base class did not use GC. */ if ((base->tp_flags & Py_TPFLAGS_HAVE_GC) || type->tp_basicsize > base->tp_basicsize) type->tp_flags |= Py_TPFLAGS_HAVE_GC; That is. "type->tp_basicsize > base->tp_basicsize" is false if empty __slots__, so we do not set HAVE_GC. And PyObject_CallFinalizer() doesn't set FINALIZED if PyType_IS_GC(tp) == F. Oleg.
participants (5)
-
Antoine Pitrou -
Armin Rigo -
MRAB -
Nathaniel Smith -
Oleg Nesterov