[issue32377] Difference in ressurrection behavior with __del__ in py2 vs. py3
New submission from Eric Cousineau <eric.cousineau@tri.global>: Due to how `PyObject_CallFinalizer` is written in python3, `__del__` will only *ever* be called once. In my use case, I am experimenting with a feature in `pybind11` to prevent slicing with Python class instances that inherit from pybind11-C++ base classes, which involves detecting when an instance loses all reference in Python (`Py_REFCNT(...) == 0`) but still has reference in C++ (`shared_ptr::count() > 0`), and reviving the Python portion when this situation happens. In python2, I could do this without a hitch, as a resurrected object could have its `__del__` method called multiple times (through `tp_dealloc` I believe?). But in python3, the object is marked with `_PyGC_SET_FINALIZED(...)`, thus preventing `__del__` from being called again. It'd be nice to either (a) somehow allow `__del__` to be called naturally without too much fuss or, at the least, (b) have this reflected in the documentation: https://docs.python.org/3/reference/datamodel.html#object.__del__ See attached `revive_test`. Example execution: ``` $ python2 ./revive_test.py Revive Destroy [ Done ] $ python3 ./revive_test.py Revive [ Done ] ``` ---------- assignee: docs@python components: Documentation files: revive_test.py messages: 308660 nosy: Eric Cousineau, docs@python priority: normal severity: normal status: open title: Difference in ressurrection behavior with __del__ in py2 vs. py3 type: behavior versions: Python 3.4, Python 3.5, Python 3.6 Added file: https://bugs.python.org/file47339/revive_test.py _______________________________________ Python tracker <report@bugs.python.org> <https://bugs.python.org/issue32377> _______________________________________
Antoine Pitrou <pitrou@free.fr> added the comment: Thanks for the report. Apparently I forgot to update that piece of documentation when PEP 442 was implemented. ---------- nosy: +pitrou stage: -> needs patch versions: +Python 3.7 -Python 3.4, Python 3.5 _______________________________________ Python tracker <report@bugs.python.org> <https://bugs.python.org/issue32377> _______________________________________
Eric Cousineau <eric.cousineau@tri.global> added the comment: You're welcome, and thank you for the prompt response! I will say that it feels a tad odd to only have `tp_finalize` be called once for the entire lifetime of the object, while still having the option of it being resurrected. Is there any way to somehow "un-mark" the object to enable this workflow that I would like to have? My current hack is to call `_PyGC_SET_FINALIZED(self, 0)` - may I ask if there is a simpler way to do this? ---------- _______________________________________ Python tracker <report@bugs.python.org> <https://bugs.python.org/issue32377> _______________________________________
Antoine Pitrou <pitrou@free.fr> added the comment: Le 19/12/2017 à 17:46, Eric Cousineau a écrit :
My current hack is to call `_PyGC_SET_FINALIZED(self, 0)` - may I ask if there is a simpler way to do this?
Well... perhaps you could create another PyObject (it's just a wrapper, right?) since the old one doesn't have any outside references to it remaining. Note that calling __del__ only once is also how PyPy works: http://doc.pypy.org/en/latest/cpython_differences.html#differences-related-t... If there is some demand we could expose a higher-level spelling of `_PyGC_SET_FINALIZED(self, 0)`. ---------- _______________________________________ Python tracker <report@bugs.python.org> <https://bugs.python.org/issue32377> _______________________________________
Change by Antoine Pitrou <pitrou@free.fr>: ---------- keywords: +patch pull_requests: +4822 stage: needs patch -> patch review _______________________________________ Python tracker <report@bugs.python.org> <https://bugs.python.org/issue32377> _______________________________________
Antoine Pitrou <pitrou@free.fr> added the comment: I've proposed a documentation improvement in https://github.com/python/cpython/pull/4927 . Please chime in if you see have issues with it. ---------- _______________________________________ Python tracker <report@bugs.python.org> <https://bugs.python.org/issue32377> _______________________________________
Antoine Pitrou <pitrou@free.fr> added the comment: New changeset 4b965930e8625f77cb0e821daf5cc40e85b45f84 by Antoine Pitrou in branch 'master': bpo-32377: improve __del__ docs and fix mention about resurrection (#4927) https://github.com/python/cpython/commit/4b965930e8625f77cb0e821daf5cc40e85b... ---------- _______________________________________ Python tracker <report@bugs.python.org> <https://bugs.python.org/issue32377> _______________________________________
Change by Roundup Robot <devnull@psf.upfronthosting.co.za>: ---------- pull_requests: +4823 _______________________________________ Python tracker <report@bugs.python.org> <https://bugs.python.org/issue32377> _______________________________________
Antoine Pitrou <pitrou@free.fr> added the comment: New changeset dc5770b161a5e28eeff73a406cd4eddb0676c5b5 by Antoine Pitrou (Miss Islington (bot)) in branch '3.6': bpo-32377: improve __del__ docs and fix mention about resurrection (GH-4927) (#4929) https://github.com/python/cpython/commit/dc5770b161a5e28eeff73a406cd4eddb067... ---------- _______________________________________ Python tracker <report@bugs.python.org> <https://bugs.python.org/issue32377> _______________________________________
Change by Antoine Pitrou <pitrou@free.fr>: ---------- resolution: -> fixed stage: patch review -> resolved status: open -> closed _______________________________________ Python tracker <report@bugs.python.org> <https://bugs.python.org/issue32377> _______________________________________
Eric Cousineau <eric.cousineau@tri.global> added the comment: Super late response, but for this part:
Well... perhaps you could create another PyObject (it's just a wrapper, right?) since the old one doesn't have any outside references to it remaining.
In certain cases, yes, that would be the case. I do have unittests that (hackily) check `id(o)` before and after resurrection by using a `weakref`, so I could relax that contract. However, in other cases, the "wrapper" object in Python is actually a C++ base class that has been derived from in Python. If ownership of the C++ instance (which should more-or-less own the Python portion) goes solely to C++, then the Python portion could be garbage collected, and you get a sort-of slicing effect: https://github.com/pybind/pybind11/issues/1145 Granted, within this web of issues, one alternative is to always extend the lifetime of the Python object by the C++ portion. However, that creates a (opaque?) reference cycle between C++ and CPython, so I am a bit afraid of what that would do. I would prefer to make memory management conservative if possible. Or rather: It's something cool to try out, but I have not yet had time to dig in :( ---------- _______________________________________ Python tracker <report@bugs.python.org> <https://bugs.python.org/issue32377> _______________________________________
Eric Cousineau <eric.cousineau@tri.global> added the comment: See also bpo-40240: "Expose public spelling of _PyGC_FINALIZED and _PyGC_SET_FINALIZED?" ---------- _______________________________________ Python tracker <report@bugs.python.org> <https://bugs.python.org/issue32377> _______________________________________
participants (3)
-
Antoine Pitrou -
Eric Cousineau
-
Roundup Robot