results of id() and weakref.getweakrefs() sometimes break on object resurrection
Hello developers,
I observed strange behaviour in CPython (tested in 2.7.5 and 3.3.3) regarding object resurrection. Yes, resurrection is evil, but it is a valid scenario. If an object is resurrected via its finalizer __del__, sometimes its unique id value as returned from id() changes. Additionally the list of weak references pointing to it as returned by weakref.getweakrefs(...) breaks (i.e. is suddenly empty, assuming it wasn't before). These issues only arise if the resurrection occurs during cyclic garbage collection. If the object is finalized because its refcount drops to zero, everything is fine. See the attached test-file.
Is this behaviour intended or is it a bug? If so, which variant is the bug and which is right? I can hardly believe that whether id() is preserved should depend on whether the garbage was cyclic or not.
This might appear of low relevance to you, since no sane program intentionally performs resurrection. However I originally became aware of the issue in Jython (where it not only occurs for cyclic garbage but in every resurrection-case), c.f. http://bugs.jython.org/issue2224. I am interested in this because I am implementing native gc support in JyNI and need to understand these details to do it right.
Thanks in advance!
Stefan
Hello Stefan,
On Sun, 26 Oct 2014 00:20:47 +0200 "Stefan Richthofer" Stefan.Richthofer@gmx.de wrote:
Hello developers,
I observed strange behaviour in CPython (tested in 2.7.5 and 3.3.3) regarding object resurrection.
Your runGC() function is buggy, it does not run the GC under CPython. Fix it and the first problem (with id()) disappears.
The second problem (with weakref) is different: weakrefs are cleared before __del__ is called, so resurrection doesn't affect the whole process. Add a callback to the weakref and you'll see it is getting called.
In other words, CPython behaves as expected. Your concern is appreciated, though.
Regards
Antoine.
Okay, sorry, I was thinking too Jython-like. I fixed runGC() just to see now that it does not even trigger resurrection, since under CPython there are no finalizers executed in ref cycles (i.e. I find my objects in gc.garbage). So I realize, my xy_cyclic tests are pointless anyway since in cyclic gc no resurrection can happen.
The second problem (with weakref) is different: weakrefs are cleared before __del__ is called, so resurrection doesn't affect the whole process.
It appears weakrefs are only cleared if this is done by gc (where no resurrection can happen anyway). If a resurrection-performing-__del__ is just called by ref-count-drop-to-0, weakrefs persist - a behavior that is very difficult and inefficient to emulate in Jython, but I'll give it some more thoughts...
However thanks for the help!
-Stefan
Gesendet: Sonntag, 26. Oktober 2014 um 01:22 Uhr Von: "Antoine Pitrou" solipsis@pitrou.net An: python-dev@python.org Betreff: Re: [Python-Dev] results of id() and weakref.getweakrefs() sometimes break on object resurrection
Hello Stefan,
On Sun, 26 Oct 2014 00:20:47 +0200 "Stefan Richthofer" Stefan.Richthofer@gmx.de wrote:
Hello developers,
I observed strange behaviour in CPython (tested in 2.7.5 and 3.3.3) regarding object resurrection.
Your runGC() function is buggy, it does not run the GC under CPython. Fix it and the first problem (with id()) disappears.
The second problem (with weakref) is different: weakrefs are cleared before __del__ is called, so resurrection doesn't affect the whole process. Add a callback to the weakref and you'll see it is getting called.
In other words, CPython behaves as expected. Your concern is appreciated, though.
Regards
Antoine.
Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/stefan.richthofer%40gmx.d...
On Sun, 26 Oct 2014 02:50:39 +0200 "Stefan Richthofer" Stefan.Richthofer@gmx.de wrote:
Okay, sorry, I was thinking too Jython-like. I fixed runGC() just to see now that it does not even trigger resurrection, since under CPython there are no finalizers executed in ref cycles (i.e. I find my objects in gc.garbage).
Try CPython 3.4 :-)
Regards
Antoine.
On Saturday, October 25, 2014, Stefan Richthofer Stefan.Richthofer@gmx.de wrote:
Okay, sorry, I was thinking too Jython-like. I fixed runGC() just to see now that it does not even trigger resurrection, since under CPython there are no finalizers executed in ref cycles (i.e. I find my objects in gc.garbage). So I realize, my xy_cyclic tests are pointless anyway since in cyclic gc no resurrection can happen.
The second problem (with weakref) is different: weakrefs are cleared before __del__ is called, so resurrection doesn't affect the whole process.
It appears weakrefs are only cleared if this is done by gc (where no resurrection can happen anyway). If a resurrection-performing-__del__ is just called by ref-count-drop-to-0, weakrefs persist - a behavior that is very difficult and inefficient to emulate in Jython, but I'll give it some more thoughts...
You shouldn't have to emulate that. The exact behavior of GC is allowed to
vary between systems.
However thanks for the help!
-Stefan
Gesendet: Sonntag, 26. Oktober 2014 um 01:22 Uhr Von: "Antoine Pitrou" <solipsis@pitrou.net javascript:;> An: python-dev@python.org javascript:; Betreff: Re: [Python-Dev] results of id() and weakref.getweakrefs()
sometimes break on object resurrection
Hello Stefan,
On Sun, 26 Oct 2014 00:20:47 +0200 "Stefan Richthofer" <Stefan.Richthofer@gmx.de javascript:;> wrote:
Hello developers,
I observed strange behaviour in CPython (tested in 2.7.5 and 3.3.3) regarding object resurrection.
Your runGC() function is buggy, it does not run the GC under CPython. Fix it and the first problem (with id()) disappears.
The second problem (with weakref) is different: weakrefs are cleared before __del__ is called, so resurrection doesn't affect the whole process. Add a callback to the weakref and you'll see it is getting called.
In other words, CPython behaves as expected. Your concern is appreciated, though.
Regards
Antoine.
Python-Dev mailing list Python-Dev@python.org javascript:; https://mail.python.org/mailman/listinfo/python-dev Unsubscribe:
https://mail.python.org/mailman/options/python-dev/stefan.richthofer%40gmx.d...
Python-Dev mailing list Python-Dev@python.org javascript:; https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org
Hi Stefan,
On 26 October 2014 02:50, Stefan Richthofer Stefan.Richthofer@gmx.de wrote:
It appears weakrefs are only cleared if this is done by gc (where no resurrection can happen anyway). If a resurrection-performing-__del__ is just called by ref-count-drop-to-0, weakrefs persist -
How do you reach this conclusion? The following test program seems to show the opposite, by printing None on Python 2.7.6:
import weakref class X(object): def __del__(self): print ref() x = X() ref = weakref.ref(x) del x
A bientôt,
Armin.
Your test program performs no resurrection of x.
Interestingly, it does not change behavior if you write
class X(object): def __del__(self): X.x = self print ref()
(Thanks for making me aware of this! My test-case was already initially the more complex one given below)
But if the resurrection occurs indirectly, the weakref persists: (I refined it to old-style class, because Jython will support new-style class finalizers only from 2.7. beta 4 onwards, i.e. the test would be pointless with any current release)
import weakref, time, gc class ReferentDummy(): pass
class X(): def __del__(self): X.y = self.z print "__del__: "+str(ref())
x = X() x2 = ReferentDummy() ref = weakref.ref(x2) x.z = x2 del x2 del x #Everything is now deleted, isn't it? gc.collect() #needed in Jython-case time.sleep(0.2) #wait for Java's async gc to finnish print ref() print weakref.getweakrefs(X.y)
---------------CPython output: __del__: <__main__.ReferentDummy instance at 0x7fd2603e1950> <__main__.ReferentDummy instance at 0x7fd2603e1950> [<weakref at 0x7fd2603d2c00; to 'instance' at 0x7fd2603e1950>]
---------------Jython 2.7 beta 3 output: __del__: None None []
One can surely argue x2 has never been dead, or see it as "it was killed along with x and then resurrected by x". Jython clearly takes the second point of view and also clears weakrefs to x.z, while CPython does not. Yes, these details probably hardly matter in practice (however could cause subtle bugs when porting complex stuff from CPython to Jython), but since I try to bridge it, I have to look into this more carefully.
Best,
Stefan
On 10/26/2014 06:44 PM, Armin Rigo wrote:
Hi Stefan,
On 26 October 2014 02:50, Stefan Richthofer Stefan.Richthofer@gmx.de wrote:
It appears weakrefs are only cleared if this is done by gc (where no resurrection can happen anyway). If a resurrection-performing-__del__ is just called by ref-count-drop-to-0, weakrefs persist -
How do you reach this conclusion? The following test program seems to show the opposite, by printing None on Python 2.7.6:
import weakref class X(object): def __del__(self): print ref() x = X() ref = weakref.ref(x) del xA bientôt,
Armin.
On Mon, 27 Oct 2014 14:36:31 +0100 Stefan Richthofer stefan.richthofer@gmx.de wrote:
Your test program performs no resurrection of x.
Interestingly, it does not change behavior if you write
class X(object): def __del__(self): X.x = self print ref()
(Thanks for making me aware of this! My test-case was already initially the more complex one given below)
But if the resurrection occurs indirectly, the weakref persists:
It's not that resurrection occurs indirectly, it's that the object pointed to by "x2" always remains alive (first as an instance attribute of x, second as a class attribute of X *before x is deleted*).
Regards
Antoine.
It's not that resurrection occurs indirectly, it's that the object pointed to by "x2" always remains alive
Yes, this is right for CPython, more precisely this is about the definition of the word "resurrection" (in language-independent sense), which seems not to be unique.
I already pointed out "One can surely argue x2 has never been dead, or see it as "it was killed along with x and then resurrected by x"."
In Java and thus in Jython, it is treated as the second one. An equal program written in Java or Jython would even call the finalizer of x2 (if it had one) and clear weakrefs before it is available "again" as a class attribute of X. So there actually *is* a notion to refer to this scenario as resurrection. I admit it is arguable and maybe misleading in CPython case and I was not aware of the whole behavior when I called the topic "resurrection".
What would still be interesting (at least when Jython 3 is born), is which of the mentioned behaviors occurs if it is performed by CPython's cyclic gc (consistently the first one I would guess). As you pointed out, this is only relevant from 3.4 on since in 2.x etc it does not call finalizers in cycles. (Since I mainly work on Jython or Python 2.7 I currently have no 3.4 installed to test this instantaneously. I will test it someday...)
Best,
Stefan
On 10/27/2014 03:14 PM, Antoine Pitrou wrote:
On Mon, 27 Oct 2014 14:36:31 +0100 Stefan Richthofer stefan.richthofer@gmx.de wrote:
Your test program performs no resurrection of x.
Interestingly, it does not change behavior if you write
class X(object): def __del__(self): X.x = self print ref()
(Thanks for making me aware of this! My test-case was already initially the more complex one given below)
But if the resurrection occurs indirectly, the weakref persists:
It's not that resurrection occurs indirectly, it's that the object pointed to by "x2" always remains alive (first as an instance attribute of x, second as a class attribute of X *before x is deleted*).
Regards
Antoine.
Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/stefan.richthofer%40gmx.d...
On Mon, 27 Oct 2014 16:20:06 +0100 Stefan Richthofer stefan.richthofer@gmx.de wrote:
I already pointed out "One can surely argue x2 has never been dead, or see it as "it was killed along with x and then resurrected by x"."
In Java and thus in Jython, it is treated as the second one.
You mean Jython deletes instance attributes before calling __del__ ? That would make most __del__ implementations quite useless... And actually your own example would fail with an AttributeError on "X.y = self.z".
What would still be interesting (at least when Jython 3 is born), is which of the mentioned behaviors occurs if it is performed by CPython's cyclic gc (consistently the first one I would guess).
In which use case exactly? :-) I've lost track a bit, since you've posted several examples...
Regards
Antoine.
participants (5)
-
Antoine Pitrou -
Armin Rigo -
Guido van Rossum -
Stefan Richthofer -
Stefan Richthofer