[Python-Dev] issue 9141, finalizers and gc module

[Kristján Valur Jónsson]

Hello there.
For those familiar with the intricacies of the gcmodule.c, I would like to draw your attention to http://bugs.python.org/issue9141.

I would like to consult with you to find out more about finalizers/gc in order to improve the in-file documentation.

Traditionally, it has not been possible to collect objects that have __del__ methods, or more generally, finalizers.  Instead, they and any objects that are reachable from them, are put in gc.garbage.

What I want to know is, why is this limitation in place?  Here are two possibilities:

1)      "The order of calling finalizers in a cycle is undefined so it is not a solvable problem".  But this would allow a single object, with only internal cycles to be collected.  Currently this is not the case.

2)      "During collection, the interpreter is in a fragile state (linked lists of gc objects with refcount bookkeeping in place) and no unknown code can be allowed to run".  This is the reason I personally think is the true reason.

The reason I'm asking is that python has traditionally tested for finalizers by checking the tp_del slot of the object's type.  This will be true if the object has a __del__ method.  Since generators were added, they use the tp_del slot for their own finalizers, but special code was put in place so that the generators could tell if the finalizer were "trivial" or not (trivial meaning "just doing Py_DECREF()).
This allowed generators to be coollected too, if they were in a common, trivial state, but otherwise, they had to be put in gc.garbage().

Yesterday, I stumbled upon the fact that tp_dealloc of iobase objects also calls an internal finalizer, one that isn't exposed in any tp_del slot:  It will invoke a PyObject_CallMethod(self, "close", "") on itself.  This will happen whenever iobase objects are part of a cycle that needs to be cleared.  This can cause arbitrary code to run.  There are even provisions made for the resurrection of the iobase objects based on the action of this close() call.

Clearly, this has the potential to be non-trivial, and therefore, again, I see this as an argument for my proposed patched in issue 9141.  But others have voiced worries that if we stop collecting iobase objects, that would be a regression.

So, I ask you:  What is allowed during tp_clear()?  Is this a hard rule?  What is the reason?

Kristján

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20120417/c2ee5fd8/attachment.html>