[Python-checkins] python/dist/src/Modules gc_weakref.txt, NONE,
2.1.2.1 gcmodule.c, 2.71, 2.71.10.1
tim_one at users.sourceforge.net
tim_one at users.sourceforge.net
Thu Nov 20 17:13:53 EST 2003
Update of /cvsroot/python/python/dist/src/Modules
In directory sc8-pr-cvs1:/tmp/cvs-serv10811/Modules
Modified Files:
Tag: release23-maint
gcmodule.c
Added Files:
Tag: release23-maint
gc_weakref.txt
Log Message:
SF bug 839548: Bug in type's GC handling causes segfaults.
Also SF patch 843455.
This is a critical bugfix, backported from 2.4 development.
I don't intend to backport beyond 2.3 maint. The bugs this fixes
have been there since weakrefs were introduced.
--- NEW FILE: gc_weakref.txt ---
Before 2.3.3, Python's cyclic gc didn't pay any attention to weakrefs.
Segfaults in Zope3 resulted.
weakrefs in Python are designed to, at worst, let *other* objects learn
that a given object has died, via a callback function. The weakly
referenced object itself is not passed to the callback, and the presumption
is that the weakly referenced object is unreachable trash at the time the
callback is invoked.
That's usually true, but not always. Suppose a weakly referenced object
becomes part of a clump of cyclic trash. When enough cycles are broken by
cyclic gc that the object is reclaimed, the callback is invoked. If it's
possible for the callback to get at objects in the cycle(s), then it may be
possible for those objects to access (via strong references in the cycle)
the weakly referenced object being torn down, or other objects in the cycle
that have already suffered a tp_clear() call. There's no guarantee that an
object is in a sane state after tp_clear(). Bad things (including
segfaults) can happen right then, during the callback's execution, or can
happen at any later time if the callback manages to resurrect an insane
object.
Note that if it's possible for the callback to get at objects in the trash
cycles, it must also be the case that the callback itself is part of the
trash cycles. Else the callback would have acted as an external root to
the current collection, and nothing reachable from it would be in cyclic
trash either.
More, if the callback itself is in cyclic trash, then the weakref to which
the callback is attached must also be trash, and for the same kind of
reason: if the weakref acted as an external root, then the callback could
not have been cyclic trash.
So a problem here requires that a weakref, that weakref's callback, and the
weakly referenced object, all be in cyclic trash at the same time. This
isn't easy to stumble into by accident while Python is running, and, indeed,
it took quite a while to dream up failing test cases. Zope3 saw segfaults
during shutdown, during the second call of gc in Py_Finalize, after most
modules had been torn down. That creates many trash cycles (esp. those
involving new-style classes), making the problem much more likely. Once you
know what's required to provoke the problem, though, it's easy to create
tests that segfault before shutdown.
In 2.3.3, before breaking cycles, we first clear all the weakrefs with
callbacks in cyclic trash. Since the weakrefs *are* trash, and there's no
defined-- or even predictable --order in which tp_clear() gets called on
cyclic trash, it's defensible to first clear weakrefs with callbacks. It's
a feature of Python's weakrefs too that when a weakref goes away, the
callback (if any) associated with it is thrown away too, unexecuted.
Just that much is almost enough to prevent problems, by throwing away
*almost* all the weakref callbacks that could get triggered by gc. The
problem remaining is that clearing a weakref with a callback decrefs the
callback object, and the callback object may *itself* be weakly referenced,
via another weakref with another callback. So the process of clearing
weakrefs can trigger callbacks attached to other weakrefs, and those
latter weakrefs may or may not be part of cyclic trash.
So, to prevent any Python code from running while gc is invoking tp_clear()
on all the objects in cyclic trash, it's not quite enough just to invoke
tp_clear() on weakrefs with callbacks first. Instead the weakref module
grew a new private function (_PyWeakref_ClearRef) that does only part of
tp_clear(): it removes the weakref from the weakly-referenced object's list
of weakrefs, but does not decref the callback object. So calling
_PyWeakref_ClearRef(wr) ensures that wr's callback object will never
trigger, and (unlike weakref's tp_clear()) also prevents any callback
associated *with* wr's callback object from triggering.
Then we can call tp_clear on all the cyclic objects and never trigger
Python code.
After we do that, the callback objects still need to be decref'ed. Callbacks
(if any) *on* the callback objects that were also part of cyclic trash won't
get invoked, because we cleared all trash weakrefs with callbacks at the
start. Callbacks on the callback objects that were not part of cyclic trash
acted as external roots to everything reachable from them, so nothing
reachable from them was part of cyclic trash, so gc didn't do any damage to
objects reachable from them, and it's safe to call them at the end of gc.
An alternative would have been to treat objects with callbacks like objects
with __del__ methods, refusing to collect them, appending them to gc.garbage
instead. That would have been much easier. Jim Fulton gave a strong
argument against that (on Python-Dev):
There's a big difference between __del__ and weakref callbacks.
The __del__ method is "internal" to a design. When you design a
class with a del method, you know you have to avoid including the
class in cycles.
Now, suppose you have a design that makes has no __del__ methods but
that does use cyclic data structures. You reason about the design,
run tests, and convince yourself you don't have a leak.
Now, suppose some external code creates a weakref to one of your
objects. All of a sudden, you start leaking. You can look at your
code all you want and you won't find a reason for the leak.
IOW, a class designer can out-think __del__ problems, but has no control
over who creates weakrefs to his classes or class instances. The class
user has little chance either of predicting when the weakrefs he creates
may end up in cycles.
Callbacks on weakref callbacks are executed in an arbitrary order, and
that's not good (a primary reason not to collect cycles with objects with
__del__ methods is to avoid running finalizers in an arbitrary order).
However, a weakref callback on a weakref callback has got to be rare.
It's possible to do such a thing, so gc has to be robust against it, but
I doubt anyone has done it outside the test case I wrote for it.
Index: gcmodule.c
===================================================================
RCS file: /cvsroot/python/python/dist/src/Modules/gcmodule.c,v
retrieving revision 2.71
retrieving revision 2.71.10.1
diff -C2 -d -r2.71 -r2.71.10.1
*** gcmodule.c 17 Apr 2003 17:29:21 -0000 2.71
--- gcmodule.c 20 Nov 2003 22:13:51 -0000 2.71.10.1
***************
*** 378,388 ****
}
! /* Move the objects in unreachable with __del__ methods into finalizers.
! * The objects remaining in unreachable do not have __del__ methods, and
! * gc_refs remains GC_TENTATIVELY_UNREACHABLE for them. The objects
! * moved into finalizers have gc_refs changed to GC_REACHABLE.
*/
static void
! move_finalizers(PyGC_Head *unreachable, PyGC_Head *finalizers)
{
PyGC_Head *gc = unreachable->gc.gc_next;
--- 378,392 ----
}
! /* Move the objects in unreachable with __del__ methods into finalizers,
! * and weakrefs with callbacks into wr_callbacks.
! * The objects remaining in unreachable do not have __del__ methods, and are
! * not weakrefs with callbacks.
! * The objects moved have gc_refs changed to GC_REACHABLE; the objects
! * remaining in unreachable are left at GC_TENTATIVELY_UNREACHABLE.
*/
static void
! move_troublemakers(PyGC_Head *unreachable,
! PyGC_Head *finalizers,
! PyGC_Head *wr_callbacks)
{
PyGC_Head *gc = unreachable->gc.gc_next;
***************
*** 399,402 ****
--- 403,412 ----
gc->gc.gc_refs = GC_REACHABLE;
}
+ else if (PyWeakref_Check(op) &&
+ ((PyWeakReference *)op)->wr_callback) {
+ gc_list_remove(gc);
+ gc_list_append(gc, wr_callbacks);
+ gc->gc.gc_refs = GC_REACHABLE;
+ }
gc = next;
}
***************
*** 435,438 ****
--- 445,535 ----
}
+ /* Clear all trash weakrefs with callbacks. This clears weakrefs first,
+ * which has the happy result of disabling the callbacks without executing
+ * them. A nasty technical complication: a weakref callback can itself be
+ * the target of a weakref, in which case decrefing the callback can cause
+ * another callback to trigger. But we can't allow arbitrary Python code to
+ * get executed at this point (the callback on the callback may try to muck
+ * with other cyclic trash we're trying to collect, even resurrecting it
+ * while we're in the middle of doing tp_clear() on the trash).
+ *
+ * The private _PyWeakref_ClearRef() function exists so that we can clear
+ * the reference in a weakref without triggering a callback on the callback.
+ *
+ * We have to save the callback objects and decref them later. But we can't
+ * allocate new memory to save them (if we can't get new memory, we're dead).
+ * So we grab a new reference on the clear'ed weakref, which prevents the
+ * rest of gc from reclaiming it. _PyWeakref_ClearRef() leaves the
+ * weakref's wr_callback member intact.
+ *
+ * In the end, then, wr_callbacks consists of cleared weakrefs that are
+ * immune from collection. Near the end of gc, after collecting all the
+ * cyclic trash, we call release_weakrefs(). That releases our references
+ * to the cleared weakrefs, which in turn may trigger callbacks on their
+ * callbacks.
+ */
+ static void
+ clear_weakrefs(PyGC_Head *wr_callbacks)
+ {
+ PyGC_Head *gc = wr_callbacks->gc.gc_next;
+
+ for (; gc != wr_callbacks; gc = gc->gc.gc_next) {
+ PyObject *op = FROM_GC(gc);
+ PyWeakReference *wr;
+
+ assert(IS_REACHABLE(op));
+ assert(PyWeakref_Check(op));
+ wr = (PyWeakReference *)op;
+ assert(wr->wr_callback != NULL);
+ Py_INCREF(op);
+ _PyWeakref_ClearRef(wr);
+ }
+ }
+
+ /* Called near the end of gc. This gives up the references we own to
+ * cleared weakrefs, allowing them to get collected, and in turn decref'ing
+ * their callbacks.
+ *
+ * If a callback object is itself the target of a weakref callback,
+ * decref'ing the callback object may trigger that other callback. If
+ * that other callback was part of the cyclic trash in this generation,
+ * that won't happen, since we cleared *all* trash-weakref callbacks near
+ * the start of gc. If that other callback was not part of the cyclic trash
+ * in this generation, then it acted like an external root to this round
+ * of gc, so all the objects reachable from that callback are still alive.
+ *
+ * Giving up the references to the weakref objects will probably make
+ * them go away too. However, if a weakref is reachable from finalizers,
+ * it won't go away. We move it to the old generation then. Since a
+ * weakref object doesn't have a finalizer, that's the right thing to do (it
+ * doesn't belong in gc.garbage).
+ *
+ * We return the number of weakref objects freed (those not appended to old).
+ */
+ static int
+ release_weakrefs(PyGC_Head *wr_callbacks, PyGC_Head *old)
+ {
+ int num_freed = 0;
+
+ while (! gc_list_is_empty(wr_callbacks)) {
+ PyGC_Head *gc = wr_callbacks->gc.gc_next;
+ PyObject *op = FROM_GC(gc);
+ PyWeakReference *wr = (PyWeakReference *)op;
+
+ assert(IS_REACHABLE(op));
+ assert(PyWeakref_Check(op));
+ assert(wr->wr_callback != NULL);
+ Py_DECREF(op);
+ if (wr_callbacks->gc.gc_next == gc) {
+ /* object is still alive -- move it */
+ gc_list_remove(gc);
+ gc_list_append(gc, old);
+ }
+ else
+ ++num_freed;
+ }
+ return num_freed;
+ }
+
static void
debug_instance(char *msg, PyInstanceObject *inst)
***************
*** 536,541 ****
PyGC_Head *young; /* the generation we are examining */
PyGC_Head *old; /* next older generation */
! PyGC_Head unreachable;
! PyGC_Head finalizers;
PyGC_Head *gc;
--- 633,639 ----
PyGC_Head *young; /* the generation we are examining */
PyGC_Head *old; /* next older generation */
! PyGC_Head unreachable; /* non-problematic unreachable trash */
! PyGC_Head finalizers; /* objects with, & reachable from, __del__ */
! PyGC_Head wr_callbacks; /* weakrefs with callbacks */
PyGC_Head *gc;
***************
*** 598,615 ****
* finalizers can't safely be deleted. Python programmers should take
* care not to create such things. For Python, finalizers means
! * instance objects with __del__ methods.
*
! * Move unreachable objects with finalizers into a different list.
*/
gc_list_init(&finalizers);
! move_finalizers(&unreachable, &finalizers);
/* finalizers contains the unreachable objects with a finalizer;
! * unreachable objects reachable only *from* those are also
! * uncollectable, and we move those into the finalizers list too.
*/
move_finalizer_reachable(&finalizers);
/* Collect statistics on collectable objects found and print
! * debugging information. */
for (gc = unreachable.gc.gc_next; gc != &unreachable;
gc = gc->gc.gc_next) {
--- 696,726 ----
* finalizers can't safely be deleted. Python programmers should take
* care not to create such things. For Python, finalizers means
! * instance objects with __del__ methods. Weakrefs with callbacks
! * can call arbitrary Python code, so those are special-cased too.
*
! * Move unreachable objects with finalizers, and weakrefs with
! * callbacks, into different lists.
*/
gc_list_init(&finalizers);
! gc_list_init(&wr_callbacks);
! move_troublemakers(&unreachable, &finalizers, &wr_callbacks);
! /* Clear the trash weakrefs with callbacks. This prevents their
! * callbacks from getting invoked (when a weakref goes away, so does
! * its callback).
! * We do this even if the weakrefs are reachable from finalizers.
! * If we didn't, breaking cycles in unreachable later could trigger
! * deallocation of objects in finalizers, which could in turn
! * cause callbacks to trigger. This may not be ideal behavior.
! */
! clear_weakrefs(&wr_callbacks);
/* finalizers contains the unreachable objects with a finalizer;
! * unreachable objects reachable *from* those are also uncollectable,
! * and we move those into the finalizers list too.
*/
move_finalizer_reachable(&finalizers);
/* Collect statistics on collectable objects found and print
! * debugging information.
! */
for (gc = unreachable.gc.gc_next; gc != &unreachable;
gc = gc->gc.gc_next) {
***************
*** 624,627 ****
--- 735,743 ----
*/
delete_garbage(&unreachable, old);
+
+ /* Now that we're done analyzing stuff and breaking cycles, let
+ * delayed weakref callbacks run.
+ */
+ m += release_weakrefs(&wr_callbacks, old);
/* Collect statistics on uncollectable objects found and print
More information about the Python-checkins
mailing list