[Jeremy Hylton]
I was wondering today how I could convince myself that a sequence of Py_CLEAR() calls in a tp_clear function was safe. Take for example a really trivial sequence of code on frame_clear():
Py_CLEAR(f->f_exc_type); Py_CLEAR(f->f_exc_value); Py_CLEAR(f->f_exc_traceback); Py_CLEAR(f->f_trace);
We use Py_CLEAR() so that multiple calls to frame_clear() are safe. The first time this runs it sets exc_type to NULL before calling DECREF. This guarantees that a subsequent frame_clear() or frame_dealloc() won't attempt to DECREF it a second time. I think I understand why that's desireable and why it works. The primary risk is that via DECREF we may execute an arbitrary amount of Python code via weakref callbacks, finalizers, and code in other threads that gets resumed while the callbacks and finalizers are running.
My question, specifically, then: Why it is safe to assume that f doesn't point to trash after a particular call to Py_CLEAR()? Any particular call to Py_CLEAR() could break the cycle that the object is involved in an lead to a call to frame_dealloc(). The frame could get returned to an obmalloc pool, returned to the OS, or just re-used by another object before we get back to Py_CLEAR(). It seems like such behavior would be exceedingly unlikely, but I can't convince myself that it is impossible. Which is it: improbable or impossible? If it is only improbable, should we figure out how to write code that is safe against such an improbable event?
As Guido pointed out, tp_clear is called by gc from only one place, which sticks an incref/decref pair around the call so that the refcount on `f` can't fall to 0 while frame_clear() is active. That doesn't mean frame_clear is always safe to call, it only means that gc's use of the tp_clear slot is safe. Nobody else "should be" calling frame_clear (and it so happens nothing else in the core does), but it's something to be dimly aware of. For example, IIRC, some of ZODB's C code internally invokes its XXX_clear() functions directly, as part of removing persistent object state (unrelated to object deallocation). Determining whether those kinds of uses are safe requires case-by-case analysis.