[Python-Dev] safety of Py_CLEAR and self

[Tim Peters]

[Jeremy Hylton]
> I was wondering today how I could convince myself that a sequence of
> Py_CLEAR() calls in a tp_clear function was safe.  Take for example a
> really trivial sequence of code on frame_clear():
>
>         Py_CLEAR(f->f_exc_type);
>         Py_CLEAR(f->f_exc_value);
>         Py_CLEAR(f->f_exc_traceback);
>         Py_CLEAR(f->f_trace);
>
> We use Py_CLEAR() so that multiple calls to frame_clear() are safe.
> The first time this runs it sets exc_type to NULL before calling
> DECREF.  This guarantees that a subsequent frame_clear() or
> frame_dealloc() won't attempt to DECREF it a second time.  I think I
> understand why that's desireable and why it works.  The primary risk
> is that via DECREF we may execute an arbitrary amount of Python code
> via weakref callbacks, finalizers, and code in other threads that gets
> resumed while the callbacks and finalizers are running.
>
> My question, specifically, then: Why it is safe to assume that f
> doesn't point to trash after a particular call to Py_CLEAR()?  Any
> particular call to Py_CLEAR() could break the cycle that the object is
> involved in an lead to a call to frame_dealloc().  The frame could get
> returned to an obmalloc pool, returned to the OS, or just re-used by
> another object before we get back to Py_CLEAR().  It seems like such
> behavior would be exceedingly unlikely, but I can't convince myself
> that it is impossible.  Which is it: improbable or impossible?  If it
> is only improbable, should we figure out how to write code that is
> safe against such an improbable event?

As Guido pointed out, tp_clear is called by gc from only one place,
which sticks an incref/decref pair around the call so that the
refcount on `f` can't fall to 0 while frame_clear() is active.

That doesn't mean frame_clear is always safe to call, it only means
that gc's use of the tp_clear slot is safe.  Nobody else "should be"
calling frame_clear (and it so happens nothing else in the core does),
but it's something to be dimly aware of.  For example, IIRC, some of
ZODB's C code internally invokes its XXX_clear() functions directly,
as part of removing persistent object state (unrelated to object
deallocation).  Determining whether those kinds of uses are safe
requires case-by-case analysis.