[Python-Dev] Evil reference cycles caused Exception.__traceback__

[Martin Panter]

On 18 September 2017 at 09:31, Victor Stinner <victor.stinner at gmail.com> wrote:
> Last years, I fixed many reference cycles in various parts of the
> Python 3 standard library. Sometimes, it takes years to become aware
> of the reference cycle and finally fix it.
>
> For example, recently, I worked on fixing all "dangling threads"
> leaked by tests of the Python test suite, and I found and fixed many
> reference cycles which probably existed since Python 3 was created
> (forked from Python 2)

My instinct is to suggest to make the “dangling threads” test tolerate
Thread objects that are no longer running. Last time I looked, it
tests a list of weak references to Thread objects. But maybe it should
check “Thread.is_alive” or use “threading.enumerate”.

> . . .
>
> Ideally, CPython 3.x should never create reference cycles. Removing
> Exception.__traceback__ is the obvious "fix" for the issue. But I
> expect that slowly, a lot of code started to rely on the attribute,
> maybe even for good reasons :-)
>
> A more practical solution would be to log a warning. Maybe the garbage
> collector can emit a warning if it detects an exception part of a
> reference cycle? Or maybe detect frames?

The “gc” module can do some of this already. In the past (when __del__
used to prevent garbage collection) I used DEBUG_SAVEALL to and
graphed the result to figure out where code was creating reference
cycles. Or maybe if you turned on DEBUG_COLLECTABLE and filtered the
objects printed out, or played with the new “gc.callbacks” API.

> If the GC cannot do it, maybe we might use a debug thread (enabled
> manually) which checks manually if an exception is part of a reference
> cycle using gc.get_objects(): check if an exception remains alive
> longer than X seconds? I had the same idea for asyncio, to detect
> reference cycles or if a task is never "awaited", but I never
> implemented the idea.