[Python-Dev] Reference cycles in Exception.__traceback__

[Victor Stinner]

2014-03-07 6:25 GMT+01:00 Nick Coghlan <ncoghlan at gmail.com>:
>> Uh, really? If you want to suppress all reference cycles, you *have* to
>> remove __traceback__.
>>
>> The problem is to make computation of the traceback summary lightweight
>> enough that it doesn't degrade performance in the common case where you
>> don't have to print the traceback later.
>
> The proposed summary extraction only keeps the exception type and its str
> output, not the exception itself (as you don't need that to create the
> formatted traceback).

My patch keeps the exception object, but it breaks links to the other
exceptions (__cause__ and __context__) and to the traceback
(__traceback__):
https://bitbucket.org/haypo/misc/src/tip/python/suppress_locals.py

Then I saw that http://bugs.python.org/issue17911 already has a patch
for the traceback module, nice! And it's very close to what I wrote:
http://bugs.python.org/review/17911/#ps8639

This patch stores the exception as string
(traceback._format_value(exc)). I prefer to avoid any useless
formatting, since the formatted exception is only needed in rare
cases. Just drop the "exception summary/view" is the most common case.

If you want to full original exception object, don't use the
summary/view but handle the exception in the except block.


I realized that "memory leaks" (reference cycles) is a common issue
with traceback objects (in Python 2) and exception objects (in Python
3):

Extracting tracebacks does too much work [open]
http://bugs.python.org/issue17911

Local variables not freed when Exception raises in function called
from cycle [wont fix]
http://bugs.python.org/issue5641

asyncio.Future.set_exception() creates a reference cycle [invalid]
http://bugs.python.org/issue20032

Do we need to call gc.collect() occasionally through the event loop?
http://code.google.com/p/tulip/issues/detail?id=42

Traceback objects not properly garbage-collected [invalid]
http://bugs.python.org/issue226254

Reference cycle in _TracebackLogger and bug in _TracebackLogger.__del__()
http://code.google.com/p/tulip/issues/detail?id=155

Twisted fake Traceback object:
http://twistedmatrix.com/trac/browser/trunk/twisted/python/failure.py#L89

frame.f_locals keeps references to things for too long [open since 2009],
request from Twisted
http://bugs.python.org/issue6116
http://twistedmatrix.com/trac/ticket/3853

assertRaises as a context manager keeps tracebacks and frames alive [open]
http://bugs.python.org/issue9815

Expose called function on frame object
http://bugs.python.org/issue12857

tracebacks eat up memory by holding references to locals and globals
when they are not wanted [fixed by traceback.clear_frames()]
http://bugs.python.org/issue1565525

Add a frame method to clear expensive details [fixed by frame.clear()]
http://bugs.python.org/issue17934

Generator cleanup without tp_del [rejected]
http://bugs.python.org/issue17807

Generator memory leak [duplicate]
http://bugs.python.org/issue17468

asyncio: remove _TracebackLogger [fixed]
http://bugs.python.org/issue19967

sys.exc_info() should not be stored on a local variable [fixed]
https://code.djangoproject.com/ticket/10758

Capturing the Currently Raised Exception
http://docs.python.org/3/howto/pyporting.html#capturing-the-currently-raised-exception
    In Python 3, the traceback is attached to the exception instance through
    the __traceback__ attribute. If the instance is saved in a local variable
    that persists outside of the except block, the traceback will create a
    reference cycle with the current frame and its dictionary of local
    variables. This will delay reclaiming dead resources until the next cyclic
    garbage collection pass.

    In Python 2, this problem only occurs if you save the traceback itself
    (e.g.  the third element of the tuple returned by sys.exc_info()) in a
    variable.

    => http://hewgill.com/journal/entries/541-python-2-to-3-upgrade-and-exception-handling

[Python-Dev] new unbounded memory leak in exception handling?
https://mail.python.org/pipermail/python-dev/2009-November/094304.html

PEP 3134: Exception Chaining and Embedded Tracebacks [final]
http://legacy.python.org/dev/peps/pep-3134/

PEP 344: Exception Chaining and Embedded Tracebacks [superseded]
http://legacy.python.org/dev/peps/pep-0344/

Victor