[Python-ideas] feature to make traceback objects usable without references to frame locals and globals

[ghazel at gmail.com]

Well, I discovered this property of traceback objects when a
real-world server of mine began eating all the memory on the server.
To me, this is the most convincing reason to address the issue.

I'm not sure what sort of profiling you're looking for, but I have
since then produced a contrived example which demonstrates a serious
memory consumption difference with a very short traceback object
lifetime: http://codepad.org/F23cwezb

If you run the test with "s.e = sys.exc_info()" commented out, the
observed memory footprint of the process quickly approaches and sits
at 5,677,056 bytes. Totally reasonable.

If you uncomment that line, the memory footprint climbs to 283,316,224
bytes quite rapidly. That's a two order of magnitude difference!

If you uncomment the "gc.collect()" line, the process still hits
148,910,080 bytes.


-Greg


On Fri, Jun 25, 2010 at 7:58 PM, Guido van Rossum <guido at python.org> wrote:
> Do you have profiling data to support your claim?
>
> On Fri, Jun 25, 2010 at 7:48 PM,  <ghazel at gmail.com> wrote:
>> Hi,
>>
>> I'm interested in a feature which allows users to discard the locals
>> and globals references from frames held by a traceback object.
>>
>> Currently, traceback objects are used when capturing and re-raising
>> exceptions. However, they hold a reference to all frames, which hold a
>> reference to their locals and globals. These are not needed by the
>> default traceback output, and can cause serious memory bloat if a
>> reference to a traceback object is kept for any significant length of
>> time, and there are even big red warnings in the Python docs about
>> using them in one frame. (
>> http://docs.python.org/release/3.1/library/sys.html#sys.exc_info ).
>>
>> Example usage would be something like:
>>
>> import sys
>> try:
>>    1/0
>> except:
>>    t, v, tb = sys.exc_info()
>>    tb.clean()
>> # ... much later ...
>> raise t, v, tb
>>
>>
>> Which would be basically a function to do this:
>>
>> import sys
>> try:
>>    1/0
>> except:
>>    t, v, tb = sys.exc_info()
>>    c = tb
>>    while c:
>>        c.tb_frame.f_locals = None
>>        c.tb_frame.f_globals = None
>>        c = c.tb_next
>> # ... much later ...
>> raise t, v, tb
>>
>>
>> Twisted has done a very similar thing with their
>> twisted.python.failure.Failure object, which stringifies the traceback
>> data and discards the reference to the Python traceback entirely (
>> http://twistedmatrix.com/trac/browser/tags/releases/twisted-10.0.0/twisted/python/failure.py#L437
>> ) - they also replicate a lot of traceback printing functions to make
>> use of this stringified data.
>>
>> It's worth noting that cgitb and other applications make use of locals
>> and globals in its traceback output. However, I believe the vast
>> majority of traceback usage does not make use of these references, and
>> a significant penalty is paid as a result.
>>
>> Is there any interest in such a feature?
>>
>>
>> -Greg
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> http://mail.python.org/mailman/listinfo/python-ideas
>>
>
>
>
> --
> --Guido van Rossum (python.org/~guido)
>