[Python-ideas] feature to make traceback objects usable without references to frame locals and globals
M.-A. Lemburg
mal at egenix.com
Sun Jun 27 01:45:48 CEST 2010
Guido van Rossum wrote:
> Please don't act so surprised. There are about 4 relevant PEPs: 344,
> 3109, 3110, 3134 (the latter replacing 344).
I knew about the discussions around chained exception. I wasn't
aware of the idea to keep a traceback object on the exception object
itself.
PEP 3134 also mentioned the case we're currently discussing:
"""
Open Issue: Garbage Collection
The strongest objection to this proposal has been that it creates
cycles between exceptions and stack frames [12]. Collection of
cyclic garbage (and therefore resource release) can be greatly
delayed.
>>> try:
>>> 1/0
>>> except Exception, err:
>>> pass
will introduce a cycle from err -> traceback -> stack frame -> err,
keeping all locals in the same scope alive until the next GC happens.
Today, these locals would go out of scope. There is lots of code
which assumes that "local" resources -- particularly open files -- will
be closed quickly. If closure has to wait for the next GC, a program
(which runs fine today) may run out of file handles.
Making the __traceback__ attribute a weak reference would avoid the
problems with cyclic garbage. Unfortunately, it would make saving
the Exception for later (as unittest does) more awkward, and it would
not allow as much cleanup of the sys module.
A possible alternate solution, suggested by Adam Olsen, would be to
instead turn the reference from the stack frame to the 'err' variable
into a weak reference when the variable goes out of scope [13].
"""
So obviously this case had already been discussed before.
Was a solution found and implemented that addresses the problem ?
> Also note that the
> traceback is only kept alove if the exception object is explicitly
> copied out of the except block that caught it -- normally the
> exception object is deleted when that block is left.
Right, but only if you do not use the exception object for other
purposes elsewhere.
If you do that a lot in your application, it appears that the
only way around keeping lots of traceback objects alive is
by explicitly setting .__traceback__ to None before storing
away the exception object.
Think of e.g. an application that does a long running calculation.
Such applications typically want to continue processing even
in case of errors and report all errors at the end of the
run. If a programmer is unaware of the traceback issue,
he'd likely run into a memory problem without really knowing
where to look for the cause.
Also note that garbage collection will not necessarily do what
the user expects: it is well possible that big amounts of
memory will stay allocated as unused space in pymalloc.
This is not specific to the discussed case, but still a valid
user concern. Greg Hazel observed this situation in his
example.
> --Guido
>
> On Sat, Jun 26, 2010 at 2:53 PM, M.-A. Lemburg <mal at egenix.com> wrote:
>> Antoine Pitrou wrote:
>>> On Sat, 26 Jun 2010 13:03:38 +0200
>>> "M.-A. Lemburg" <mal at egenix.com> wrote:
>>>> Greg Ewing wrote:
>>>>> ghazel at gmail.com wrote:
>>>>>
>>>>>> I'm interested in a feature which allows users to discard the locals
>>>>>> and globals references from frames held by a traceback object.
>>>>
>>>> Wouldn't it be better to write safer code and not store
>>>> a reference to the traceback object in the first place ?
>>>
>>> In Python 3, tracebacks are stored as an attribute of the corresponding
>>> exception:
>>>
>>>>>> try: 1/0
>>> ... except Exception as _: e = _
>>> ...
>>>>>> e.__traceback__
>>> <traceback object at 0x7ff69fdbf908>
>>
>> Ouch.
>>
>> So you explicitly need get rid off the traceback in Python3 if
>> you want to avoid keeping the associated objects alive during
>> exception processing ?
>>
>> I think that design decision needs to be revisited. Tracebacks
>> are needed for error reporting, but (normally) not for
>> managing error handling or recovery.
>>
>> E.g. it is not uncommon to store exception objects in a list for
>> later batched error reporting. With the traceback being referenced
>> on those object and the traceback chain keeping references to all
>> frames alive, this kind of processing won't be feasible
>> anymore.
>>
>> What's even more important is that programmers are unlikely
>> going to be aware of this detail and its implications.
>>
>>>> Also: What's the use case for creating traceback objects
>>>> outside the Python interpreter core ?
>>>
>>> He's not talking about creating traceback objects outside the core, but
>>> being able to reuse tracebacks created by the core without keeping alive
>>> a whole chain of objects.
>>
>> With the question I was referring to the suggestion by
>> Greg Ewing in which he seemed to imply that Pyrex and Cython
>> create traceback objects.
>>
>>> It's a real need when you want to do careful error handling/reporting
>>> without wasting too many resources. As already mentioned, Twisted has a
>>> bunch of code to work around that problem, since errors can be quite
>>> long-lived in a pipelined asynchronous execution model.
>>
>> With the above detail, I completely agree. In fact, more than that:
>> I think we should make storing the traceback in exception.__traceback__
>> optional and not the default, much like .__context__ and .__cause__.
>>
>> --
>> Marc-Andre Lemburg
>> eGenix.com
>>
>> Professional Python Services directly from the Source (#1, Jun 26 2010)
>>>>> Python/Zope Consulting and Support ... http://www.egenix.com/
>>>>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
>>>>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
>> ________________________________________________________________________
>> 2010-07-19: EuroPython 2010, Birmingham, UK 22 days to go
>>
>> ::: Try our new mxODBC.Connect Python Database Interface for free ! ::::
>>
>>
>> eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48
>> D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
>> Registered at Amtsgericht Duesseldorf: HRB 46611
>> http://www.egenix.com/company/contact/
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> http://mail.python.org/mailman/listinfo/python-ideas
>>
>
>
>
--
Marc-Andre Lemburg
eGenix.com
Professional Python Services directly from the Source (#1, Jun 27 2010)
>>> Python/Zope Consulting and Support ... http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
________________________________________________________________________
2010-07-19: EuroPython 2010, Birmingham, UK 21 days to go
::: Try our new mxODBC.Connect Python Database Interface for free ! ::::
eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
Registered at Amtsgericht Duesseldorf: HRB 46611
http://www.egenix.com/company/contact/
More information about the Python-ideas
mailing list