Do you have profiling data to support your claim? On Fri, Jun 25, 2010 at 7:48 PM, wrote:

Hi,

I'm interested in a feature which allows users to discard the locals and globals references from frames held by a traceback object.

Currently, traceback objects are used when capturing and re-raising exceptions. However, they hold a reference to all frames, which hold a reference to their locals and globals. These are not needed by the default traceback output, and can cause serious memory bloat if a reference to a traceback object is kept for any significant length of time, and there are even big red warnings in the Python docs about using them in one frame. ( http://docs.python.org/release/3.1/library/sys.html#sys.exc_info ).

Example usage would be something like:

import sys try:    1/0 except:    t, v, tb = sys.exc_info()    tb.clean() # ... much later ... raise t, v, tb

Which would be basically a function to do this:

import sys try:    1/0 except:    t, v, tb = sys.exc_info()    c = tb    while c:        c.tb_frame.f_locals = None        c.tb_frame.f_globals = None        c = c.tb_next # ... much later ... raise t, v, tb

Twisted has done a very similar thing with their twisted.python.failure.Failure object, which stringifies the traceback data and discards the reference to the Python traceback entirely ( http://twistedmatrix.com/trac/browser/tags/releases/twisted-10.0.0/twisted/p... ) - they also replicate a lot of traceback printing functions to make use of this stringified data.

It's worth noting that cgitb and other applications make use of locals and globals in its traceback output. However, I believe the vast majority of traceback usage does not make use of these references, and a significant penalty is paid as a result.

Is there any interest in such a feature?

-Greg _______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas

-- --Guido van Rossum (python.org/~guido)

ghazel@gmail.com wrote:

I'm interested in a feature which allows users to discard the locals and globals references from frames held by a traceback object.

I'd like to take this further and remove the need for traceback objects to refer to a frame object at all. The standard traceback printout only needs two pieces of information from the traceback, the file name and line number. The line number is already present in the traceback object. All it would take is the addition of a file name attribute to the traceback object, and the frame reference could be made optional. This would be a big help for Pyrex and Cython, which currently have to create entire dummy frame objects in order to add entries to the traceback. Not only is this tedious and inefficient, it ties them to internal details of the frame object that are vulnerable to change. It would be much nicer to have a simple API function such as PyTraceback_AddEntry(filename, lineno) to add a frameless traceback object. -- Greg

Greg Ewing wrote:

ghazel@gmail.com wrote:

...

I'm interested in a feature which allows users to discard the locals and globals references from frames held by a traceback object.

Wouldn't it be better to write safer code and not store a reference to the traceback object in the first place ? Working with traceback objects can easily introduce hidden circular references, so it usually better not access them at all, if you don't have a need for them: Either like this: try: raise Exception except Exception, reason: pass or by using slicing: try: raise Exception except Exception, reason: errorclass, errorobject = sys.exc_info()[:2] pass If you do need to access them, make sure you clean up the reference as soon as you can: try: raise Exception except Exception, reason: errorclass, errorobject, tb = sys.exc_info() ... tb = None

I'd like to take this further and remove the need for traceback objects to refer to a frame object at all. The standard traceback printout only needs two pieces of information from the traceback, the file name and line number.

The line number is already present in the traceback object. All it would take is the addition of a file name attribute to the traceback object, and the frame reference could be made optional.

How would you make that reference optional ? The frames are needed to inspect the locals and globals of the call stack and debugging code relies on them being available. Also: What's the use case for creating traceback objects outside the Python interpreter core ? -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jun 26 2010)

...

...

Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

2010-07-19: EuroPython 2010, Birmingham, UK 22 days to go ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/

On Sat, 26 Jun 2010 13:03:38 +0200 "M.-A. Lemburg" wrote:

Greg Ewing wrote:

...

ghazel@gmail.com wrote:

...

I'm interested in a feature which allows users to discard the locals and globals references from frames held by a traceback object.

Wouldn't it be better to write safer code and not store a reference to the traceback object in the first place ?

In Python 3, tracebacks are stored as an attribute of the corresponding exception:

...

...

try: 1/0 ... except Exception as _: e = _ ... e.__traceback__

Also: What's the use case for creating traceback objects outside the Python interpreter core ?

He's not talking about creating traceback objects outside the core, but being able to reuse tracebacks created by the core without keeping alive a whole chain of objects. It's a real need when you want to do careful error handling/reporting without wasting too many resources. As already mentioned, Twisted has a bunch of code to work around that problem, since errors can be quite long-lived in a pipelined asynchronous execution model. Antoine.

Antoine Pitrou wrote:

On Sat, 26 Jun 2010 13:03:38 +0200 "M.-A. Lemburg" wrote:

...

Greg Ewing wrote:

...

ghazel@gmail.com wrote:

...

I'm interested in a feature which allows users to discard the locals and globals references from frames held by a traceback object.

Wouldn't it be better to write safer code and not store a reference to the traceback object in the first place ?

In Python 3, tracebacks are stored as an attribute of the corresponding exception:

...

...

...

try: 1/0 ... except Exception as _: e = _ ... e.__traceback__

Ouch. So you explicitly need get rid off the traceback in Python3 if you want to avoid keeping the associated objects alive during exception processing ? I think that design decision needs to be revisited. Tracebacks are needed for error reporting, but (normally) not for managing error handling or recovery. E.g. it is not uncommon to store exception objects in a list for later batched error reporting. With the traceback being referenced on those object and the traceback chain keeping references to all frames alive, this kind of processing won't be feasible anymore. What's even more important is that programmers are unlikely going to be aware of this detail and its implications.

...

Also: What's the use case for creating traceback objects outside the Python interpreter core ?

He's not talking about creating traceback objects outside the core, but being able to reuse tracebacks created by the core without keeping alive a whole chain of objects.

With the question I was referring to the suggestion by Greg Ewing in which he seemed to imply that Pyrex and Cython create traceback objects.

It's a real need when you want to do careful error handling/reporting without wasting too many resources. As already mentioned, Twisted has a bunch of code to work around that problem, since errors can be quite long-lived in a pipelined asynchronous execution model.

With the above detail, I completely agree. In fact, more than that: I think we should make storing the traceback in exception.__traceback__ optional and not the default, much like .__context__ and .__cause__. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jun 26 2010)

...

...

Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

2010-07-19: EuroPython 2010, Birmingham, UK 22 days to go ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/

Guido van Rossum wrote:

Please don't act so surprised. There are about 4 relevant PEPs: 344, 3109, 3110, 3134 (the latter replacing 344).

I knew about the discussions around chained exception. I wasn't aware of the idea to keep a traceback object on the exception object itself. PEP 3134 also mentioned the case we're currently discussing: """ Open Issue: Garbage Collection The strongest objection to this proposal has been that it creates cycles between exceptions and stack frames [12]. Collection of cyclic garbage (and therefore resource release) can be greatly delayed. >>> try: >>> 1/0 >>> except Exception, err: >>> pass will introduce a cycle from err -> traceback -> stack frame -> err, keeping all locals in the same scope alive until the next GC happens. Today, these locals would go out of scope. There is lots of code which assumes that "local" resources -- particularly open files -- will be closed quickly. If closure has to wait for the next GC, a program (which runs fine today) may run out of file handles. Making the __traceback__ attribute a weak reference would avoid the problems with cyclic garbage. Unfortunately, it would make saving the Exception for later (as unittest does) more awkward, and it would not allow as much cleanup of the sys module. A possible alternate solution, suggested by Adam Olsen, would be to instead turn the reference from the stack frame to the 'err' variable into a weak reference when the variable goes out of scope [13]. """ So obviously this case had already been discussed before. Was a solution found and implemented that addresses the problem ?

Also note that the traceback is only kept alove if the exception object is explicitly copied out of the except block that caught it -- normally the exception object is deleted when that block is left.

Right, but only if you do not use the exception object for other purposes elsewhere. If you do that a lot in your application, it appears that the only way around keeping lots of traceback objects alive is by explicitly setting .__traceback__ to None before storing away the exception object. Think of e.g. an application that does a long running calculation. Such applications typically want to continue processing even in case of errors and report all errors at the end of the run. If a programmer is unaware of the traceback issue, he'd likely run into a memory problem without really knowing where to look for the cause. Also note that garbage collection will not necessarily do what the user expects: it is well possible that big amounts of memory will stay allocated as unused space in pymalloc. This is not specific to the discussed case, but still a valid user concern. Greg Hazel observed this situation in his example.

--Guido

On Sat, Jun 26, 2010 at 2:53 PM, M.-A. Lemburg wrote:

...

Antoine Pitrou wrote:

...

On Sat, 26 Jun 2010 13:03:38 +0200 "M.-A. Lemburg" wrote:

...

Greg Ewing wrote:

...

ghazel@gmail.com wrote:

...

I'm interested in a feature which allows users to discard the locals and globals references from frames held by a traceback object.

Wouldn't it be better to write safer code and not store a reference to the traceback object in the first place ?

In Python 3, tracebacks are stored as an attribute of the corresponding exception:

...

...

...

try: 1/0 ... except Exception as _: e = _ ... e.__traceback__

Ouch.

So you explicitly need get rid off the traceback in Python3 if you want to avoid keeping the associated objects alive during exception processing ?

I think that design decision needs to be revisited. Tracebacks are needed for error reporting, but (normally) not for managing error handling or recovery.

E.g. it is not uncommon to store exception objects in a list for later batched error reporting. With the traceback being referenced on those object and the traceback chain keeping references to all frames alive, this kind of processing won't be feasible anymore.

What's even more important is that programmers are unlikely going to be aware of this detail and its implications.

...

...

Also: What's the use case for creating traceback objects outside the Python interpreter core ?

He's not talking about creating traceback objects outside the core, but being able to reuse tracebacks created by the core without keeping alive a whole chain of objects.

With the question I was referring to the suggestion by Greg Ewing in which he seemed to imply that Pyrex and Cython create traceback objects.

...

It's a real need when you want to do careful error handling/reporting without wasting too many resources. As already mentioned, Twisted has a bunch of code to work around that problem, since errors can be quite long-lived in a pipelined asynchronous execution model.

With the above detail, I completely agree. In fact, more than that: I think we should make storing the traceback in exception.__traceback__ optional and not the default, much like .__context__ and .__cause__.

-- Marc-Andre Lemburg eGenix.com

Professional Python Services directly from the Source (#1, Jun 26 2010)

...

...

...

Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

---

2010-07-19: EuroPython 2010, Birmingham, UK 22 days to go

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::

eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ _______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas

-- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jun 27 2010)

...

...

Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

2010-07-19: EuroPython 2010, Birmingham, UK 21 days to go ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/

On Sat, Jun 26, 2010 at 4:45 PM, M.-A. Lemburg wrote:

Also note that garbage collection will not necessarily do what the user expects: it is well possible that big amounts of memory will stay allocated as unused space in pymalloc. This is not specific to the discussed case, but still a valid user concern. Greg Hazel observed this situation in his example.

Aha. So whereas the process size ballooned, there is no actual memory leak (his example threw away the exception each time through the loop), it's just that looking at process size is a bad way to assess memory leaks. I would like to reject this then as "that's just how Python's memory allocation works". As you say, it's not specific to this case; it comes up occasionally and it's just a matter of user education. I don't think anything should be done about __traceback__ either -- frameworks that have this problem can work around it in various ways. Or, at least I don't see a reason to panic and roll back the feature. Maybe eventually it can be improved by adding some kind of functionality to control some details of the behavior. -- --Guido van Rossum (python.org/~guido)

On Sun, Jun 27, 2010 at 8:33 AM, Guido van Rossum wrote:

On Sat, Jun 26, 2010 at 4:45 PM, M.-A. Lemburg wrote:

...

Also note that garbage collection will not necessarily do what the user expects: it is well possible that big amounts of memory will stay allocated as unused space in pymalloc. This is not specific to the discussed case, but still a valid user concern. Greg Hazel observed this situation in his example.

Aha. So whereas the process size ballooned, there is no actual memory leak (his example threw away the exception each time through the loop), it's just that looking at process size is a bad way to assess memory leaks. I would like to reject this then as "that's just how Python's memory allocation works". As you say, it's not specific to this case; it comes up occasionally and it's just a matter of user education.

Leak? My example does not try to demonstrate a leak. It demonstrates excessive allocation. If you collect a few times after the test the memory usage of the process does drop to a reasonable level again. In a real-world application with long-lived traceback objects and more state, this excessive allocation becomes crippling. Go ahead, add a zero to the size of that list being created in the example. Without the traceback reference the process stays stable at 17MB, with the reference it balloons to consume all of the 2GB of RAM in my laptop, causing swapping. This is similar to the observed behavior of a real application, which is completely stable and requires relatively little memory when not using traceback objects, but quickly grows to an unmanageable size with traceback objects.

I don't think anything should be done about __traceback__ either -- frameworks that have this problem can work around it in various ways. Or, at least I don't see a reason to panic and roll back the feature. Maybe eventually it can be improved by adding some kind of functionality to control some details of the behavior.

This idea is about an improvement to control some details of the behavior. Keeping __traceback__ in more cases would be nothing to "panic" about, if tracebacks were not such "unsafe" objects. I have not yet seen any way for a framework to work around the references issue without discarding the traceback object entirely and losing the ability to re-raise. -Greg

On Sun, 27 Jun 2010 19:20:07 +0200 "M.-A. Lemburg" wrote:

Not necessarily roll back the feature, but an implementation that deliberately introduces circular references is not really ideal.

Since tracebacks on exceptions are rarely used by applications, I think it would be better to turn them into weak references.

How do you manage to get a strong reference before the traceback object gets deleted? Besides, an API which gives some information in an unreliable manner does not seem very user-friendly to me. I think I like the OP's idea better: allow to release the references to local and global variables from the frames in the traceback. This is keeps a lot of potentially large objects alive - some of which may also keep some OS resources busy.

On 28 June 2010 04:00, Antoine Pitrou wrote:

On Sun, 27 Jun 2010 19:20:07 +0200 "M.-A. Lemburg" wrote:

...

Not necessarily roll back the feature, but an implementation that deliberately introduces circular references is not really ideal.

Since tracebacks on exceptions are rarely used by applications, I think it would be better to turn them into weak references.

How do you manage to get a strong reference before the traceback object gets deleted?

At the beginning of the 'except' block, a strong local (but hidden) reference is obtained to the traceback (if it exists). This is deleted at the end of the 'except' block. Besides, an API which gives some information in an unreliable manner

does not seem very user-friendly to me.

I think I like the OP's idea better: allow to release the references to local and global variables from the frames in the traceback. This is keeps a lot of potentially large objects alive - some of which may also keep some OS resources busy.

I agree, with a variation - keep a weak reference to the frame in the traceback, and have a way for the application to specify that it wants to retain strong references to frames (so unittest for example can guarantee access to locals and globals). Possibly a context manager could be used for this, and decorators could be used to wrap an entire method in the context manager. A dummy frame would also be stored that contained enough info to replicate the existing stack trace (file, line number, etc). A strong reference could be obtained via the existing attribute, converted to a property, which does: a. return the internal reference if it is not a dummy frame; b. return the result of the weak reference if it still exists; c. return the dummy frame reference. I think this gives us the best of all worlds: 1. No strong reference to locals/globals in tracebacks by default; 2. Able to force strong references to frames; 3. We don't lose the ability to compose a full and complete stack trace. Tim Delaney

Nick Coghlan wrote:

On Sun, Jun 27, 2010 at 7:53 AM, M.-A. Lemburg wrote:

...

...

He's not talking about creating traceback objects outside the core, but being able to reuse tracebacks created by the core without keeping alive a whole chain of objects.

With the question I was referring to the suggestion by Greg Ewing in which he seemed to imply that Pyrex and Cython create traceback objects.

When Python code calls into Pyrex/C code which then call back into Python, I understand they insert dummy frames into the tracebacks to make the call stack more complete.

Thanks for that bit of information. I suppose they do this for better error reporting, right ? -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jun 28 2010)

...

...

Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

2010-07-19: EuroPython 2010, Birmingham, UK 20 days to go ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/

On Mon, 28 Jun 2010 13:14:21 +0200 "M.-A. Lemburg" wrote:

BTW: I wonder why proxy objects don't provide a direct access to the weakref object they are using. That would make keeping that extra variable around unnecessary.

Probably because the proxy would then have an additional attribute which isn't on the proxied object. Or, worse, it could also shadow one of the proxied object's existing attributes.

Perhaps someone could highlight a use case where the traceback is needed outside the except clause ?!

Well, it's needed if you want delayed error reporting and still display a comprehensive stack trace (rather than just the exception message). Frameworks often need this kind of behaviour; Twisted was already mentioned in this thread. But, even outside of frameworks, there are situations where you want to process a bunch of data and present all processing errors at the end. However, as the OP argued, most often you need the traceback in order to display file names and line numbers, but you don't need the attached variables (locals and globals). Regards Antoine.

On Mon, 28 Jun 2010 15:29:25 +0200 "M.-A. Lemburg" wrote:

Antoine Pitrou wrote:

...

On Mon, 28 Jun 2010 13:14:21 +0200 "M.-A. Lemburg" wrote:

...

BTW: I wonder why proxy objects don't provide a direct access to the weakref object they are using. That would make keeping that extra variable around unnecessary.

Probably because the proxy would then have an additional attribute which isn't on the proxied object. Or, worse, it could also shadow one of the proxied object's existing attributes.

That's a very weak argument, IMHO. It all depends on the naming of the attribute.

What name do you suggest that isn't cumbersome or awkward, and yet doesn't present any risk of conflict with attributes of the proxied object?

We've used such an approach on our mxProxy object for years without any problems or naming conflicts so far:

http://www.egenix.com/products/python/mxBase/mxProxy/ http://www.egenix.com/products/python/mxBase/mxProxy/doc/#_Toc162774452

Well, if some features of mxProxy are useful, perhaps it would be worth integrating them in the stdlib. Regards Antoine.

M.-A. Lemburg wrote:

Nick Coghlan wrote:

...

When Python code calls into Pyrex/C code which then call back into Python, I understand they insert dummy frames into the tracebacks to make the call stack more complete.

I suppose they do this for better error reporting, right ?

Yes. This is one reason I would like to be able to have traceback objects without a corresponding frame. Having to create an entire frame just to have somewhere to put the file name is very annoying. Also, being able to remove the whole frame from a traceback object seems like a cleaner and more complete way to implement what the OP wanted. -- Greg

On 6/28/2010 8:39 AM, Antoine Pitrou wrote:

However, as the OP argued, most often you need the traceback in order to display file names and line numbers, but you don't need the attached variables (locals and globals).

It then seems to me that one should extract the file name and line number info one wants to save before exiting the exception clause and let the traceback exception and traceback go on exit. Is a library function needed to make extraction easier? Perhaps this "The reason for this [deletion on exit] is that with the traceback attached to them, exceptions will form a reference cycle with the stack frame, keeping all locals in that frame alive until the next garbage collection occurs." could be strengthened into a better warning that this "That means that you have to assign the exception to a different name if you want to be able to refer to it after the except clause. " may really, really not be a good idea. -- Terry Jan Reedy

Terry Reedy wrote:

It then seems to me that one should extract the file name and line number info one wants to save before exiting the exception clause and let the traceback exception and traceback go on exit. Is a library function needed to make extraction easier?

That would require building your own custom traceback structure that would be incompatible with any of the standard functions available for formatting and printing tracebacks. -- Greg