Guido van Rossum wrote:
On Sat, Jun 26, 2010 at 4:45 PM, M.-A. Lemburg email@example.com wrote:
Also note that garbage collection will not necessarily do what the user expects: it is well possible that big amounts of memory will stay allocated as unused space in pymalloc. This is not specific to the discussed case, but still a valid user concern. Greg Hazel observed this situation in his example.
Aha. So whereas the process size ballooned, there is no actual memory leak (his example threw away the exception each time through the loop), it's just that looking at process size is a bad way to assess memory leaks. I would like to reject this then as "that's just how Python's memory allocation works". As you say, it's not specific to this case; it comes up occasionally and it's just a matter of user education.
pymalloc has gotten a lot better since it was fixed in Python 2.5 to return unused chunks of memory to the OS, but we still have the issue of fragmented arenas with cases of just a few bytes keeping 256kB (the size of an arena) allocated.
I don't think anything should be done about __traceback__ either -- frameworks that have this problem can work around it in various ways. Or, at least I don't see a reason to panic and roll back the feature. Maybe eventually it can be improved by adding some kind of functionality to control some details of the behavior.
Not necessarily roll back the feature, but an implementation that deliberately introduces circular references is not really ideal.
Since tracebacks on exceptions are rarely used by applications, I think it would be better to turn them into weak references.
The arguments against doing this in the PEP appear rather weak compared to the potential issue for non-expert Python programmers:
""" Making the __traceback__ attribute a weak reference would avoid the problems with cyclic garbage. Unfortunately, it would make saving the Exception for later (as unittest does) more awkward, and it would not allow as much cleanup of the sys module. """
Special use cases that want to save the traceback for later use can always explicitly convert the traceback into a real (non-weak) reference. I don't understand the reference to the sys module cleanup, so can't comment on that.