On Sun, Jun 27, 2010 at 10:20, M.-A. Lemburg firstname.lastname@example.org wrote:
Guido van Rossum wrote:
On Sat, Jun 26, 2010 at 4:45 PM, M.-A. Lemburg email@example.com wrote:
Also note that garbage collection will not necessarily do what the user expects: it is well possible that big amounts of memory will stay allocated as unused space in pymalloc. This is not specific to the discussed case, but still a valid user concern. Greg Hazel observed this situation in his example.
Aha. So whereas the process size ballooned, there is no actual memory leak (his example threw away the exception each time through the loop), it's just that looking at process size is a bad way to assess memory leaks. I would like to reject this then as "that's just how Python's memory allocation works". As you say, it's not specific to this case; it comes up occasionally and it's just a matter of user education.
pymalloc has gotten a lot better since it was fixed in Python 2.5 to return unused chunks of memory to the OS, but we still have the issue of fragmented arenas with cases of just a few bytes keeping 256kB (the size of an arena) allocated.
I don't think anything should be done about __traceback__ either -- frameworks that have this problem can work around it in various ways. Or, at least I don't see a reason to panic and roll back the feature. Maybe eventually it can be improved by adding some kind of functionality to control some details of the behavior.
Not necessarily roll back the feature, but an implementation that deliberately introduces circular references is not really ideal.
But the circular reference only occurs if you store a reference outside the 'except' clause; Python 3 explicitly deletes any caught exception variable to prevent the loop.
Since tracebacks on exceptions are rarely used by applications, I think it would be better to turn them into weak references.
While I would be fine with that if that many people save raised exceptions outside of an 'except' clause, I doubt that happens very often and there would be backward-compatibility issues at this point.