Garbage collection fails after numarray exception
![](https://secure.gravatar.com/avatar/eaa03b1fc7f27d44967195e65bc62852.jpg?s=120&d=mm&r=g)
Hi all, I'm not sure if I have found a bug, or simply a type of behaviour that should not have surprised me -- but I most definitely did not expect the following numarray 0.8 behaviour:
If I execute these statements alongside 'top' or another memory monitor, I of course see a big memory increase after each call to 'zeros', as well as a big decrease after the first 'del a' -- but no change whatsoever any more after the second 'del a', not even if I explicitly call the garbage collector via the gc module. As far as I can tell, the occurrence of the exception somehow causes a permanent increase in a's refcount, with a big memory leak as a result. (In case anyone's curious about the context for this, the """ a + '' """" construct was taken straight from the Python Cookbook, as a way of determining whether an object exhibits string-like behaviour (useful for printing methods, etc.)) I understand that I can easily work around this problem so that numarray objects will not be fed to this construct to begin with, but I do wonder if it is really true (and intended) that numarray exceptions have the side effect of making certain potentially huge objects indestructible. Can anyone here shed some light on this? Many thanks in advance, -- Leo Breebaart <leo@lspace.org>
![](https://secure.gravatar.com/avatar/faf9400121dca9940496a7473b1d8179.jpg?s=120&d=mm&r=g)
On Tue, 2004-02-10 at 15:09, Leo Breebaart wrote:
It looks like a bug. I logged it on Source Forge and I'm working on the solution.
Many thanks in advance,
Thanks for the report, Todd -- Todd Miller <jmiller@stsci.edu>
![](https://secure.gravatar.com/avatar/802b07155a8ab3996b825eca778f770e.jpg?s=120&d=mm&r=g)
Todd Miller wrote:
Hmm.... Isn't this the standard python thing that it keeps the context of the last exception somewhere for queries? Just generate another exception and you should see that the memory taken by a is liberated because the indirect reference from sys.exc_traceback to a disappears. Rob -- Rob W.W. Hooft || rob@hooft.net || http://www.hooft.net/people/rob/
![](https://secure.gravatar.com/avatar/eaa03b1fc7f27d44967195e65bc62852.jpg?s=120&d=mm&r=g)
Rob Hooft <rob@hooft.net> writes:
Well, I've tried to test this, but even after multiple subsequent exceptions the memory taken up by the numarrays is not freed, at least not on my machine and with Python 2.3.3. Does it work for you if you try running the above code and then generating an exception? BTW Todd, if you are reading this: you said in your earlier response that you logged this bug on SourceForge somewhere, but I can't seem to find it in what I think is the numarray bug page, at: <http://sourceforge.net/tracker/?group_id=1369&atid=101369>. Am I looking in the wrong place? -- Leo Breebaart <leo@lspace.org>
![](https://secure.gravatar.com/avatar/faf9400121dca9940496a7473b1d8179.jpg?s=120&d=mm&r=g)
On Sun, 2004-02-29 at 04:28, Leo Breebaart wrote:
This is a good point, something I rediscovered for myself a couple weeks ago: since Python holds onto the context at the point of an exception, it will keep arrays alive in the exception context.
But... I looked into this some more and found that there were errors in the exception handling for ufuncs which resulted in leaked memory. Regards, Todd -- Todd Miller <jmiller@stsci.edu>
![](https://secure.gravatar.com/avatar/faf9400121dca9940496a7473b1d8179.jpg?s=120&d=mm&r=g)
Look here: http://sourceforge.net/tracker/index.php?func=detail&aid=894892&group_id=1369&atid=450446 Alternately, look for: [ 894892 ] Memory leak after ufunc exception in the tracker: Numarray Bugs Note that "Numarray Bugs" is distinct from the easier to find "Bugs" which is where Numeric bugs go. Regards, Todd -- Todd Miller <jmiller@stsci.edu>
![](https://secure.gravatar.com/avatar/faf9400121dca9940496a7473b1d8179.jpg?s=120&d=mm&r=g)
On Tue, 2004-02-10 at 15:09, Leo Breebaart wrote:
It looks like a bug. I logged it on Source Forge and I'm working on the solution.
Many thanks in advance,
Thanks for the report, Todd -- Todd Miller <jmiller@stsci.edu>
![](https://secure.gravatar.com/avatar/802b07155a8ab3996b825eca778f770e.jpg?s=120&d=mm&r=g)
Todd Miller wrote:
Hmm.... Isn't this the standard python thing that it keeps the context of the last exception somewhere for queries? Just generate another exception and you should see that the memory taken by a is liberated because the indirect reference from sys.exc_traceback to a disappears. Rob -- Rob W.W. Hooft || rob@hooft.net || http://www.hooft.net/people/rob/
![](https://secure.gravatar.com/avatar/eaa03b1fc7f27d44967195e65bc62852.jpg?s=120&d=mm&r=g)
Rob Hooft <rob@hooft.net> writes:
Well, I've tried to test this, but even after multiple subsequent exceptions the memory taken up by the numarrays is not freed, at least not on my machine and with Python 2.3.3. Does it work for you if you try running the above code and then generating an exception? BTW Todd, if you are reading this: you said in your earlier response that you logged this bug on SourceForge somewhere, but I can't seem to find it in what I think is the numarray bug page, at: <http://sourceforge.net/tracker/?group_id=1369&atid=101369>. Am I looking in the wrong place? -- Leo Breebaart <leo@lspace.org>
![](https://secure.gravatar.com/avatar/faf9400121dca9940496a7473b1d8179.jpg?s=120&d=mm&r=g)
On Sun, 2004-02-29 at 04:28, Leo Breebaart wrote:
This is a good point, something I rediscovered for myself a couple weeks ago: since Python holds onto the context at the point of an exception, it will keep arrays alive in the exception context.
But... I looked into this some more and found that there were errors in the exception handling for ufuncs which resulted in leaked memory. Regards, Todd -- Todd Miller <jmiller@stsci.edu>
![](https://secure.gravatar.com/avatar/faf9400121dca9940496a7473b1d8179.jpg?s=120&d=mm&r=g)
Look here: http://sourceforge.net/tracker/index.php?func=detail&aid=894892&group_id=1369&atid=450446 Alternately, look for: [ 894892 ] Memory leak after ufunc exception in the tracker: Numarray Bugs Note that "Numarray Bugs" is distinct from the easier to find "Bugs" which is where Numeric bugs go. Regards, Todd -- Todd Miller <jmiller@stsci.edu>
participants (3)
-
Leo Breebaart
-
Rob Hooft
-
Todd Miller