Numeric memory leak when building Numeric.array from numarray.array
Hi, I'm facing a memory leak on an application that has to use numarray and Numeric (because of external dependencies). The problem occurs when building a Numeric array from a numarray array: import Numeric import numarray import sys atest = numarray.arange(200) temp = Numeric.array(atest) print sys.getrefcount(atest) # prints 3 print sys.getrefcount(temp) # prints 2 I'm running numarray 1.5.2 and Numeric 24.2 I can work around this by using an intermediate string representation: temp = Numeric.fromstring(atest.tostring(), atest.typecode()) temp.shape = atest.shape -- Alexandre Fayolle LOGILAB, Paris (France) Formations Python, Zope, Plone, Debian: http://www.logilab.fr/formations Développement logiciel sur mesure: http://www.logilab.fr/services Informatique scientifique: http://www.logilab.fr/science Reprise et maintenance de sites CPS: http://www.migration-cms.com/
El dj 07 de 12 del 2006 a les 16:50 +0100, en/na Alexandre Fayolle va escriure:
Hi,
I'm facing a memory leak on an application that has to use numarray and Numeric (because of external dependencies).
The problem occurs when building a Numeric array from a numarray array:
import Numeric import numarray import sys atest = numarray.arange(200) temp = Numeric.array(atest) print sys.getrefcount(atest) # prints 3 print sys.getrefcount(temp) # prints 2
I'm running numarray 1.5.2 and Numeric 24.2
Yeah, it seems like the array protocol implementation in Numeric is leaking. Unfortunately, as Numeric maintenance has been dropped, there is small chances that this would be fixed in the future.
I can work around this by using an intermediate string representation:
temp = Numeric.fromstring(atest.tostring(), atest.typecode()) temp.shape = atest.shape
Another (faster) workaround would be: temp2 = Numeric.fromstring(atest._data, typecode=atest.typecode()) which is pretty fast: In [20]:Timer("Numeric.fromstring(atest._data, typecode=atest.typecode())", "import numarray, Numeric; atest=numarray.arange(200)").repeat(3,10000) Out[20]:[0.18092107772827148, 0.13870906829833984, 0.13995194435119629] i.e. more than 2x faster than your current solution: In [21]:Timer("Numeric.fromstring(atest.tostring(), typecode=atest.typecode())", "import numarray, Numeric; atest=numarray.arange(200)").repeat(3,10000) Out[21]:[0.37756705284118652, 0.32852792739868164, 0.32704305648803711] and similar in speed to the native .array() and .asarray() based on the array protocol: In [22]:Timer("Numeric.array(atest)", "import numarray, Numeric; atest=numarray.arange(200)").repeat(3,10000) Out[22]:[0.17277789115905762, 0.12470793724060059, 0.12530016899108887] In [23]:Timer("Numeric.asarray(atest)", "import numarray, Numeric; atest=numarray.arange(200)").repeat(3,10000) Out[23]:[0.20457005500793457, 0.15211081504821777, 0.15212082862854004] As an aside, and curiously enough, Numeric.array() (a copy is done) is faster than Numeric.asarray() (a copy shouldn't be done) :-/ HTH, -- Francesc Altet | Be careful about using the following code -- Carabos Coop. V. | I've only proven that it works, www.carabos.com | I haven't tested it. -- Donald Knuth
On Thu, Dec 07, 2006 at 05:36:22PM +0100, Francesc Altet wrote:
El dj 07 de 12 del 2006 a les 16:50 +0100, en/na Alexandre Fayolle va escriure:
Hi,
I'm facing a memory leak on an application that has to use numarray and Numeric (because of external dependencies).
The problem occurs when building a Numeric array from a numarray array:
import Numeric import numarray import sys atest = numarray.arange(200) temp = Numeric.array(atest) print sys.getrefcount(atest) # prints 3 print sys.getrefcount(temp) # prints 2
I'm running numarray 1.5.2 and Numeric 24.2
Yeah, it seems like the array protocol implementation in Numeric is leaking. Unfortunately, as Numeric maintenance has been dropped, there is small chances that this would be fixed in the future.
I can work around this by using an intermediate string representation:
temp = Numeric.fromstring(atest.tostring(), atest.typecode()) temp.shape = atest.shape
Another (faster) workaround would be:
temp2 = Numeric.fromstring(atest._data, typecode=atest.typecode())
Nice! Thanks Francesc. -- Alexandre Fayolle LOGILAB, Paris (France) Formations Python, Zope, Plone, Debian: http://www.logilab.fr/formations Développement logiciel sur mesure: http://www.logilab.fr/services Informatique scientifique: http://www.logilab.fr/science Reprise et maintenance de sites CPS: http://www.migration-cms.com/
El dl 11 de 12 del 2006 a les 14:16 +0100, en/na Alexandre Fayolle va escriure:
I can work around this by using an intermediate string representation:
temp = Numeric.fromstring(atest.tostring(), atest.typecode()) temp.shape = atest.shape
Another (faster) workaround would be:
temp2 = Numeric.fromstring(atest._data, typecode=atest.typecode())
Nice!
Well, I've to say that this approach only work for contiguous, non-offseted arrays, as can be seen in: In [59]:atest = numarray.arange(10) In [60]:Numeric.fromstring(atest[5:]._data, typecode=atest.typecode()) Out[60]:array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9],'i') # wrong! In [61]:Numeric.fromstring(atest[5:].tostring(), atest.typecode()) Out[61]:array([5, 6, 7, 8, 9],'i') # good In [62]:Numeric.fromstring(atest[::2]._data, typecode=atest.typecode()) Out[62]:array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9],'i') # wrong! In [63]:Numeric.fromstring(atest[::2].tostring(), atest.typecode()) Out[63]:array([0, 2, 4, 6, 8],'i') # good So, be careful when using it. I'd rather keep using your approach, which is the faster one that is completely general. -- Francesc Altet | Be careful about using the following code -- Carabos Coop. V. | I've only proven that it works, www.carabos.com | I haven't tested it. -- Donald Knuth
participants (2)
-
Alexandre Fayolle -
Francesc Altet