[Numpy-discussion] Memory leak in array protocol numarray<--numpy

[Travis Oliphant]

Francesc Altet wrote:
> Hi,
>
> I was tracking down a memory leak in PyTables and it boiled down to a problem 
> in the array protocol. The issue is easily exposed by:
>
> for i in range(1000000):
>     numarray.array(numpy.zeros(dtype=numpy.float64, shape=3))
>
> and looking at the memory consumption of the process. The same happens with:
>
> for i in range(1000000):
>     numarray.asarray(numpy.zeros(dtype=numpy.float64, shape=3))
>
> However, the numpy<--numarray sense seems to work well.
>
> for i in range(1000000):
>     numpy.array(numarray.zeros(type="Float64", shape=3))
>
> Using numarray 1.5.1 and numpy 1.0b1
>
> I think this is a relatively important problem, because it somewhat prevents a 
> smooth transition from numarray to NumPy. 
>
>   

I tracked the leak to the numarray function

NA_FromDimsStridesDescrAndData

This function calls NA_NewAllFromBuffer  with a brand-new buffer object 
when data is passed in (like in the case with the array protocol).  That 
function then takes  a reference to the buffer object but then the 
calling function never releases the reference it already holds.  This 
creates the leak.

I added the line

if (data) {Py_DECREF(buf);}

right after the call to NA_NewAllFromBuffer and the leak disappeared. 

For what it's worth, I also think the base object for the new numarray 
object should be the object passed in and not the C-object that is 
created from it. 

In other words in the NA_FromArrayStruct  function

a->base = cobj

should be replaced with

Py_INCREF(obj)
a->base = obj
Py_DECREF(cobj)


Best,


-Travis