DECREFing and PyArray functions in C Extensions

[Travis Oliphant]

> In article <ya04s7hcox1.fsf at trymheim.ifi.uio.no>, Roger Hansen
> <rogerha at ifi.uio.no> wrote:
> 
> > Now assume I have an extension where a C function should create and
> > return a NumPy array.
> > 
> > static PyObject *
> > foo_retarray(PyObject *self, PyObject *args)
> > {
> >   int length, i;
> >   double x;
> >   double* a;        
> >   PyArrayObject *array;
> > 
> >   if(!PyArg_ParseTuple(args, "i", &length))
> >     return NULL;
> > 
> >   array = (PyArrayObject *) 
> >     PyArray_FromDims(1, &length, PyArray_DOUBLE);
> >     
> >   a = (double*) array->data;
> >   for (i = 0; i < length; i++) {
> >     x = (double) i/(length-1);
> >     a[i] = f(x);
> >   }
> > 
> >   return PyArray_Return(array);
> > }
> > 
> > Should I DECREF array here before the return? 
> 
> I'll take a stab at this and say, no.  Using PyArray_Return(array)
> automatically DECREFs the array.
> 

Thanks for helping out here.  This is the right answer but not the correct
reason.  You don't DECREF the array because you want to return the array
(with it's reference count).  If you DECREF the array, the Py_DECREF
routine will free the memory created with PyArray_FromDims.   My "general"
rule doesn't apply when you want to return the array you've created in C.

> 
> > If we rewrite it like this
> > 
> > static PyObject *
> > foo_retarray2(PyObject *self, PyObject *args)
> > {
> >   int length, i;
> >   double x;
> >   double* a;        
> > 
> >   if(!PyArg_ParseTuple(args, "i", &length))
> >     return NULL;
> > 
> >   a = malloc(length*sizeof(double));
> >   for (i = 0; i < length; i++) {
> >     x = (double) i/(length-1);
> >     a[i] = f(x);
> >   }
> > 
> >   return PyArray_FromDimsAndData(1, &length, PyArray_DOUBLE, (char*) a);
> > }
> > 
> > I'm sure we don't have to DECREF anything. But is this a good way to
> > create a NumPy array in C?

No, this is not the recommended way to do it.  Sometimes you have to do it
this way though (to link against C codes which allocate their own memory)
or to make the memory Fortran COMMON blocks look like arrays.
PyArray_FromDims creates it's own memory while PyArray_FromDimsAndData
does not create its own memory.  The array returned 
from PyArray_FromDimsAndData also does not set a
flag so the object destructor routine (called automatically when reference
count gets to zero) does not free the memory
pointed to by data member of the structure.  As a
result, PyArray_FromDimsAndData requires more attention from the
programmer because it can easily create a memory leak.

> 
> Hmmm...I would have said, yes.  You malloc the memory, then return a
> pointer, but you Must DECREF when any Pyarray is created, unless you
> use return PyArray_Return(obj) or PyArray_BuildValue with the "N"
> construct (quoting Travis).  In fact, I would say you have created two
> memory leaks.  One with the C array through malloc and the other with
> the PyArray created through PyArray_FromDimsAndData.  I don't think the
> latter uses the C allocated memory, but allocates its own.  Not sure,
> but I would check this out

PyArray_FromDimsAndData does not allocated memory for the data member but
uses the supplied pointer.  The programmer is responsible for making sure
such memory is later deallocated when using this call.  That's why it is
not recommended use.  

-Travis