lists and c-extensions (Swig), almost there, but still need help

[Marcus Stojek]

Hi,
sorry for bothering you again, but this is getting a real problem for
me. I need to write some c extensions that will receive large lists
(float only) do some calcs and return other lists with the results.
I have learned a lot about Swig and typemaps and below you find an
example of what I have right now. If I pass an list argument (input)
to my c extension it is transformed into a contiguous Numeric array
and a second var holding its size. The pointer to the data field of
this array and the size var are passed to the c-function.

If the results have to be stored in another list, I have to pass the
size of this resultlist to my python function. The second typemap
allocates a memory block of this size and passes the pointer and the
size to the c-function.

When returning from the c-function the "out-lists" are transformed
into python Numeric array objects and are attached to the result
tuple.

so i can write:

import Numeric
import ctest
result=ctest([1,2,3,4],8) #inputlist, size of resultlist

and will receive:
(30, [1,2,3,4,2,4,6,8]) #return value of c function and resultlist

all this is really wonderful. But if I do:

for i in range(1000):
    result=ctest(longlist,largeresult)

i can watch my heap grow and grow. I have played around with some
Py_DECREF here and there but this isn't getting me anywhere.
I can't get rid of these memory leaks and I have no idea what to do
now.

Please,please, if someone has written something similar or knows what
to do ... let me know.

Thanks
marcus

P.S. There is a remark of Konrad Hinsen that PyArray_FromDimsAndData()
might never be freed. Although I don't understand the reason I tried
PyArray_FromDims with copying the data one by one. -> Slow and didn't
help.



#--------------------snip-------------------
%module ctest
%{
#include "ctest.h"
%}
//-----------------------------------------------------------------------------
typedef double d_list_in;
typedef double d_list_out;
//-----------------------------------------------------------------------------
%include "typemaps.i"
//-----------------------------------------------------------------------------
// doublelist coming in
%typemap(in) (d_list_in *, int) {
  int size;
  if (!PySequence_Check($input)) {
    PyErr_SetString(PyExc_ValueError,"Expected a sequence as double
(in)");
    return NULL;
  }
  size=PySequence_Length($input);
  PyArrayObject  *data;
  data = (PyArrayObject *) PyArray_ContiguousFromObject($input,
PyArray_DOUBLE, 1, 0);
  if (data == NULL){
    PyErr_SetString(PyExc_ValueError,"PyArray_ContiguousFromObject
returns NULL");
     return NULL;
  }
  $1= (double *) data->data;
  $2= size;
}
%typemap(freearg) double * {
   if ($1) free($1);
}
//-----------------------------------------------------------------------------
//-----------------------------------------------------------------------------
// result list that will be filled with doubles
%typemap(in) ( d_list_out *, int) {
   if (!PyInt_Check($input)) {
       PyErr_SetString(PyExc_ValueError, "Expecting an integer (size
of double resultlist)");
       return NULL;
   }
   $2 = PyInt_AsLong($input);
   if ($2 < 0) {
       PyErr_SetString(PyExc_ValueError, "Positive integer expected");
       return NULL;
   }
   $1 = (double *) malloc($2*sizeof(double));
}
//
%typemap(freearg) d_list_out * {
   if ($1) free($1);
}
//-----------------------------------------------------------------------------
//-----------------------------------------------------------------------------
//filled out list that will be appended to current result
%typemap(argout) (d_list_out *, int) {
    PyObject *retarray;
    PyObject *o2, *o3;
    int size[1];

    size[0] = $2
    size[1] = 0;

    retarray = (PyObject *)
PyArray_FromDimsAndData(1,size,PyArray_DOUBLE,(char *) $1);

    if (retarray == NULL){
      PyErr_SetString(PyExc_ValueError,"PyArray_FromDims returns
NULL");
       return NULL;
    }

    if ((!$result) || ($result == Py_None)) {
        $result = retarray;
    } else {
        if (!PyTuple_Check($result)) {
            PyObject *o2 = $result;
            $result = PyTuple_New(1);
            PyTuple_SET_ITEM($result,0,o2);
        }
        o3 = PyTuple_New(1);
        PyTuple_SetItem(o3,0,retarray);
        o2 = $result;
        $result = PySequence_Concat(o2,o3);
        Py_DECREF(o2);
        Py_DECREF(o3);
    }
}
//-----------------------------------------------------------------------------
//include header file or enter declarations
//-----------------------------------------------------------------------------
double calclist (d_list_in *, int, d_list_out *, int);
//-----------------------------------------------------------------------------