[Numpy-discussion] Numarrays from C arrays without copying

[Sebastian Haase]

Hi Craig,
I am sharing arrays between C and Python numarray (w/o any copying) for more 
than a year by now - please check my postings and (mostly) Todd's replys on 
this list. The code looks mostly like this:
<code sniplet>
 int shape[4]= { n,nCams,pic_ny,pic_nx };
   PyObject *obj, *obj2;

   obj = PyBuffer_FromReadWriteMemory(theRing.bufferPtr(), 
theRing.bufferSize());
   obj2 =(PyObject*) NA_NewAllFromBuffer( 4, shape, tUInt16, obj, 0, 0, 
NA_ByteOrder(), 1, 1/*writable*/) ;
   wxASSERT(obj != NULL);
   wxASSERT(obj2 != NULL);
   PyDict_SetItemString(globals, "ring", obj2);
   Py_XDECREF(obj);
   Py_XDECREF(obj2);
</code sniplet>
It even works with records arrays!
Luckily my arrays stay (mostly) around until the program quits. 
So I also don't have a good answer for 'who should free the memory?'

As far as numarray/Python concerns - once it's reference counter goes to zero 
- a 'delete[]' would be needed if the array was allocated with a C++ 'new' 
but a 'free()' is required if the memory got "malloc()'ed" in C.
Our internal discussion here found that the only real solution to this might 
be to extend numarray's object structure by a functions pointer 
('callToKill') and also an  argument pointer (the array should be delete'd or 
free'd).
But then we felt this might be to much to ask (to much of a change / overhead) 
that this proposal would not get accepted - so we choose to not bring it up.
Please comment,
Regards
Sebastian Haase





On Thursday 22 April 2004 07:28 am, Craig Rasmussen wrote:
> On Apr 22, 2004, at 7:10 AM, Todd Miller wrote:
> >> I would like to turn a large C array allocated by a library into a
> >> Numarray object without copying the contents. In other words I have
> >> a given array of double precision floats and I want to get a Numarray
> >> object, where the data pointer points to this array and no additional
> >> memory is allocated. Naturally when the reference count of the
> >> Numarray
> >> object gets back to zero, the object is freed, but the array itself
> >> is left as a nuisance for the C code.
> >>
> >> Is there a standard way of doing this?
> >
> > No, not yet.  You're not the first person to ask for this but I'd
> > appreciate it if you'd explain why you need it.  So far,  not having
> > that feature is a conscious omission to keep complexity down.
>
> The reason I need this is two fold:
>
> 1. I want to export to Python existing Fortran arrays.  If you want
> Fortran and Python to share array memory, you can't always expect
> Fortran programmers to use memory allocated in Python.  In fact,
> since there is no standards conforming way to associate a Fortran
> array pointer with Python memory, you have to do the opposite,
> associate a Python array with Fortran memory.
>
> 2. In the CoArray Python module, we MUST use special memory
> allocators that are parallel aware.
>
> Because of these requirements, our only choice is to use Numeric.
>
> Regards,
> Craig