Numarrays from C arrays without copying
Dear all, I would like to turn a large C array allocated by a library into a Numarray object without copying the contents. In other words I have a given array of double precision floats and I want to get a Numarray object, where the data pointer points to this array and no additional memory is allocated. Naturally when the reference count of the Numarray object gets back to zero, the object is freed, but the array itself is left as a nuisance for the C code. Is there a standard way of doing this? Thanks in advance, -- Ville
On Thu, 2004-04-22 at 08:51, Ville Hakulinen wrote:
Dear all,
I would like to turn a large C array allocated by a library into a Numarray object without copying the contents. In other words I have a given array of double precision floats and I want to get a Numarray object, where the data pointer points to this array and no additional memory is allocated. Naturally when the reference count of the Numarray object gets back to zero, the object is freed, but the array itself is left as a nuisance for the C code.
Is there a standard way of doing this?
No, not yet. You're not the first person to ask for this but I'd appreciate it if you'd explain why you need it. So far, not having that feature is a conscious omission to keep complexity down. Regards, Todd -- Todd Miller <jmiller@stsci.edu>
On Apr 22, 2004, at 7:10 AM, Todd Miller wrote:
I would like to turn a large C array allocated by a library into a Numarray object without copying the contents. In other words I have a given array of double precision floats and I want to get a Numarray object, where the data pointer points to this array and no additional memory is allocated. Naturally when the reference count of the Numarray object gets back to zero, the object is freed, but the array itself is left as a nuisance for the C code.
Is there a standard way of doing this?
No, not yet. You're not the first person to ask for this but I'd appreciate it if you'd explain why you need it. So far, not having that feature is a conscious omission to keep complexity down.
The reason I need this is two fold: 1. I want to export to Python existing Fortran arrays. If you want Fortran and Python to share array memory, you can't always expect Fortran programmers to use memory allocated in Python. In fact, since there is no standards conforming way to associate a Fortran array pointer with Python memory, you have to do the opposite, associate a Python array with Fortran memory. 2. In the CoArray Python module, we MUST use special memory allocators that are parallel aware. Because of these requirements, our only choice is to use Numeric. Regards, Craig
Hi Craig, I am sharing arrays between C and Python numarray (w/o any copying) for more than a year by now - please check my postings and (mostly) Todd's replys on this list. The code looks mostly like this: <code sniplet> int shape[4]= { n,nCams,pic_ny,pic_nx }; PyObject *obj, *obj2; obj = PyBuffer_FromReadWriteMemory(theRing.bufferPtr(), theRing.bufferSize()); obj2 =(PyObject*) NA_NewAllFromBuffer( 4, shape, tUInt16, obj, 0, 0, NA_ByteOrder(), 1, 1/*writable*/) ; wxASSERT(obj != NULL); wxASSERT(obj2 != NULL); PyDict_SetItemString(globals, "ring", obj2); Py_XDECREF(obj); Py_XDECREF(obj2); </code sniplet> It even works with records arrays! Luckily my arrays stay (mostly) around until the program quits. So I also don't have a good answer for 'who should free the memory?' As far as numarray/Python concerns - once it's reference counter goes to zero - a 'delete[]' would be needed if the array was allocated with a C++ 'new' but a 'free()' is required if the memory got "malloc()'ed" in C. Our internal discussion here found that the only real solution to this might be to extend numarray's object structure by a functions pointer ('callToKill') and also an argument pointer (the array should be delete'd or free'd). But then we felt this might be to much to ask (to much of a change / overhead) that this proposal would not get accepted - so we choose to not bring it up. Please comment, Regards Sebastian Haase On Thursday 22 April 2004 07:28 am, Craig Rasmussen wrote:
On Apr 22, 2004, at 7:10 AM, Todd Miller wrote:
I would like to turn a large C array allocated by a library into a Numarray object without copying the contents. In other words I have a given array of double precision floats and I want to get a Numarray object, where the data pointer points to this array and no additional memory is allocated. Naturally when the reference count of the Numarray object gets back to zero, the object is freed, but the array itself is left as a nuisance for the C code.
Is there a standard way of doing this?
No, not yet. You're not the first person to ask for this but I'd appreciate it if you'd explain why you need it. So far, not having that feature is a conscious omission to keep complexity down.
The reason I need this is two fold:
1. I want to export to Python existing Fortran arrays. If you want Fortran and Python to share array memory, you can't always expect Fortran programmers to use memory allocated in Python. In fact, since there is no standards conforming way to associate a Fortran array pointer with Python memory, you have to do the opposite, associate a Python array with Fortran memory.
2. In the CoArray Python module, we MUST use special memory allocators that are parallel aware.
Because of these requirements, our only choice is to use Numeric.
Regards, Craig
On Apr 22, 2004, at 10:46 AM, Sebastian Haase wrote:
obj = PyBuffer_FromReadWriteMemory(theRing.bufferPtr(), theRing.bufferSize()); obj2 =(PyObject*) NA_NewAllFromBuffer( 4, shape, tUInt16, obj, 0, 0, NA_ByteOrder(), 1, 1/*writable*/) ;
It looks like this will work. I thought Todd had been saying that this wasn't possible. Perhaps I did not phrase my question very well when I asked earlier. Thanks, Craig
On Thu, 2004-04-22 at 14:32, Craig Rasmussen wrote:
On Apr 22, 2004, at 10:46 AM, Sebastian Haase wrote:
obj = PyBuffer_FromReadWriteMemory(theRing.bufferPtr(), theRing.bufferSize()); obj2 =(PyObject*) NA_NewAllFromBuffer( 4, shape, tUInt16, obj, 0, 0, NA_ByteOrder(), 1, 1/*writable*/) ;
It looks like this will work.
It'll definitely work. I'd forgotten about it.
I thought Todd had been saying that this wasn't possible. Perhaps I did not phrase my question very well when I asked earlier.
Nope, your question was fine. I'm hopeful that PyArray_FromDimsAndData can be improved so that this becomes easier for numarray-1.0. Regards, Todd -- Todd Miller <jmiller@stsci.edu>
On Thu, Apr 22, 2004 at 09:10:07AM -0400, Todd Miller wrote:
No, not yet. You're not the first person to ask for this but I'd appreciate it if you'd explain why you need it. So far, not having that feature is a conscious omission to keep complexity down.
First I managed not to reply to numpy-discussion, so here we go again: We are building a Python interface to PETSc 2.2 (http://www-unix.mcs.anl.gov/petsc/petsc-2/) using SWIG. It would be very convenient to access the part of a PETSc vector that is local to a process(or) using Numarray. Other options would be to just write functions to access the part of a vector independently of Numarray or to figure out whether one could tell PETSc to use the memory area allocated by Numarray. -- Ville
participants (4)
-
Craig Rasmussen -
Sebastian Haase -
Todd Miller -
Ville Hakulinen