
What's the right way to make a new numpy array that's a copy of some C data? There doesn't seem to be any API like PyArray_NewFromDescr that /copies/ the void*data pointer for you. Do I have to write my own loops for this? I can do that, it just seems like it should be a library function already, so I'm guessing I'm just overlooking it. There seem to be lots of APIs that will wrap pre-existing memory, but the ones that allocate for you do not seem to copy. A related question -- I'm only trying to copy in order to save myself a little hassle regarding how to clean up the allocated chunks. If there's some simple way to trigger a particular deallocation function to be called at the right time, then that would be the ideal, really. Does that exist? Thanks! --bb

On 4/19/07, Bill Baxter <wbaxter@gmail.com> wrote:
What's the right way to make a new numpy array that's a copy of some C data?
There doesn't seem to be any API like PyArray_NewFromDescr that /copies/ the void*data pointer for you. Do I have to write my own loops for this? I can do that, it just seems like it should be a library function already, so I'm guessing I'm just overlooking it. There seem to be lots of APIs that will wrap pre-existing memory, but the ones that allocate for you do not seem to copy.
A related question -- I'm only trying to copy in order to save myself a little hassle regarding how to clean up the allocated chunks. If there's some simple way to trigger a particular deallocation function to be called at the right time, then that would be the ideal, really. Does that exist?
This is a situation I have been waiting to address for a long time ! In our case the data size is generally considered to be to large to accept the copy-solution. A cookbook-wiki entry would be wonderful! Generally we would have data memory that was allocated via alloc() and data that was allocated via new[] -- so two different deallocation functions (free() and delete[], respectively) would be required for this to be trigged, once the reference counter goes back to zero. Thanks, Sebastian Haase

Bill Baxter wrote:
What's the right way to make a new numpy array that's a copy of some C data?
There doesn't seem to be any API like PyArray_NewFromDescr that /copies/ the void*data pointer for you. Do I have to write my own loops for this? I can do that, it just seems like it should be a library function already, so I'm guessing I'm just overlooking it. There seem to be lots of APIs that will wrap pre-existing memory, but the ones that allocate for you do not seem to copy.
What do you mean by /copies/ the void * data pointer for you? Do you mean the API would 1) Create new memory for the array 2) Copy the data from another void * pointer to the memory just created for the new array? If that is what you mean, then you are right there is no such API. I'm not sure that there needs to be one. It is a two-liner using memcpy.
A related question -- I'm only trying to copy in order to save myself a little hassle regarding how to clean up the allocated chunks. If there's some simple way to trigger a particular deallocation function to be called at the right time, then that would be the ideal, really.
No, there is no place to store that information in NumPy. Either the ndarray dealloc function frees the memory it created or it doesn't free any memory. I think the best thing to do in this case would be to create a memory object wrapping the pointer and then point the ndarray to it as the source of memory. -Travis

On 4/21/07, Travis Oliphant <oliphant.travis@ieee.org> wrote:
Bill Baxter wrote:
What's the right way to make a new numpy array that's a copy of some C data?
There doesn't seem to be any API like PyArray_NewFromDescr that /copies/ the void*data pointer for you. Do I have to write my own loops for this? I can do that, it just seems like it should be a library function already, so I'm guessing I'm just overlooking it. There seem to be lots of APIs that will wrap pre-existing memory, but the ones that allocate for you do not seem to copy.
What do you mean by /copies/ the void * data pointer for you? Do you mean the API would
1) Create new memory for the array 2) Copy the data from another void * pointer to the memory just created for the new array?
If that is what you mean, then you are right there is no such API. I'm not sure that there needs to be one. It is a two-liner using memcpy.
Yes, I was thinking about memcpy() -- but how about non-contiguous data ? or other non well behaved ndarrays (non-aligned, byte-swapped, ...?) ?
A related question -- I'm only trying to copy in order to save myself a little hassle regarding how to clean up the allocated chunks. If there's some simple way to trigger a particular deallocation function to be called at the right time, then that would be the ideal, really.
No, there is no place to store that information in NumPy. Either the ndarray dealloc function frees the memory it created or it doesn't free any memory. I think the best thing to do in this case would be to create a memory object wrapping the pointer and then point the ndarray to it as the source of memory.
Yes, I think one would probably want to create a custom-made python class that provides the memory as buffer or so. This is what you meant - right ? And that class could then define /any/ function to be called once the ref.count goes to zero - right? Could someone with C-API knowledge put a sample together !? This would also be quite useful to be used with a SWIG output typemap. -Sebastian

On 4/22/07, Travis Oliphant <oliphant.travis@ieee.org> wrote:
Bill Baxter wrote:
What's the right way to make a new numpy array that's a copy of some C data?
What do you mean by /copies/ the void * data pointer for you? Do you mean the API would
1) Create new memory for the array 2) Copy the data from another void * pointer to the memory just created for the new array?
Yes.
If that is what you mean, then you are right there is no such API. I'm not sure that there needs to be one. It is a two-liner using memcpy.
Ok, I see. That's not too hard. Though as Sebastian said below it would be a little trickier if I needed to do a transpose or something at the same time. I guess for those cases it should be possible to make a temporary PyArrayObject with PyArray_NewFromDescr that wraps the memory and then do a PyArray_GETCONTIGUOUS?
A related question -- I'm only trying to copy in order to save myself a little hassle regarding how to clean up the allocated chunks. If there's some simple way to trigger a particular deallocation function to be called at the right time, then that would be the ideal, really.
No, there is no place to store that information in NumPy. Either the ndarray dealloc function frees the memory it created or it doesn't free any memory. I think the best thing to do in this case would be to create a memory object wrapping the pointer and then point the ndarray to it as the source of memory.
Hmm. Sounds a little over my head. I'll stick with copying for now. --bb
participants (3)
-
Bill Baxter
-
Sebastian Haase
-
Travis Oliphant