Hi, I'm running a simulation program partly written in C partly in Python and I use Numeric arrays extensively. At the end of a single run I have a bunch of Numeric arrays containing partial results that I don't need anymore. These arrays have been created on the C side of my code and then copied (actually linked via PyArray_FromDimsAndData) to Numeric arrays. How can I release these arrays and get back memory in order to be able to start the next run? Thanks in advance, Andrea. --- Andrea Riciputi "Science is like sex: sometimes something useful comes out, but that is not the reason we are doing it" -- (Richard Feynman)
On Tuesday 24 February 2004 22:10, Andrea Riciputi wrote:
anymore. These arrays have been created on the C side of my code and then copied (actually linked via PyArray_FromDimsAndData) to Numeric arrays. How can I release these arrays and get back memory in order to be able to start the next run?
I am not aware of any clean way. A hack that might help you out is array->flags |= OWN_DATA; The data will then be freed when the array is released. However, there are a few caveats: 1) This is not documented. 2) It works only if the data was allocated by the same memory manager that is used by NumPy. 3) You have to make sure that no other code keeps a reference to the data. If modifying the C code is an option, then it is probably the better one. Konrad. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen@cnrs-orleans.fr Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.56.24 Rue Charles Sadron | Fax: +33-2.38.63.15.17 45071 Orleans Cedex 2 | Deutsch/Esperanto/English/ France | Nederlands/Francais -------------------------------------------------------------------------------
It could be, but how? On 25 Feb 2004, at 11:49, Konrad Hinsen wrote:
If modifying the C code is an option, then it is probably the better one.
--- Andrea Riciputi "Science is like sex: sometimes something useful comes out, but that is not the reason we are doing it" -- (Richard Feynman)
On Wednesday 25 February 2004 16:18, Andrea Riciputi wrote:
It could be, but how?
If you don't mind making it Python-specific, you can have it create NumPy arrays directly and return them to Python. If you do want to keep it "Python clean", then modify it in such a way that it doesn't allocate any memory, but takes the data space of the array as input. Konrad. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen@cnrs-orleans.fr Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.56.24 Rue Charles Sadron | Fax: +33-2.38.63.15.17 45071 Orleans Cedex 2 | Deutsch/Esperanto/English/ France | Nederlands/Francais -------------------------------------------------------------------------------
I saw but it's too late. At this point I can't modify my code so deeply, but I'll keep your suggestion for the next project. Just for sake of curiosity, working with C++ arrays (I mean those defined in STL) would have simpliefied things? A. On 25 Feb 2004, at 17:32, Konrad Hinsen wrote:
If you don't mind making it Python-specific, you can have it create NumPy arrays directly and return them to Python. If you do want to keep it "Python clean", then modify it in such a way that it doesn't allocate any memory, but takes the data space of the array as input.
Konrad.
--- Andrea Riciputi "Science is like sex: sometimes something useful comes out, but that is not the reason we are doing it" -- (Richard Feynman)
On Tue, Feb 24, 2004 at 10:10:51PM +0100, Andrea Riciputi wrote:
Hi, I'm running a simulation program partly written in C partly in Python and I use Numeric arrays extensively. At the end of a single run I have a bunch of Numeric arrays containing partial results that I don't need anymore. These arrays have been created on the C side of my code and then copied (actually linked via PyArray_FromDimsAndData) to Numeric arrays. How can I release these arrays and get back memory in order to be able to start the next run?
Simple answer: don't use PyArray_FromDimsAndData. Allocate your array with Python using PyArray_FromDims, and use the data element for your C array, instead of allocating using C and convincing Python to use that. However, there are instances where you can't do that (the memory was allocated by some library, for instance, that needs a custom deallocator, or is from another array type). Then I'd suggest 1) copying the data (depends on how much there is) 2) if the array memory was originally allocated using malloc, you set the OWNDATA flag so Python will free it (going from memory here on the name; check the source). Note that you should be really sure you and Python are using the same allocator/deallocator routines. 3) or using the following trick, best described with some code: MyOtherArrayType *A = a_library_routine_that_allocates_a_routine(); PyObject *oA = PyArray_FromDimsAndData(rank, dims, PyArray_DOUBLE, (char *)A->array_data); PyObject *w = PyCObject_FromVoidPtr(A, free_myotherarraytype); ((PyArrayObject *)oA)->base = w; Now, when oA is DECREF'd latter, oA->base will also be DECREF'd. Since it's a PyCObject, free_myotherarraytype(A) will be called, free'ing the data. Since PyArray_FromDimsAndData was used, Python won't try to deallocate oA->data. I haven't checked whether this works under numarray (and I'm sure I saw a better way to do it there). I have used it for a Numeric<->Blitz++ wrapper. -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm@physics.mcmaster.ca
participants (3)
-
Andrea Riciputi -
David M. Cooke -
Konrad Hinsen