ANN: numpy.i - added managed deallocation to ARGOUTVIEW_ARRAY1 (ARGOUTVIEWM_ARRAY1)
Dear List,
after I tried to write a simple ARGOUTVIEW_ARRAY1 example (see http://code.google.com/p/ezwidgets/wiki/NumpySWIGMinGW#A_simple_ARGOUTVIEW_A... ), I started wondering about memory deallocation. Turns out a few clever people already did all the thinking (see http://blog.enthought.com/?p=62 ) and a few more clever people use this in a swig file (see http://niftilib.sourceforge.net/pynifti, file nifticlib.i).
All this concentrated knowledge helped me implement a single ARGOUTVIEWM_ARRAY1 fragment in numpy.i to do some testing (see attached numpy.i).
How to use it? In yourfile.i, the %init function has a few lines added to it:
%init %{ import_array();
/* initialize the new Python type for memory deallocation */ _MyDeallocType.tp_new = PyType_GenericNew; if (PyType_Ready(&_MyDeallocType) < 0) return; %}
... and that's it! then just use ARGOUTVIEWM_ARRAY1 instead of ARGOUTVIEW_ARRAY1 and python does the deallocation for you when the python array is destroyed (see the examples attached).
Everything compiles, but it would be nice to get rid of the warnings...
ezalloc_wrap.c:2763: warning: initialization from incompatible pointer type ezalloc_wrap.c: In function `_wrap_alloc_managed': ezalloc_wrap.c:2844: warning: assignment from incompatible pointer type writing build\temp.win32-2.5\Release\_ezalloc.def
Compilation: python setup_alloc.py build
Testing: The attached test_alloc.py does 2048 allocations of 1MB each for managed and unmanaged arrays. Output on my XP laptop with 1GB RAM as follows:
ARGOUTVIEWM_ARRAY1 (managed arrays) - 2048 allocations (1048576 bytes each) Done!
ARGOUTVIEW_ARRAY1 (unmanaged, leaking) - 2048 allocations (1048576 bytes each) Step 482 failed
TODO: Find a better name for the methods (not sure I like ARGOUTVIEWM_ARRAY), then do the missing fragments (2D arrays), clear the warnings and verify the method for a possible inclusion in the official numpy.i file.
Thank you for reading!
Regards, Egor
Sorry to reply to myself.
I've managed to get rid of the two warnings I was talking about, added proper attribution to the method (thank you Travis Oliphant!) and am now raising an exception in the swig wrapper of my example code when running out of memory (although using arg1,arg2,arg3 in %exception feels to me a bit hack-ish to me).
Oh, and another detail: since I'm allocating memory for int arrays, the memory allocated in bytes is of course 4x what I initially wrote it was (in test_alloc.py, on my 32 bit machine)!
Anyway, made a new zip with my example. As before, there is only one fragment available: ARGOUTVIEWM_ARRAY1, the other fragments still need to be written (I guess when the ARGOUTVIEWM_ARRAY1 fragment is validated).
That's all I can think of. As usual, comments welcome!
Regards, Egor
Egor Zindy wrote:
Dear List,
after I tried to write a simple ARGOUTVIEW_ARRAY1 example (see http://code.google.com/p/ezwidgets/wiki/NumpySWIGMinGW#A_simple_ARGOUTVIEW_A... ), I started wondering about memory deallocation. Turns out a few clever people already did all the thinking (see http://blog.enthought.com/?p=62 ) and a few more clever people use this in a swig file (see http://niftilib.sourceforge.net/pynifti, file nifticlib.i).
All this concentrated knowledge helped me implement a single ARGOUTVIEWM_ARRAY1 fragment in numpy.i to do some testing (see attached numpy.i).
How to use it? In yourfile.i, the %init function has a few lines added to it:
%init %{ import_array();
/* initialize the new Python type for memory deallocation */ _MyDeallocType.tp_new = PyType_GenericNew; if (PyType_Ready(&_MyDeallocType) < 0) return; %}
... and that's it! then just use ARGOUTVIEWM_ARRAY1 instead of ARGOUTVIEW_ARRAY1 and python does the deallocation for you when the python array is destroyed (see the examples attached).
Everything compiles, but it would be nice to get rid of the warnings...
ezalloc_wrap.c:2763: warning: initialization from incompatible pointer type ezalloc_wrap.c: In function `_wrap_alloc_managed': ezalloc_wrap.c:2844: warning: assignment from incompatible pointer type writing build\temp.win32-2.5\Release\_ezalloc.def
Compilation: python setup_alloc.py build
Testing: The attached test_alloc.py does 2048 allocations of 1MB each for managed and unmanaged arrays. Output on my XP laptop with 1GB RAM as follows:
ARGOUTVIEWM_ARRAY1 (managed arrays) - 2048 allocations (1048576 bytes each) Done!
ARGOUTVIEW_ARRAY1 (unmanaged, leaking) - 2048 allocations (1048576 bytes each) Step 482 failed
TODO: Find a better name for the methods (not sure I like ARGOUTVIEWM_ARRAY), then do the missing fragments (2D arrays), clear the warnings and verify the method for a possible inclusion in the official numpy.i file.
Thank you for reading!
Regards, Egor
Hello list,
I just finished coding all the ARGOUTVIEWM fragments in numpy.i and wrote a wiki to explain what I did: http://code.google.com/p/ezwidgets/wiki/NumpyManagedMemory
No new information in there, but everything (explanations, links, code) is provided on a single page.
Regards, Egor
Egor Zindy wrote:
Dear List,
after I tried to write a simple ARGOUTVIEW_ARRAY1 example (see http://code.google.com/p/ezwidgets/wiki/NumpySWIGMinGW#A_simple_ARGOUTVIEW_A... ), I started wondering about memory deallocation. Turns out a few clever people already did all the thinking (see http://blog.enthought.com/?p=62 ) and a few more clever people use this in a swig file (see http://niftilib.sourceforge.net/pynifti, file nifticlib.i).
All this concentrated knowledge helped me implement a single ARGOUTVIEWM_ARRAY1 fragment in numpy.i to do some testing (see attached numpy.i).
How to use it? In yourfile.i, the %init function has a few lines added to it:
%init %{ import_array();
/* initialize the new Python type for memory deallocation */ _MyDeallocType.tp_new = PyType_GenericNew; if (PyType_Ready(&_MyDeallocType) < 0) return; %}
... and that's it! then just use ARGOUTVIEWM_ARRAY1 instead of ARGOUTVIEW_ARRAY1 and python does the deallocation for you when the python array is destroyed (see the examples attached).
Everything compiles, but it would be nice to get rid of the warnings...
ezalloc_wrap.c:2763: warning: initialization from incompatible pointer type ezalloc_wrap.c: In function `_wrap_alloc_managed': ezalloc_wrap.c:2844: warning: assignment from incompatible pointer type writing build\temp.win32-2.5\Release\_ezalloc.def
Compilation: python setup_alloc.py build
Testing: The attached test_alloc.py does 2048 allocations of 1MB each for managed and unmanaged arrays. Output on my XP laptop with 1GB RAM as follows:
ARGOUTVIEWM_ARRAY1 (managed arrays) - 2048 allocations (1048576 bytes each) Done!
ARGOUTVIEW_ARRAY1 (unmanaged, leaking) - 2048 allocations (1048576 bytes each) Step 482 failed
TODO: Find a better name for the methods (not sure I like ARGOUTVIEWM_ARRAY), then do the missing fragments (2D arrays), clear the warnings and verify the method for a possible inclusion in the official numpy.i file.
Thank you for reading!
Regards, Egor
Egor Zindy wrote:
I just finished coding all the ARGOUTVIEWM fragments in numpy.i and wrote a wiki to explain what I did: http://code.google.com/p/ezwidgets/wiki/NumpyManagedMemory
thanks! good stuff.
It would be great if you could put that in the numpy (scipy?) wiki though, so more folks will find it.
-Chris
Christopher Barker wrote:
thanks! good stuff. It would be great if you could put that in the numpy (scipy?) wiki though, so more folks will find it.
-Chris
Hello Chris,
no problems, you are absolutely right, this is where the documents will have to eventually end up for maximum visibility. There is already a bit of numpy + SWIG in the cookbook, but that could well have been written before numpy.i
http://www.scipy.org/Cookbook/SWIG_and_NumPy
For added exposure, there is also the numpy.i document written by Bill Spotz. That could do with more examples (separate document maybe). The lack of examples is what prompted me to write my wiki in the first place!
I've also updated my other document with a more credible ARGOUTVIEW example. The part about numpy+SWIG+MinGW is now dwarfed by the body of numpy.i examples (not necessary a good thing).
Plus all the examples are ARRAY1, people have also asked for some ARRAY2 / ARRAY3 examples and FORTRAN arrays (which I don't know anything about I'm afraid).
Here are the two wikis so far: http://code.google.com/p/ezwidgets/wiki/NumpySWIGMinGW http://code.google.com/p/ezwidgets/wiki/NumpyManagedMemory
Still looking for a good name for my "argout arrays with managed deallocation"... After writing my ARGOUTVIEW example, I am not even sure my addition to numpy.i should be called a "view" anymore.
How does ARGOUTMAD_ARRAY sound? (for Managed Allocation / Deallocation) :-)
Regards, Egor
Hello list,
just a quick follow-up on the managed deallocation. This is what I've done this week-end:
In numpy.i, I have redefined the import_array() function to also take care of the managed memory initialisation (the _MyDeallocType.tp_new = PyType_GenericNew; statement). This means that in %init(), the only call is to import_array(). Basically, the same as with the "normal" numpy.i. Only difference in a swig file (.i) between "unmanaged" and "managed" memory allocation is the use of either the ARGOUTVIEW_ARRAY or ARGOUTVIEWM_ARRAY fragments. Everything else is hidden.
In numpy.i, this is what's now happening (my previous attempts were a bit clumsy):
%#undef import_array %#define import_array() {if (_import_array() < 0) {PyErr_Print(); PyErr_SetString(PyExc_ImportError, "numpy.core.multiarray failed to import"); return; }; _MyDeallocType.tp_new = PyType_GenericNew; if (PyType_Ready(&_MyDeallocType) < 0) {PyErr_Print(); PyErr_SetString(PyExc_ImportError, "Custom memory management failed to initialize (numpy.i)"); return; } }
%#undef import_array1 %#define import_array1(ret) {if (_import_array() < 0) {PyErr_Print(); PyErr_SetString(PyExc_ImportError, "numpy.core.multiarray failed to import"); return ret; }; _MyDeallocType.tp_new = PyType_GenericNew; if (PyType_Ready(&_MyDeallocType) < 0) {PyErr_Print(); PyErr_SetString(PyExc_ImportError, "Custom memory management failed to initialize (numpy.i)"); return ret; } }
%#undef import_array2 %#define import_array2(msg, ret) {if (_import_array() < 0) {PyErr_Print(); PyErr_SetString(PyExc_ImportError, msg); return ret; }; _MyDeallocType.tp_new = PyType_GenericNew; if (PyType_Ready(&_MyDeallocType) < 0) {PyErr_Print(); PyErr_SetString(PyExc_ImportError, msg); return ret; } }
My wiki (sorry, haven't moved it to the scipy cookbook yet) has all the details (the modified numpy.i, explanations, and some test code): http://code.google.com/p/ezwidgets/wiki/NumpyManagedMemory
Regards, Egor
Hello list,
I've moved my wiki to the scipy cookbook: http://www.scipy.org/Cookbook/SWIG_Memory_Deallocation
For the time being, the listed example files are still stored on the google code SVN, but these could easily be moved if necessary.
I've also just finished adding an ARGOUTVIEWM_ARRAY2 example. The example shows how to return a two-dimensional array from C which also benefits from the automatic memory deallocation.
A naive "crop" function is wrapped using SWIG/numpy.i and returns a slice of the input array. When used as
array_out = crop.crop(array_in, d1_0,d1_1, d2_0,d2_1)
it is equivalent to the native numpy slicing
array_out = array_in[d1_0:d1_1, d2_0:d2_1]
Hope this helps!
Regards, Egor
participants (2)
-
Christopher Barker -
Egor Zindy