swig numpy2carray converters
Hallo! Because I had some troubles in wrapping my C++ library in python/numpy, I did (another) numpy2carray.i file for SWIG. With that interface file it is possible to input/output arrays with or without copying data (I also included an example for an interface to fortran style arrays). I am still a numpy newbie, so this might not be perferct, but however, I finally found some time to put it online - maybe it is helpfull for other beginners: http://grh.mur.at/software/numpy2carray.html LG Georg
Georg Holzmann wrote:
Because I had some troubles in wrapping my C++ library in python/numpy, I did (another) numpy2carray.i file for SWIG.
How is this better/different than numpy.i in: numpy/doc/swig/numpy.i
With that interface file it is possible to input/output arrays with or without copying data
numpy.i support that too.
(I also included an example for an interface to fortran style arrays).
That, it doesn't have. Just trying to keep too much effort from being duplicated.... -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
Hallo!
How is this better/different than numpy.i in:
numpy/doc/swig/numpy.i
The problem I had with numpy.i: - it copies the arrays on output (Argout Arrays) which was not possible for my algorithms (I have some very big matrices) - it is not possible to 2D or 3D Argout Arrays (why?), in the docs: ---8<--- Note that we support DATA_TYPE* argout typemaps in 1D, but not 2D or 3D. This because of a quirk with the SWIG typemap syntax and cannot be avoided. Note that for these types of 1D typemaps, the python function will take a single argument representing DIM1. ---8<--- - I also needed an output of fortran style arrays
(I also included an example for an interface to fortran style arrays).
That, it doesn't have.
It has ;) ... look in numpy2carray.i, FARRAY2_OUT (line 175). But yes, sorry, I did no example in example.cpp ...
Just trying to keep too much effort from being duplicated....
Yes, of course. Actually I did not want to write that code ;). I was (and still am) a numpy beginner and needed to use my library in it - so all the people said it's so easy to do ... Then I tried boost.python and swig and I spent a lot of time trying to get that work, writing on maiiling lists ... - I did not think before that it would be that hard - and I don't understand why nobody else needed that before. Maybe this could also be integrated in numpy/doc/swig/numpy.i - but however, this is so much code and I didn't really understand how I should change that to my needs and also asked on the mailinglists ... OTOH the umfpack.i bindings in scipy where quite nice, much easier to read, but of course tuned to their task ... Therefore I made these wrappers, which should be easier to understand for beginners, how to write there own wrappers and tune them to their needs. LG Georg
Georg Holzmann wrote:
(I also included an example for an interface to fortran style arrays).
That, it doesn't have. It has ;) ... look in numpy2carray.i, FARRAY2_OUT (line 175). But yes, sorry, I did no example in example.cpp ...
I'm pretty sure Chris meant that numpy.i doesn't have it (and that consequently, your inclusion of it in your numpy2carray.i is quite welcome). :-)
DG
On Nov 20, 2007, at 1:12 AM, Georg Holzmann wrote:
The problem I had with numpy.i:
- it copies the arrays on output (Argout Arrays) which was not possible for my algorithms (I have some very big matrices)
Really? I worked pretty hard to avoid copies when they were not necessary. For the ARGOUT typemaps, I allocate an array of the requested size and then pass its data buffer to your function. If that is not what you want, then you should probably be using the INPLACE typemap. Do you have some different use case? If so, it would be far better for us (me) to add this to numpy.i, which covers more data types (especially dimension data types) and has a lot more error checking in it, rather than develop yet another numpy/swig interface file.
- it is not possible to 2D or 3D Argout Arrays (why?), in the docs: ---8<--- Note that we support DATA_TYPE* argout typemaps in 1D, but not 2D or 3D. This because of a quirk with the SWIG typemap syntax and cannot be avoided. Note that for these types of 1D typemaps, the python function will take a single argument representing DIM1. ---8<---
I don't see where your numpy2carray.i file handles arrays greater than 1D either. Doing a search on "argout", the corresponding typemaps all apply to ttype*.
- I also needed an output of fortran style arrays
This would be a good thing to add to numpy.i.
Yes, of course. Actually I did not want to write that code ;). I was (and still am) a numpy beginner and needed to use my library in it - so all the people said it's so easy to do ... Then I tried boost.python and swig and I spent a lot of time trying to get that work, writing on maiiling lists ... - I did not think before that it would be that hard - and I don't understand why nobody else needed that before.
I follow both the swig and numpy mail lists. I don't see any posts from you prior to 10/25/07. I probably just skimmed past your first posts because the subject referred to fortran, so I assumed you were dealing with fortran code. Sorry about that.
Maybe this could also be integrated in numpy/doc/swig/numpy.i - but however, this is so much code and I didn't really understand how I should change that to my needs and also asked on the mailinglists ... OTOH the umfpack.i bindings in scipy where quite nice, much easier to read, but of course tuned to their task ...
The numpy.i file does get to be pretty hard to follow, largely because it uses nested macros to handle the large combination of cases. It is a question of how to handle all of the possible use cases. Even so, it would appear that you may have came up with one that wasn't covered, so you see my problem.
Therefore I made these wrappers, which should be easier to understand for beginners, how to write there own wrappers and tune them to their needs.
In the end, though, I think it would be far better to add new capabilities to numpy.i rather than duplicate effort. This will never be as big as the Numeric/numarray split, but still.... Let me know what you are trying to do. If numpy.i can do it, I'll show you how. If not, I'll work to upgrade numpy.i. I suspect it can do everything except the fortran ordering. ** Bill Spotz ** ** Sandia National Laboratories Voice: (505)845-0170 ** ** P.O. Box 5800 Fax: (505)284-0154 ** ** Albuquerque, NM 87185-0370 Email: wfspotz@sandia.gov **
Hallo!
Really? I worked pretty hard to avoid copies when they were not necessary. For the ARGOUT typemaps, I allocate an array of the requested size and then pass its data buffer to your function. If
Yes but this means that you again allocate an array of the same size. E.g. in my algorithm I can have a very big internal matrix in C++ (say 700 MB - in fortran style). Now I want to have this matrix in numpy to plot some parts of it, get some data out of it ... whatever - if I again allocate an array of the same size, I am out of memory. Therefore I simply used the PyArray_FromDimsAndData() function to allocate the array. BTW: for me this also seems to be more the "numpy way", because usually numpy doesn't copy data. And if I want to copy this data I can simply call .copy() in python - however, there is of course the risk that this data is freed in C and you can have a crash ... so maybe this is no good idea for a general interface ...
that is not what you want, then you should probably be using the INPLACE typemap.
Yeah, but this results in the same as above ...
Do you have some different use case? If so, it would be far better for us (me) to add this to numpy.i, which covers more data types (especially dimension data types) and has a lot more error checking in it, rather than develop yet another numpy/swig interface file.
Yes I would very much appreciate that. My use case is (as described above) an ARGOUT 1D and/or 2D array in C and/or Fortran style storage without copying the data (and also not allocating it again).
I don't see where your numpy2carray.i file handles arrays greater than 1D either. Doing a search on "argout", the corresponding typemaps all apply to ttype*.
Hm ... maybe this is a misunderstanding ? - I mean with 2D output a two dimensional numpy array (simply a matrix). In numpy2carray.i the macros are called ARRAY2_OUT and FARRAY2_OUT.
I follow both the swig and numpy mail lists. I don't see any posts from you prior to 10/25/07. I probably just skimmed past your first posts because the subject referred to fortran, so I assumed you were dealing with fortran code. Sorry about that.
I did not write to the swig list - I think it was the [C++-sig], but however, thats no longer important ... ;)
In the end, though, I think it would be far better to add new capabilities to numpy.i rather than duplicate effort. This will
of course ...
Let me know what you are trying to do. If numpy.i can do it, I'll show you how. If not, I'll work to upgrade numpy.i. I suspect it can do everything except the fortran ordering.
Again, see above for my use case. But the fortran ordering should not be that hard (only setting the flags and strides right, as in FARRAY2_OUT in numpy2carray.i) - but of course someone has to do it ... ;) LG Georg
2007/11/20, Georg Holzmann
Hallo!
Really? I worked pretty hard to avoid copies when they were not necessary. For the ARGOUT typemaps, I allocate an array of the requested size and then pass its data buffer to your function. If
Yes but this means that you again allocate an array of the same size.
Well, this is logical as you want a new argument in output. E.g. in my algorithm I can have a very big internal matrix in C++ (say
700 MB - in fortran style). Now I want to have this matrix in numpy to plot some parts of it, get some data out of it ... whatever - if I again allocate an array of the same size, I am out of memory. Therefore I simply used the PyArray_FromDimsAndData() function to allocate the array.
This is why you use INPLACE typemaps that will NOT copy your data.
that is not what you want, then you should probably be using the
INPLACE typemap.
Yeah, but this results in the same as above ...
Are you sure ? Because the original object is not modified, so it is still the same data. If what you want is to provide a view from your C++ matrix, this is different. You must either : - propose the array interface - use a Python object inside your C++ matrix (this is to be done, I've a basic example in my blog) Matthieu -- French PhD student Website : http://miles.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher
Hallo!
E.g. in my algorithm I can have a very big internal matrix in C++ (say 700 MB - in fortran style). Now I want to have this matrix in numpy to plot some parts of it, get some data out of it ... whatever - if I again allocate an array of the same size, I am out of memory. Therefore I simply used the PyArray_FromDimsAndData() function to allocate the array.
This is why you use INPLACE typemaps that will NOT copy your data.
[...]
Are you sure ? Because the original object is not modified, so it is still the same data.
Hm ... lets consider the same example as before (one 700MB matrix in C++). If I want to get this data with an INPLACE typemap I again have to allocate an 700 MB array in python, then passing it to my C++ library which puts in the data of it - so in the end I have to use two times 700 MB matrices ? (or maybe I don't understand something ;) ?)
If what you want is to provide a view from your C++ matrix, this is different. You must either : - propose the array interface - use a Python object inside your C++ matrix (this is to be done, I've a basic example in my blog)
Yes, maybe thats what I need. Do you have a link to that blog ? LG GEorg
If what you want is to provide a view from your C++ matrix, this is different. You must either : - propose the array interface - use a Python object inside your C++ matrix (this is to be done, I've a basic example in my blog)
Of course : http://matt.eifelle.com/item/5 It's a basic version of the wrapper I use in my lab (pay attention to the constructor for instance), I hope you will be able to do something alike for your library. If it's not the case, you will have to fall back to the numpy way : allocating a new array, giving a pointer to the array, using it and stop using it after the function call is finished (with the wrapper I propose, you have the array for the life-time of your matrix instead). Matthieu -- French PhD student Website : http://miles.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher
Hallo!
Of course : http://matt.eifelle.com/item/5 It's a basic version of the wrapper I use in my lab (pay attention to the constructor for instance), I hope you will be able to do something
Thanks ! But this assumes that the data in my C++ library is stored in a PyArrayObject ? This is of course not possible in my case - I need this library also in other situations, where I don't want to depend on python/numpy ... LG Georg
Of course : http://matt.eifelle.com/item/5 It's a basic version of the wrapper I use in my lab (pay attention to the constructor for instance), I hope you will be able to do something
Thanks ! But this assumes that the data in my C++ library is stored in a PyArrayObject ?
Yes, but if your data storage is decoupled from the data use, you can replace the data storage by a Python one This is of course not possible in my case - I need this library also in
other situations, where I don't want to depend on python/numpy ...
In this case, you use the default storage policy. If you only call functions and do not use classes (construct a class in Python and then call a method), this is an overkill. Matthieu -- French PhD student Website : http://miles.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher
On Nov 20, 2007, at 7:24 AM, Georg Holzmann wrote:
Yes but this means that you again allocate an array of the same size. E.g. in my algorithm I can have a very big internal matrix in C++
OK, so the key here is the *internal* matrix. I think you need to provide a way to extract that matrix from the C++ application as a numpy array. Then you can provide it to your function/method as an INPLACE array. No new memory will be allocated.
that is not what you want, then you should probably be using the INPLACE typemap.
Yeah, but this results in the same as above ...
The INPLACE typemaps force you to provide the allocated memory. If you do it by accessing whatever your C++ app has already allocated, you should be fine.
Yes I would very much appreciate that. My use case is (as described above) an ARGOUT 1D and/or 2D array in C and/or Fortran style storage without copying the data (and also not allocating it again).
So assuming the INPLACE solution is sufficient for you, then the only new feature is fortran ordering.
Hm ... maybe this is a misunderstanding ? - I mean with 2D output a two dimensional numpy array (simply a matrix). In numpy2carray.i the macros are called ARRAY2_OUT and FARRAY2_OUT.
I guess my assumption was that in the general case you would want to retain the dimensions as input arguments, since it is logical for ARGOUT typemaps to allocate new memory. Since swig typemaps only allow numinputs=0 or numinputs=1, this was problematic. I guess the user could provide a sequence (tuple or list) as the single argument . . . don't know why I didn't think of that before. But I still think you want an INPLACE typemap.
Again, see above for my use case. But the fortran ordering should not be that hard (only setting the flags and strides right, as in FARRAY2_OUT in numpy2carray.i) - but of course someone has to do it ... ;)
Yes, it shouldn't be too hard. And I like your FARRAY notation for the interface. So your experience is that the flags and strides are all that have to be set differently? ** Bill Spotz ** ** Sandia National Laboratories Voice: (505)845-0170 ** ** P.O. Box 5800 Fax: (505)284-0154 ** ** Albuquerque, NM 87185-0370 Email: wfspotz@sandia.gov **
Bill Spotz wrote:
Again, see above for my use case. But the fortran ordering should not be that hard (only setting the flags and strides right, as in FARRAY2_OUT in numpy2carray.i) - but of course someone has to do it ... ;)
Yes, it shouldn't be too hard. And I like your FARRAY notation for the interface. So your experience is that the flags and strides are all that have to be set differently?
In my experience, so does the "zero-offset," (or is this controlled by flags in this context?) DG
** Bill Spotz ** ** Sandia National Laboratories Voice: (505)845-0170 ** ** P.O. Box 5800 Fax: (505)284-0154 ** ** Albuquerque, NM 87185-0370 Email: wfspotz@sandia.gov **
_______________________________________________ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Hallo!
OK, so the key here is the *internal* matrix. I think you need to provide a way to extract that matrix from the C++ application as a numpy array. Then you can provide it to your function/method as an INPLACE array. No new memory will be allocated. [...] The INPLACE typemaps force you to provide the allocated memory. If you do it by accessing whatever your C++ app has already allocated, you should be fine.
Hm ... I still don't understand that ... With INPLACE I have to allocate the numpy array before, then pass it to my getMyMatrix(my_inplace_numarray) method, then copy the data in that method to the my_inplace_numarray - right ? But I am still not convinced by the INPLACE method - the more natural way for me is to simply return the matrix ... Maybe the most usefull way would be to add also OUT_WITHOUTCOPY macros to numpy.i, where the data is allocated with PyArray_FromDimsAndData() ? (as a "long term" goal ...)
Hm ... maybe this is a misunderstanding ? - I mean with 2D output a two dimensional numpy array (simply a matrix). In numpy2carray.i the macros are called ARRAY2_OUT and FARRAY2_OUT.
I guess my assumption was that in the general case you would want to retain the dimensions as input arguments, since it is logical for ARGOUT typemaps to allocate new memory. Since swig typemaps only allow numinputs=0 or numinputs=1, this was problematic. I guess the user could provide a sequence (tuple or list) as the single argument . . . don't know why I didn't think of that before.
Again I do not see the problem - see e.g. ARRAY2_OUT_COPY in numpy2carray.i, shouldn't this be the same ?
Yes, it shouldn't be too hard. And I like your FARRAY notation for the interface. So your experience is that the flags and strides are all that have to be set differently?
Yes, so far I had no crash ... ;) The only thing to do is the following: -------8<----------- PyArrayObject *tmp = (PyArrayObject*)obj; tmp->flags = NPY_FARRAY; int s = tmp->strides[1]; tmp->strides[0] = s; tmp->strides[1] = s * dim0[0]; -------8<----------- LG Georg
Here is what I am proposing you do: in your interface file, add something like PyObject * getMatrix() { npy_intp dims[2] = { /* Obtain the dimensions to your internal matrix */ }; double * data = /* Obtain the pointer to you internal matrix */; return PyArray_SimpleNewFromData(2, dims, NPY_DOUBLE, (void*) data); } For your function, use the INPLACE_ARRAY2 typemap: %apply (double* INPLACE_ARRAY2, int DIM1, int DIM2) {(double* matrix, int rows, int cols)}; void myFunction(double* matrix, int rows, int cols, double parameter, ...); And then in python you can do this: m = getMatrix() myFunction(m, 3.14, ...) However myFunction() alters your internal matrix, it will be reflected in the array m. What I don't understand is if double* matrix is *internal*, why is it in the argument list? Does myFunction() actually allocate the matrix? If so, then getMatrix() won't be able to get access to the matrix, but myFunction() is really a factory function or constructor. If this is true, then writing a simple function like getMatrix() as a SUBSTITUTE wrapper for myFunction() (i.e., that calls myFunction()) is MUCH, MUCH simpler than trying to maintain a general typemap that does the same thing. If my assumptions are wrong then you will need to explain to me exactly what myFunction() expects of its arguments, and why double* matrix is in the argument list at all. On Nov 20, 2007, at 1:38 PM, Georg Holzmann wrote:
Hallo!
OK, so the key here is the *internal* matrix. I think you need to provide a way to extract that matrix from the C++ application as a numpy array. Then you can provide it to your function/method as an INPLACE array. No new memory will be allocated. [...] The INPLACE typemaps force you to provide the allocated memory. If you do it by accessing whatever your C++ app has already allocated, you should be fine.
Hm ... I still don't understand that ... With INPLACE I have to allocate the numpy array before, then pass it to my getMyMatrix(my_inplace_numarray) method, then copy the data in that method to the my_inplace_numarray - right ?
No. INPLACE means "in place". The typemap expects that the array has already been allocated -- either you allocated something new or you obtained the array from existing data -- it doesn't care. It passes the pointer to the buffer to the function, which can alter the data IN PLACE, and you will see these changes to your array after the wrapper returns control to python.
But I am still not convinced by the INPLACE method - the more natural way for me is to simply return the matrix ...
This is only natural or makes sense if the function allocates the data for the matrix. Does it? If so, then multiple calls to it ensures duplicate allocations, which you indicate is not possible. If not, then the function expects the data to be allocated elsewhere.
Maybe the most usefull way would be to add also OUT_WITHOUTCOPY macros to numpy.i, where the data is allocated with PyArray_FromDimsAndData() ? (as a "long term" goal ...)
Again I do not see the problem - see e.g. ARRAY2_OUT_COPY in numpy2carray.i, shouldn't this be the same ?
I do not understand the use case for which that typemap works. You create "ttype t1", then pass its address to the function as the pointer to the data. This has to be useless to the function. After the function returns, you access this pointer as if it points to meaningful data. How is this possible? The address of t1 isn't going to change, and only a single element is allocated. ** Bill Spotz ** ** Sandia National Laboratories Voice: (505)845-0170 ** ** P.O. Box 5800 Fax: (505)284-0154 ** ** Albuquerque, NM 87185-0370 Email: wfspotz@sandia.gov **
I'm a bit confused too. What would be great is a simple trimmed down example -- a small-as-you-can-make-it class with a method that shows what you need, perhaps with a little C++ sample that uses it. Then we can see how best to wrap it for python. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
Christopher Barker wrote:
What would be great is a simple trimmed down example --
.. and then we'd have an example to put in the numpy.i docs and examples, too. By the way Bill, I haven't forgotten the examples I said I'd add to the docs. I've been distracted away from my SWIG work lately, but really hope to get back to it. I also had trouble being able to commit to SVN, but I'll address that when I have something worth committing. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
Chris, just to be clear, this is addressed to the OP, correct? DG Christopher Barker wrote:
I'm a bit confused too.
What would be great is a simple trimmed down example -- a small-as-you-can-make-it class with a method that shows what you need, perhaps with a little C++ sample that uses it. Then we can see how best to wrap it for python.
-Chris
David.Goldsmith wrote:
Chris, just to be clear, this is addressed to the OP, correct?
yes, but if anyone else want to come up with one, that would work too. -Chris
What would be great is a simple trimmed down example -- a small-as-you-can-make-it class with a method that shows what you need, perhaps with a little C++ sample that uses it. Then we can see how best to wrap it for python.
-- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
Hallo! As chris said, I need to make an example: http://grh.mur.at/software/numpy2carray.tar.gz I added the following class-example: class_example.h: the C++ code class_example.i: the SWIG interface file class_example_usage.py: example usage in python And some comments: Bill Spotz schrieb:
Here is what I am proposing you do: in your interface file, add something like
PyObject * getMatrix() { npy_intp dims[2] = { /* Obtain the dimensions to your internal matrix */ }; double * data = /* Obtain the pointer to you internal matrix */; return PyArray_SimpleNewFromData(2, dims, NPY_DOUBLE, (void*)data); }
For your function, use the INPLACE_ARRAY2 typemap:
%apply (double* INPLACE_ARRAY2, int DIM1, int DIM2) {(double* matrix, int rows, int cols)}; void myFunction(double* matrix, int rows, int cols, double parameter, ...);
And then in python you can do this:
m = getMatrix() myFunction(m, 3.14, ...)
I see ... but what I mean with output of a matrix without copy is actually only your getMatrix() funtion I guess - this is basically what getBigData() does in class_example.h together with class_example.i. BTW: what is the difference between PyArray_SimpleNewFromData() and PyArray_FromDimsAndData() ? (I don't have this book ...)
Again I do not see the problem - see e.g. ARRAY2_OUT_COPY in numpy2carray.i, shouldn't this be the same ?
I do not understand the use case for which that typemap works. You create "ttype t1", then pass its address to the function as the pointer to the data. This has to be useless to the function. After the function returns, you access this pointer as if it points to meaningful data. How is this possible? The address of t1 isn't going to change, and only a single element is allocated.
See also the class_example_usage.py - there I used the ARRAY2_OUT_COPY for the getBigDataCopy() method. LG Georg
Georg Holzmann wrote:
As chris said, I need to make an example: http://grh.mur.at/software/numpy2carray.tar.gz
Ah, I see now: /// @return internal big data without copying void getBigData(double **mtx, int *rows, int *cols) { *rows = drows; *cols = dcols; *mtx = very_big_data; } This is the one we've been talking about, correct? So you need to pass a pointer to a pointer in, and it gets set to the existing pointer -- thus no new data allocation. It's going to take Bill to figure out how to extend numpy.i to cover this case, but imagine it will be pretty straightforward. Thanks for the examples. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
On Nov 21, 2007, at 10:07 AM, Georg Holzmann wrote:
BTW: what is the difference between PyArray_SimpleNewFromData() and PyArray_FromDimsAndData() ? (I don't have this book ...)
PyArray_SimpleNewFromData() is the new version and PyArray_FromDimsAndData() is the old version :-) Travis may be able to better explain. ** Bill Spotz ** ** Sandia National Laboratories Voice: (505)845-0170 ** ** P.O. Box 5800 Fax: (505)284-0154 ** ** Albuquerque, NM 87185-0370 Email: wfspotz@sandia.gov **
OK, I'm going to try to get to this soon, but I want to discuss the interface some before committing to development. First, my plan is to add to numpy.i, typemaps for signatures like the following: %typemap(argout) (double** ARGOUT_ARRAY1, int* DIM1) It is important to note that even though the same argument *names* are used, this is a different typemap signature than %typemap(argout) (double* ARGOUT_ARRAY1, int DIM1) and thus can have (logically) different implementations. For the latter, the typemap allocates a new numpy array and passes the buffer pointer and dimension in; for the former, the buffer pointer and dimension will be obtained from the wrapped function and used to build a new numpy array. As for having a COPY version of the first typemap signature (in addition to the non-copy, or "view" version), I currently do not plan to do this. Here's why: the only case I can conceive of for needing it is when the class provides a "view" method but not a "copy" method. Since there is no copy method to apply a typemap to, the swig user will have to add a new one with %extend, in which case the swig user can provide the copy logic. The example Georg gives in his link below is not a counter-example to this. He provides two methods, void getBigData(double **mtx, int *rows, int *cols) void getBigDataCopy(double **cpmtx, int *cprows, int *cpcols) but in addition to the same argument types, they have the exact same implementation. The only thing that is different is the names of the arguments, which is clearly so that we can apply different swig typemaps to the two methods. I submit that you will not find a class written by someone else that will be designed this way. You will only find it if you have control over the class, in which case you should put the copy logic inside the second method. (It is important to understand that if you have a class with functioning view and copy methods, then the view version of the typemap will work for both.) Maybe I'm wrong. But that is why I want to discuss this before committing to code. On Nov 21, 2007, at 10:07 AM, Georg Holzmann wrote:
Hallo!
As chris said, I need to make an example: http://grh.mur.at/software/numpy2carray.tar.gz
I added the following class-example: class_example.h: the C++ code class_example.i: the SWIG interface file class_example_usage.py: example usage in python
And some comments:
Bill Spotz schrieb:
Here is what I am proposing you do: in your interface file, add something like PyObject * getMatrix() { npy_intp dims[2] = { /* Obtain the dimensions to your internal matrix */ }; double * data = /* Obtain the pointer to you internal matrix */; return PyArray_SimpleNewFromData(2, dims, NPY_DOUBLE, (void*)data); } For your function, use the INPLACE_ARRAY2 typemap: %apply (double* INPLACE_ARRAY2, int DIM1, int DIM2) {(double* matrix, int rows, int cols)}; void myFunction(double* matrix, int rows, int cols, double parameter, ...); And then in python you can do this: m = getMatrix() myFunction(m, 3.14, ...)
I see ... but what I mean with output of a matrix without copy is actually only your getMatrix() funtion I guess - this is basically what getBigData() does in class_example.h together with class_example.i.
BTW: what is the difference between PyArray_SimpleNewFromData() and PyArray_FromDimsAndData() ? (I don't have this book ...)
Again I do not see the problem - see e.g. ARRAY2_OUT_COPY in numpy2carray.i, shouldn't this be the same ? I do not understand the use case for which that typemap works. You create "ttype t1", then pass its address to the function as the pointer to the data. This has to be useless to the function. After the function returns, you access this pointer as if it points to meaningful data. How is this possible? The address of t1 isn't going to change, and only a single element is allocated.
See also the class_example_usage.py - there I used the ARRAY2_OUT_COPY for the getBigDataCopy() method.
LG Georg
** Bill Spotz ** ** Sandia National Laboratories Voice: (505)845-0170 ** ** P.O. Box 5800 Fax: (505)284-0154 ** ** Albuquerque, NM 87185-0370 Email: wfspotz@sandia.gov **
Hallo!
First, my plan is to add to numpy.i, typemaps for signatures like the following:
%typemap(argout) (double** ARGOUT_ARRAY1, int* DIM1)
It is important to note that even though the same argument *names* are used, this is a different typemap signature than
%typemap(argout) (double* ARGOUT_ARRAY1, int DIM1)
and thus can have (logically) different implementations. For the latter, the typemap allocates a new numpy array and passes the buffer pointer and dimension in; for the former, the buffer pointer and dimension will be obtained from the wrapped function and used to build a new numpy array.
Hm ... maybe it is more clear for users then, if the new typemap has a different name ? In example ARRAY_VIEW1, or something else with "VIEW" (although it's not really a view ?) ?
As for having a COPY version of the first typemap signature (in addition to the non-copy, or "view" version), I currently do not plan to do [...] The example Georg gives in his link below is not a counter-example to this. He provides two methods,
void getBigData(double **mtx, int *rows, int *cols) void getBigDataCopy(double **cpmtx, int *cprows, int *cpcols)
but in addition to the same argument types, they have the exact same implementation. The only thing that is different is the names of the arguments, which is clearly so that we can apply different swig typemaps to the two methods. I submit that you will not find a class written by someone else that will be designed this way. You will only find it if
Yes this copy method was only for demonstration - I also did not use this method for my "real" code. One can simply call .copy() in python ... LG Georg
Bill Spotz wrote:
First, my plan is to add to numpy.i, typemaps for signatures like the following:
%typemap(argout) (double** ARGOUT_ARRAY1, int* DIM1)
It is important to note that even though the same argument *names* are used, this is a different typemap signature than
%typemap(argout) (double* ARGOUT_ARRAY1, int DIM1)
and thus can have (logically) different implementations.
maybe it's a good idea to give the arguments different names, though, just for clarity.
As for having a COPY version of the first typemap signature (in addition to the non-copy, or "view" version), I currently do not plan to do this.
I think you've got it right. Thanks for all this! -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
I have just committed the latest version of numpy.i (a swig interface file for bridging between C arrays and numerical python) to the numpy svn repository. There are three relatively new features that are now supported: * It is now possible to wrap functions that expect integer arguments and have swig generate wrappers that allow you to provide numpy array scalars. * A new ARGOUTVIEW suite of typemaps is provided that allows your wrapped function to provide a pointer to internal data and that returns a numpy array encapsulating it. This is potentially dangerous, but necessary in some situations for very large data buffers. * New typemaps are provided that correctly handle FORTRAN ordered 2D and 3D arrays. Tests for the ARGOUTVIEW and FORTRAN ordered arrays have also been added, and the documentation (doc/numpy_swig.*) has been updated to reflect all of these changes. On Nov 26, 2007, at 3:52 PM, Christopher Barker wrote:
Bill Spotz wrote:
First, my plan is to add to numpy.i, typemaps for signatures like the following:
%typemap(argout) (double** ARGOUT_ARRAY1, int* DIM1)
It is important to note that even though the same argument *names* are used, this is a different typemap signature than
%typemap(argout) (double* ARGOUT_ARRAY1, int DIM1)
and thus can have (logically) different implementations.
maybe it's a good idea to give the arguments different names, though, just for clarity.
As for having a COPY version of the first typemap signature (in addition to the non-copy, or "view" version), I currently do not plan to do this.
I think you've got it right.
Thanks for all this!
-Chris
-- Christopher Barker, Ph.D. Oceanographer
Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception
Chris.Barker@noaa.gov _______________________________________________ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
** Bill Spotz ** ** Sandia National Laboratories Voice: (505)845-0170 ** ** P.O. Box 5800 Fax: (505)284-0154 ** ** Albuquerque, NM 87185-0370 Email: wfspotz@sandia.gov **
On Nov 30, 2007 5:16 AM, Bill Spotz
I have just committed the latest version of numpy.i (a swig interface file for bridging between C arrays and numerical python) to the numpy svn repository. There are three relatively new features that are now supported:
* It is now possible to wrap functions that expect integer arguments and have swig generate wrappers that allow you to provide numpy array scalars.
* A new ARGOUTVIEW suite of typemaps is provided that allows your wrapped function to provide a pointer to internal data and that returns a numpy array encapsulating it. This is potentially dangerous, but necessary in some situations for very large data buffers.
Hi, This sounds great !! Could you elaborate on what you mean by "potentially dangerous" !? Is it important *how* the internal data memory was allocated ? Using "new" or using "malloc" ? This would require (following the standard) that the correct, i.e. corresponding, delete[] or free() [respectively], de-allocation function is called. When is the memory being freed ? Is (or can !) python taking ownership of the memory and calls the correct "free"/"delete[]" function ? Thanks, Sebastian Haase
The numpy array is created using PyArray_SimpleNewFromData(). From the Guide to NumPy, "[y]ou should ensure that the provided memory is not freed while the returned array is in existence." That is, numpy does not try to deallocate the memory when the ndarray object is destroyed, but it also *assumes* that you do not deallocate it yourself. Since numpy.i is dealing with essentially non-reference- counted pointers to raw buffers, I have no way to guarantee this. I think the typical use case will be that you wrap a class that allocates a chunk of memory (that it deallocates when it gets destroyed). It gives you a view to that data, which we then encapsulate with a numpy array. If you create such an object, extract such an array, and then destroy the object but not the array, then the numpy array will point to garbage, possibly resulting in a segmentation fault when you try to access it. The only answer I know of is to not destroy the object before the array. There are situations where you just can't get around it (based on how the C++ class was designed). But if you can get around it, you should. On Nov 30, 2007, at 3:55 AM, Sebastian Haase wrote:
Could you elaborate on what you mean by "potentially dangerous" !? Is it important *how* the internal data memory was allocated ? Using "new" or using "malloc" ? This would require (following the standard) that the correct, i.e. corresponding, delete[] or free() [respectively], de-allocation function is called.
When is the memory being freed ? Is (or can !) python taking ownership of the memory and calls the correct "free"/"delete[]" function ?
** Bill Spotz ** ** Sandia National Laboratories Voice: (505)845-0170 ** ** P.O. Box 5800 Fax: (505)284-0154 ** ** Albuquerque, NM 87185-0370 Email: wfspotz@sandia.gov **
Hallo!
* A new ARGOUTVIEW suite of typemaps is provided that allows your wrapped function to provide a pointer to internal data and that returns a numpy array encapsulating it.
Thanks for integrating it !
* New typemaps are provided that correctly handle FORTRAN ordered 2D and 3D arrays.
I have some problem with your FORTRAN implementation. The problem is how you set the flags in numpy.i "int require_fortran(PyArrayObject* ary)" (~ line 402). You do it like this: ary->flags = ary->flags | NPY_F_CONTIGUOUS; which does not work (at least on my computer) - I still get usual C-ordered arrays returned. However, it does work if you set the flags like this: ary->flags = NPY_FARRAY;
Tests for the ARGOUTVIEW and FORTRAN ordered arrays have also been added, and the documentation (doc/numpy_swig.*) has been updated to reflect all of these changes.
A small typo: in the docs you also write about 1D FORTRAN ARGOUTVIEW wrappers: ( DATA_TYPE** ARGOUTVIEW_FARRAY1, DIM_TYPE* DIM1 ) ( DIM_TYPE* DIM1, DATA_TYPE** ARGOUTVIEW_FARRAY1 ) which of course do not exist. LG Georg
These corrections have been committed. Thanks. On Dec 1, 2007, at 9:21 AM, Georg Holzmann wrote:
* A new ARGOUTVIEW suite of typemaps is provided that allows your wrapped function to provide a pointer to internal data and that returns a numpy array encapsulating it.
Thanks for integrating it !
* New typemaps are provided that correctly handle FORTRAN ordered 2D and 3D arrays.
I have some problem with your FORTRAN implementation. The problem is how you set the flags in numpy.i "int require_fortran(PyArrayObject* ary)" (~ line 402).
You do it like this: ary->flags = ary->flags | NPY_F_CONTIGUOUS; which does not work (at least on my computer) - I still get usual C-ordered arrays returned.
However, it does work if you set the flags like this: ary->flags = NPY_FARRAY;
Tests for the ARGOUTVIEW and FORTRAN ordered arrays have also been added, and the documentation (doc/numpy_swig.*) has been updated to reflect all of these changes.
A small typo: in the docs you also write about 1D FORTRAN ARGOUTVIEW wrappers:
( DATA_TYPE** ARGOUTVIEW_FARRAY1, DIM_TYPE* DIM1 ) ( DIM_TYPE* DIM1, DATA_TYPE** ARGOUTVIEW_FARRAY1 )
which of course do not exist.
** Bill Spotz ** ** Sandia National Laboratories Voice: (505)845-0170 ** ** P.O. Box 5800 Fax: (505)284-0154 ** ** Albuquerque, NM 87185-0370 Email: wfspotz@sandia.gov **
Christopher Barker wrote:
Georg Holzmann wrote:
Because I had some troubles in wrapping my C++ library in python/numpy, I did (another) numpy2carray.i file for SWIG.
How is this better/different than numpy.i in:
numpy/doc/swig/numpy.i
With that interface file it is possible to input/output arrays with or without copying data
numpy.i support that too.
(I also included an example for an interface to fortran style arrays).
That, it doesn't have.
Just trying to keep too much effort from being duplicated....
-Chris
Is there any doc on numpy.i usage?
participants (7)
-
Bill Spotz -
Christopher Barker -
David.Goldsmith -
Georg Holzmann -
Matthieu Brucher -
Neal Becker -
Sebastian Haase