![](https://secure.gravatar.com/avatar/b3308afd77cbb561a4c676aa091e55fc.jpg?s=120&d=mm&r=g)
Hi, I'm considering the use of the Blitz++ C++ library (http://www.oonumerics.org/blitz/) along with numarray/Numeric. I want to pass an array down to C++ and then manipulate it using Blitz++. My current thinking is to use the Numeric interface to pass down the array, and then copy it to a blitz++ array object. This seems straightforward except I am a little concerned about those reference counting issues, which I don't properly understand (I'm a Python beginner). I do get the impression that it is mainly an issue if I am working with Python objects in C/C++ code, though, and I am mostly going to try to avoid doing so by declaring C++ objects within the C++ code and working with those as much as possible. Other possibilities include somehow using the Boost.Python package, though I am not sure how. One advantage of this is that it apparently takes care of the reference counting issue. However, I don't want to learn yet another API. Also, weave.blitz() from SciPy looks relevant/useful, but this automatically generates its own C++ code, and I'd prefer to write my own. Can anyone suggested a good basic tutorial which discusses the issues of reference counting in a C/C++ API context? I have tried reading the official documentation and will continue to do so, but something gentler would be useful. Comments, suggestions gratefully appreciated. Please cc me on any reply to faheem@email.unc.edu. Thanks in advance. Faheem.
![](https://secure.gravatar.com/avatar/5a7d8a4d756bb1f1b2ea729a7e5dcbce.jpg?s=120&d=mm&r=g)
Faheem Mitha wrote:
[snip]
The following should be enough to get you started: #include "Python.h" #include "Numeric/arrayobject.h" #include "blitz/array.h" using namespace std; using namespace blitz; // Convert a Numpy array to a blitz one, using the original's data (no copy) template<class T, int N> static Array<T,N> py_to_blitz(PyArrayObject* arr_obj) { int T_size = sizeof(T); TinyVector<int,N> shape(0); TinyVector<int,N> strides(0); int *arr_dimensions = arr_obj->dimensions; int *arr_strides = arr_obj->strides; for (int i=0;i<N;++i) { shape[i] = arr_dimensions[i]; strides[i] = arr_strides[i]/T_size; } return Array<T,N>((T*) arr_obj->data,shape,strides,neverDeleteData); } This is what I use for exactly the problem you are describing, and this code was pretty much lifted, with minor changes, from weave's auto-generated C++. What I do, to sidestep the memory management problems, is let python allocate all my Numeric arrays (using zeros() if I have no data). I then use those inside my C++ code as blitz++ arrays via the above snippet. Any changes made by the C++ code are automatically reflected in the Numeric arary, since the blitz object is using the Numeric data area. I hope this helps. Let me know if you need more help, I can mail you complete example code. Cheers, f
![](https://secure.gravatar.com/avatar/b3308afd77cbb561a4c676aa091e55fc.jpg?s=120&d=mm&r=g)
On Mon, 29 Mar 2004, Fernando Perez wrote:
Thanks. That is very helpful. I've incorporated this into my code, and managed to get a simple example with boost working (extracting an element of an array). See below. Do you have a similar function which converts a blitz array to a numarray/Numeric one? If not, I can easily concoct one for myself. I now need to try to figure out what this reference counting stuff is about. I read the official Python docs but am still currently very confused. Faheem. *************************************************************************** #include "Python.h" #include "Numeric/arrayobject.h" #include <blitz/array.h> using namespace std; using namespace blitz; template<class T, int N> static Array<T,N> py_to_blitz(PyArrayObject* arr_obj); static PyObject * arrmod_elem(PyObject *self, PyObject *args); static PyMethodDef arrmod_methods[] = { {"elem", (PyCFunction)arrmod_elem, METH_VARARGS, "Returns the trace of a two-dimensional array.\n"}, {NULL, NULL, 0, NULL} }; PyMODINIT_FUNC initarrmod(void) { (void) Py_InitModule3("arrmod", arrmod_methods, "Returns the Trace of a two-dimensional array.\n"); import_array(); } static PyObject * arrmod_elem(PyObject *self, PyObject *args) { PyObject *input, *result; PyArrayObject *array; double el; int i, j; if (!PyArg_ParseTuple(args, "Oii", &input, &i, &j)) return NULL; array = (PyArrayObject *) PyArray_ContiguousFromObject(input, PyArray_DOUBLE, 2, 2); if (array == NULL) return NULL; Array<double,2> arr = py_to_blitz<double,2>(array); el = arr(i,j); result = Py_BuildValue("d",el); return result; } // Convert a Numpy array to a blitz one, using the original's data (no // copy) template<class T, int N> static Array<T,N> py_to_blitz(PyArrayObject* arr_obj) { int T_size = sizeof(T); TinyVector<int,N> shape(0); TinyVector<int,N> strides(0); int *arr_dimensions = arr_obj->dimensions; int *arr_strides = arr_obj->strides; for (int i=0;i<N;++i) { shape[i] = arr_dimensions[i]; strides[i] = arr_strides[i]/T_size; } return Array<T,N>((T*) arr_obj->data,shape,strides,neverDeleteData); }
![](https://secure.gravatar.com/avatar/5a7d8a4d756bb1f1b2ea729a7e5dcbce.jpg?s=120&d=mm&r=g)
Glad it helped.
Do you have a similar function which converts a blitz array to a numarray/Numeric one? If not, I can easily concoct one for myself.
No, I don't have one. But it's pretty straightforward to write one, the Numeric C API provides functions to create arrays from existing data. You'll be surprised how conceptually similar Blitz and Numeric turn out to be (which is a good thing).
As I said, so far I've been able to bypass this issue by creating my arrays in python and 'filling' them in blitz. But this is not always a realistic possibility, so you may indeed have to learn the reference counting management details. Best, f
![](https://secure.gravatar.com/avatar/c7976f03fcae7e1199d28d1c20e34647.jpg?s=120&d=mm&r=g)
Todd pointed out to me that he's gotten some complaints about the difference in how numarray nonzero behavior works. In version 0.9 the use of an array for a truth value was changed from a deprecation warning to an exception. We deprecated it because the meaning of an array as a truth value was ambiguous (or potentially confusing at best) and we felt it was better to force users to be explicit about what they wanted. Nevertheless, Numeric does allow that and removing the capability to use it in that context breaks some code. We can give some thought to reversing that decision, but since we only hear from those that don't like the new behavior, we'd like to get some sense if there are many that do like the new behavior. To clarify, what happens when: x = zeros(2) if x: print "true!" I see 3 possibilities: 1) Keep it the way it is, no use of an array is permitted; an exception is raised. 2) Print a warning message when it is used that way, and always keep it at that, i.e., warning message only. 3) Replicate Numeric behavior. Personally, I prefer 1) since I think there is too much confusion between whether false means an empty (len=0) array (like lists), or an array that contains all zeros, and that it is worth breaking backward compatibility on this (functions should be used to distinguish which meaning is desired). Perry [Option 2 on the previous question wins hands down]
![](https://secure.gravatar.com/avatar/5a7d8a4d756bb1f1b2ea729a7e5dcbce.jpg?s=120&d=mm&r=g)
Perry Greenfield wrote:
I tend to prefer keeping compatibility with Python list behavior. I think that in general, numarray arrays should behave different from lists only when truly justified (for example, at least in numeric all ufuncs can be applied to a list of floats as much as to an array, which is nice). Since anyone who knows python already knows that lists are true based on whether they contain _anything_: In [9]: a=[] In [10]: b=[0] In [11]: if a: ....: print 1 ....: In [12]: if b: ....: print 1 ....: 1 then I think it's a fair, clean extension to make NumArray behave similarly. I think a good guiding principle is to keep the number of special cases for arrays at a strict minimum (there are obviously cases where it does make sense to have special cases). Besides, this is an O(1) check, so it doesn't have the dangers of checking for all elements being zero. Such a potentially expensive operation should _definitely_ happen only if explicitly requested, IMHO. Regards, f
![](https://secure.gravatar.com/avatar/911fc13a8e992ab39b62f91d6eba5f54.jpg?s=120&d=mm&r=g)
* Fernando Perez [2004-04-02 00:51]:
How often does it happen that you have an empty array, which you need to test for? Certainly not as frequently as in case of a list. I think that if
then I would follow that
should also _both_ evaluate as "false".
But if the behaviour is unambiguously defined, I see no problem here, because I can as well check the length of a list using len(). BTW, How about >>> a = [[]] >>> if a: ... ? Cheers, Joachim
![](https://secure.gravatar.com/avatar/5a7d8a4d756bb1f1b2ea729a7e5dcbce.jpg?s=120&d=mm&r=g)
Joachim Saul wrote:
Well, just these days I'm working with code where I'm precisely using that kind of check, relying on Numeric's behavior. But I know that with potentially nested structures, it's probably best to be explicit and do all checks via things like: if len(a) and if a==0 etc. So I could also be +1 on Perry's wish for turning it into an extension, at the expense of my current code breaking. Cheers, f.
![](https://secure.gravatar.com/avatar/911fc13a8e992ab39b62f91d6eba5f54.jpg?s=120&d=mm&r=g)
* Fernando Perez [2004-04-02 02:15]:
But what would a==0 return? A boolean? An array of booleans? An array if integers? The way it is evaluated now in Numeric is pretty clear, though there could be other evaluations. For instance, if a==1: might be understood as "if a is everywhere one" or "if the length of vector a is one", etc. The current meaning is "if a is anywhere one", which may not be intuitive under all circumstances, but as long as it is the well defined behaviour, I simply don't see the need to change or forbid that. Nested structures are an important argument against keeping compatibility with Python list behaviour. For instance >>> if []: print "True" >>> if [[]]: print "True" True Which perfectly makes sense in Python, but an equivalent behaviour of a numeric array would be complete humbug. Thus, I don't think that it would make any sense to emulate Python list behaviour, rather I would support keeping the *existing* Numeric behaviour. Cheers, Joachim
![](https://secure.gravatar.com/avatar/911fc13a8e992ab39b62f91d6eba5f54.jpg?s=120&d=mm&r=g)
* I claimed:
This is of course wrong and would probably not be misunderstood as that. What I meant to say is that >>> if a: ... would be evaluated as "true" if a is anywhere non-zero. It was too late in the night it seems... Sorry for the confusion. Joachim
![](https://secure.gravatar.com/avatar/b24e93182e89a519546baa7bafe054ed.jpg?s=120&d=mm&r=g)
Perry Greenfield wrote:
I favour option 1, otherwise there is a need to define just what it means. Does it return a Boolean array indicating those cells which are nonzero, or a single Boolean value indicating that some cell is nonzero, or a single Boolean value indicating that all cells are nonzero? Colin W.
![](https://secure.gravatar.com/avatar/5a7d8a4d756bb1f1b2ea729a7e5dcbce.jpg?s=120&d=mm&r=g)
Faheem Mitha wrote:
[snip]
The following should be enough to get you started: #include "Python.h" #include "Numeric/arrayobject.h" #include "blitz/array.h" using namespace std; using namespace blitz; // Convert a Numpy array to a blitz one, using the original's data (no copy) template<class T, int N> static Array<T,N> py_to_blitz(PyArrayObject* arr_obj) { int T_size = sizeof(T); TinyVector<int,N> shape(0); TinyVector<int,N> strides(0); int *arr_dimensions = arr_obj->dimensions; int *arr_strides = arr_obj->strides; for (int i=0;i<N;++i) { shape[i] = arr_dimensions[i]; strides[i] = arr_strides[i]/T_size; } return Array<T,N>((T*) arr_obj->data,shape,strides,neverDeleteData); } This is what I use for exactly the problem you are describing, and this code was pretty much lifted, with minor changes, from weave's auto-generated C++. What I do, to sidestep the memory management problems, is let python allocate all my Numeric arrays (using zeros() if I have no data). I then use those inside my C++ code as blitz++ arrays via the above snippet. Any changes made by the C++ code are automatically reflected in the Numeric arary, since the blitz object is using the Numeric data area. I hope this helps. Let me know if you need more help, I can mail you complete example code. Cheers, f
![](https://secure.gravatar.com/avatar/b3308afd77cbb561a4c676aa091e55fc.jpg?s=120&d=mm&r=g)
On Mon, 29 Mar 2004, Fernando Perez wrote:
Thanks. That is very helpful. I've incorporated this into my code, and managed to get a simple example with boost working (extracting an element of an array). See below. Do you have a similar function which converts a blitz array to a numarray/Numeric one? If not, I can easily concoct one for myself. I now need to try to figure out what this reference counting stuff is about. I read the official Python docs but am still currently very confused. Faheem. *************************************************************************** #include "Python.h" #include "Numeric/arrayobject.h" #include <blitz/array.h> using namespace std; using namespace blitz; template<class T, int N> static Array<T,N> py_to_blitz(PyArrayObject* arr_obj); static PyObject * arrmod_elem(PyObject *self, PyObject *args); static PyMethodDef arrmod_methods[] = { {"elem", (PyCFunction)arrmod_elem, METH_VARARGS, "Returns the trace of a two-dimensional array.\n"}, {NULL, NULL, 0, NULL} }; PyMODINIT_FUNC initarrmod(void) { (void) Py_InitModule3("arrmod", arrmod_methods, "Returns the Trace of a two-dimensional array.\n"); import_array(); } static PyObject * arrmod_elem(PyObject *self, PyObject *args) { PyObject *input, *result; PyArrayObject *array; double el; int i, j; if (!PyArg_ParseTuple(args, "Oii", &input, &i, &j)) return NULL; array = (PyArrayObject *) PyArray_ContiguousFromObject(input, PyArray_DOUBLE, 2, 2); if (array == NULL) return NULL; Array<double,2> arr = py_to_blitz<double,2>(array); el = arr(i,j); result = Py_BuildValue("d",el); return result; } // Convert a Numpy array to a blitz one, using the original's data (no // copy) template<class T, int N> static Array<T,N> py_to_blitz(PyArrayObject* arr_obj) { int T_size = sizeof(T); TinyVector<int,N> shape(0); TinyVector<int,N> strides(0); int *arr_dimensions = arr_obj->dimensions; int *arr_strides = arr_obj->strides; for (int i=0;i<N;++i) { shape[i] = arr_dimensions[i]; strides[i] = arr_strides[i]/T_size; } return Array<T,N>((T*) arr_obj->data,shape,strides,neverDeleteData); }
![](https://secure.gravatar.com/avatar/5a7d8a4d756bb1f1b2ea729a7e5dcbce.jpg?s=120&d=mm&r=g)
Glad it helped.
Do you have a similar function which converts a blitz array to a numarray/Numeric one? If not, I can easily concoct one for myself.
No, I don't have one. But it's pretty straightforward to write one, the Numeric C API provides functions to create arrays from existing data. You'll be surprised how conceptually similar Blitz and Numeric turn out to be (which is a good thing).
As I said, so far I've been able to bypass this issue by creating my arrays in python and 'filling' them in blitz. But this is not always a realistic possibility, so you may indeed have to learn the reference counting management details. Best, f
![](https://secure.gravatar.com/avatar/c7976f03fcae7e1199d28d1c20e34647.jpg?s=120&d=mm&r=g)
Todd pointed out to me that he's gotten some complaints about the difference in how numarray nonzero behavior works. In version 0.9 the use of an array for a truth value was changed from a deprecation warning to an exception. We deprecated it because the meaning of an array as a truth value was ambiguous (or potentially confusing at best) and we felt it was better to force users to be explicit about what they wanted. Nevertheless, Numeric does allow that and removing the capability to use it in that context breaks some code. We can give some thought to reversing that decision, but since we only hear from those that don't like the new behavior, we'd like to get some sense if there are many that do like the new behavior. To clarify, what happens when: x = zeros(2) if x: print "true!" I see 3 possibilities: 1) Keep it the way it is, no use of an array is permitted; an exception is raised. 2) Print a warning message when it is used that way, and always keep it at that, i.e., warning message only. 3) Replicate Numeric behavior. Personally, I prefer 1) since I think there is too much confusion between whether false means an empty (len=0) array (like lists), or an array that contains all zeros, and that it is worth breaking backward compatibility on this (functions should be used to distinguish which meaning is desired). Perry [Option 2 on the previous question wins hands down]
![](https://secure.gravatar.com/avatar/5a7d8a4d756bb1f1b2ea729a7e5dcbce.jpg?s=120&d=mm&r=g)
Perry Greenfield wrote:
I tend to prefer keeping compatibility with Python list behavior. I think that in general, numarray arrays should behave different from lists only when truly justified (for example, at least in numeric all ufuncs can be applied to a list of floats as much as to an array, which is nice). Since anyone who knows python already knows that lists are true based on whether they contain _anything_: In [9]: a=[] In [10]: b=[0] In [11]: if a: ....: print 1 ....: In [12]: if b: ....: print 1 ....: 1 then I think it's a fair, clean extension to make NumArray behave similarly. I think a good guiding principle is to keep the number of special cases for arrays at a strict minimum (there are obviously cases where it does make sense to have special cases). Besides, this is an O(1) check, so it doesn't have the dangers of checking for all elements being zero. Such a potentially expensive operation should _definitely_ happen only if explicitly requested, IMHO. Regards, f
![](https://secure.gravatar.com/avatar/911fc13a8e992ab39b62f91d6eba5f54.jpg?s=120&d=mm&r=g)
* Fernando Perez [2004-04-02 00:51]:
How often does it happen that you have an empty array, which you need to test for? Certainly not as frequently as in case of a list. I think that if
then I would follow that
should also _both_ evaluate as "false".
But if the behaviour is unambiguously defined, I see no problem here, because I can as well check the length of a list using len(). BTW, How about >>> a = [[]] >>> if a: ... ? Cheers, Joachim
![](https://secure.gravatar.com/avatar/5a7d8a4d756bb1f1b2ea729a7e5dcbce.jpg?s=120&d=mm&r=g)
Joachim Saul wrote:
Well, just these days I'm working with code where I'm precisely using that kind of check, relying on Numeric's behavior. But I know that with potentially nested structures, it's probably best to be explicit and do all checks via things like: if len(a) and if a==0 etc. So I could also be +1 on Perry's wish for turning it into an extension, at the expense of my current code breaking. Cheers, f.
![](https://secure.gravatar.com/avatar/911fc13a8e992ab39b62f91d6eba5f54.jpg?s=120&d=mm&r=g)
* Fernando Perez [2004-04-02 02:15]:
But what would a==0 return? A boolean? An array of booleans? An array if integers? The way it is evaluated now in Numeric is pretty clear, though there could be other evaluations. For instance, if a==1: might be understood as "if a is everywhere one" or "if the length of vector a is one", etc. The current meaning is "if a is anywhere one", which may not be intuitive under all circumstances, but as long as it is the well defined behaviour, I simply don't see the need to change or forbid that. Nested structures are an important argument against keeping compatibility with Python list behaviour. For instance >>> if []: print "True" >>> if [[]]: print "True" True Which perfectly makes sense in Python, but an equivalent behaviour of a numeric array would be complete humbug. Thus, I don't think that it would make any sense to emulate Python list behaviour, rather I would support keeping the *existing* Numeric behaviour. Cheers, Joachim
![](https://secure.gravatar.com/avatar/911fc13a8e992ab39b62f91d6eba5f54.jpg?s=120&d=mm&r=g)
* I claimed:
This is of course wrong and would probably not be misunderstood as that. What I meant to say is that >>> if a: ... would be evaluated as "true" if a is anywhere non-zero. It was too late in the night it seems... Sorry for the confusion. Joachim
![](https://secure.gravatar.com/avatar/b24e93182e89a519546baa7bafe054ed.jpg?s=120&d=mm&r=g)
Perry Greenfield wrote:
I favour option 1, otherwise there is a need to define just what it means. Does it return a Boolean array indicating those cells which are nonzero, or a single Boolean value indicating that some cell is nonzero, or a single Boolean value indicating that all cells are nonzero? Colin W.
participants (5)
-
Colin J. Williams
-
Faheem Mitha
-
Fernando Perez
-
Joachim Saul
-
Perry Greenfield