Changing return types in C API
One thing I like about numarray's "native" C API is that for functions that return arrays, the return type is PyArrayObject *, as opposed to the Numeric-compatible API, where the return types are PyObject *. Returning PyObject * in this case basically means you're *always* doing a cast like this: PyObject *o; PyArrayObject *a; a = (PyArrayObject *)PyArray_ContiguousFromObject(o, PyArray_DOUBLE, 2, 2); Could we change the API so that PyArrayObject * is used as the return type? On a similar note: the 'data' member of PyArrayObject's is a char *, where it really should be a void *. Being a void pointer would have the advantage of not needing the cast in cases like this: double *adata = (double)(a->data); It would also mean that accidentally dereferencing the pointer would be a compiler error: currently, a->data[0] is valid, but if a->data was a void *, it wouldn't be. Both of these changes wouldn't require recompiling extensions (since we're just relabelling what the pointer types are), and would be source-compatible with old code (since the casts used would now be superfluous). Newer code will also be easier to write. I'll work on the changes in a svn branch if people think this is a good idea. -- |>|\/|< /------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm@physics.mcmaster.ca
This probably belongs to numpy-discussion. More below. On 1/31/06, David M. Cooke <cookedm@physics.mcmaster.ca> wrote:
... Could we change the API so that PyArrayObject * is used as the return type?
-1 Exposing the definition of the object structs is discouraged in python code. See for example a comment in intobject.h /*The type PyIntObject is (unfortunately) exposed here so we can declare _Py_TrueStruct and _Py_ZeroStruct in boolobject.h; don't use this. */ In numpy this is necessary for performance reasons so that accessors can be implemented as macros. Making return types PyArrayObject * will likely lead people to use direct access to struct members instead of macros.
On a similar note: the 'data' member of PyArrayObject's is a char *, where it really should be a void *. Being a void pointer would have the advantage of not needing the cast in cases like this:
double *adata = (double)(a->data);
I think you meant (double*)
It would also mean that accidentally dereferencing the pointer would be a compiler error: currently, a->data[0] is valid, but if a->data was a void *, it wouldn't be.
+1 Note, however that implicit cast is illegal in C++ and may generate warnings on some compilers (I think gcc -pedantic would, but not sure). Another problem with void* is that a->data + n will become invalid, which is used in many places if I recall correctly. (a->data + n code is actually suboptimal when n is not constant because most CPUs have special opcodes for pointer arithmetics that compilers can generate if n is compile time constant)
Sasha <ndarray@mac.com> writes:
This probably belongs to numpy-discussion. More below.
On 1/31/06, David M. Cooke <cookedm@physics.mcmaster.ca> wrote:
... Could we change the API so that PyArrayObject * is used as the return type?
-1
Exposing the definition of the object structs is discouraged in python code. See for example a comment in intobject.h /*The type PyIntObject is (unfortunately) exposed here so we can declare _Py_TrueStruct and _Py_ZeroStruct in boolobject.h; don't use this. */ In numpy this is necessary for performance reasons so that accessors can be implemented as macros. Making return types PyArrayObject * will likely lead people to use direct access to struct members instead of macros.
Well, I *already* use direct access :-). That's the way you do it with Numeric... But I'll concede your point: the preferred way should be PyArray_DATA and friends. I notice that CAPI.txt in numpy/doc just mentions the macros. Changing the data member of PyArrayObject to void* would make those easier to use.
On a similar note: the 'data' member of PyArrayObject's is a char *, where it really should be a void *. Being a void pointer would have the advantage of not needing the cast in cases like this:
double *adata = (double)(a->data);
I think you meant (double*)
Yep.
It would also mean that accidentally dereferencing the pointer would be a compiler error: currently, a->data[0] is valid, but if a->data was a void *, it wouldn't be.
+1
Note, however that implicit cast is illegal in C++ and may generate warnings on some compilers (I think gcc -pedantic would, but not sure).
As long as it's fine in the headers, C++ can use the explicit cast. However, for C++ I'd suggest wrapping using templates. I've got something like this that I used for Numeric template <typename T> struct PyArray_type_info { static PyArray_TYPES type() { return PyArray_NOTYPE; } static const char* name() { return "notype"; } } template <> struct PyArray_type_info<double> { static PyArray_TYPES type() { return PyArray_DOUBLE } static const char* name() { return "double"; } } then, something like template <typename T> struct ArrayObject { typedef PyArray_type_info<T> type_info; PyObject *a; T* data() { return static_cast<T *>(PyArray_DATA(a)); } // Add strides, flags, [] and () operators, etc. } template <typename T> ArrayObject<T> contiguous_from_any(PyObject *obj, int min, int max) { PyObject *ao = PyArray_ContiguousFromAny(obj, PyArray_type_info<T>.type(), min, max); ArrayObject<T> a = {ao}; return a; } then your code becomes the simple ArrayObject<double> a = contiguous_from_any<double>(obj, 1, 1); double *d = ao.data(); Or something like that; this is untested code. But using template specialization like this could cut down a lot of casting, and make things safer in C++. Plus you could add refcount handling and all that :-)
Another problem with void* is that a->data + n will become invalid, which is used in many places if I recall correctly. (a->data + n code is actually suboptimal when n is not constant because most CPUs have special opcodes for pointer arithmetics that compilers can generate if n is compile time constant)
I'll look for those. One trouble is the strides member of PyArrayObject; it's harder by a bit to apply a stride given in bytes to a double * (say). Although this is the same problem that you'd have after casting from the char* to the double *. -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm@physics.mcmaster.ca
David M. Cooke writes:
ArrayObject<double> a = contiguous_from_any<double>(obj, 1, 1); double *d = ao.data();
Or something like that; this is untested code. But using template specialization like this could cut down a lot of casting, and make things safer in C++. Plus you could add refcount handling and all that :-)
On this topic: I've just about finished testing our boost:::python::numeric helper functions (http://www.eos.ubc.ca/research/clouds/num_util.html) with numpy. The only change required was a new free function to replace array.typecode(). Once I finish the cleanup and a couple of new examples I'll post the link to the topical software page with a short description. Regards, Phil
On Wednesday 01 February 2006 18:13, Philip Austin wrote:
On this topic: I've just about finished testing our boost:::python::numeric helper functions (http://www.eos.ubc.ca/research/clouds/num_util.html) with numpy. The only change required was a new free function to replace array.typecode(). Once I finish the cleanup and a couple of new examples I'll post the link to the topical software page with a short description.
Any news on this? The code on the page above still points to the Numeric version. Regards, Ravi
David M. Cooke wrote:
Sasha <ndarray@mac.com> writes:
I'll look for those. One trouble is the strides member of PyArrayObject; it's harder by a bit to apply a stride given in bytes to a double * (say). Although this is the same problem that you'd have after casting from the char* to the double *.
This is the real trouble. The reason data is a char* is because strides is given in bytes. If we were to change the meaning of strides to "elements", this would break a lot of code that assumes it is given in bytes. Also, there is much more flexibility when strides can be given in terms of bytes. You can define a record with float, int16 elements, and now the float array can also be defined easily using strides. Of course it is a misaligned array, but that is the point of the ALIGNED flag... -Travis
Travis Oliphant <oliphant@ee.byu.edu> writes:
David M. Cooke wrote:
Sasha <ndarray@mac.com> writes:
I'll look for those. One trouble is the strides member of PyArrayObject; it's harder by a bit to apply a stride given in bytes to a double * (say). Although this is the same problem that you'd have after casting from the char* to the double *.
This is the real trouble. The reason data is a char* is because strides is given in bytes. If we were to change the meaning of strides to "elements", this would break a lot of code that assumes it is given in bytes.
Also, there is much more flexibility when strides can be given in terms of bytes. You can define a record with float, int16 elements, and now the float array can also be defined easily using strides. Of course it is a misaligned array, but that is the point of the ALIGNED flag...
Hmm.. Ok, how about this? Change the definition of PyArray_DATA to #define PyArray_DATA(obj) ((void *)(((PyArrayObject *)(obj))->data)) (i.e., throw in an explicit void * cast). If you're assigning to a char *, this will still work for bytes. If you're dereferencing it, the compiler should throw an error. Also, introduce #define PyArray_BYTES(obj) (((PyArrayObject *)(obj))->data) when you actually want the byte-based version. -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm@physics.mcmaster.ca
David M. Cooke wrote:
Ok, how about this? Change the definition of PyArray_DATA to
#define PyArray_DATA(obj) ((void *)(((PyArrayObject *)(obj))->data))
(i.e., throw in an explicit void * cast). If you're assigning to a char *, this will still work for bytes. If you're dereferencing it, the compiler should throw an error. Also, introduce
#define PyArray_BYTES(obj) (((PyArrayObject *)(obj))->data)
when you actually want the byte-based version.
Sounds great. -Travis
participants (6)
-
cookedm＠physics.mcmaster.ca -
David M. Cooke
-
Philip Austin -
Ravikiran Rajagopal -
Sasha -
Travis Oliphant