Philip Austin wrote:

Todd, thanks for the quick response. Regarding your header comments:

/* Extensions constructed from seperate compilation units can access the C-API defined here by defining "libnumarray_UNIQUE_SYMBOL" to a global name unique to the extension. Doing this circumvents the requirement to import libnumarray into each compilation unit, but is nevertheless mildly discouraged as "outside the Python norm" and potentially leading to problems. Looking around at "existing Python art", most extension modules are monolithic C files, and likely for good reason. */

I'd agree with this sentiment, but with numarray's NDInfo interface I think we were stuck (we have an implementation file for a bunch of boost numarray helper functions that require the API but have static data initializations, so they can't go into a header).

Long term, you would be better off ditching NDInfo and working with the Numeric compatability API. Since it's a subset of the Numeric API, it may not support your purposes. However, if it doesn't, I want to extend it until it does.

With Numeric, we could avoid import_array altogether, and get everything we needed from PyObject_CallObject calls to Numeric to create arrays using zeros or ones, followed by a cast to PyArrayObject* to get the data pointer and fill the array. With numarray, we need (or needed) getNDInfo to do the same thing, mandating an import_libnumarray. Are the plans for the revised interface to behave in the same way as NDInfo?

The current plan is to deprecate the "native" APIs which make use of NDInfo and to make numarray as compatible as possible with Numeric. The bulk of the changes which make this reasonable are in numarray CVS now. In a nutshell, prior to our use of C basetypes, we needed something like NDInfo. Now that we're using C basetypes, NDInfo appears less efficient, less compatible, and more complex than the "Numeric compatability" interface. Its possible for our "basetypes" to appear to be Numeric arrays, so that's what we did. If you guys find that you need missing parts of the Numeric C-API, I'll add them to numarray. So far, I've just added what I needed or knew someone else needed.

It's not a big deal to us one way or another, but it would simplify Dave Abraham's boost support for numeric/numarray if much of the functionality could be done through python calls from the C side.

Can you send me the Numeric wrapper code or let me know where to get it?

Thanks, Phil

------------------------------------------------------- This sf.net email is sponsored by:ThinkGeek Welcome to geek heaven. http://thinkgeek.com/sf _______________________________________________ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion

Thanks, Todd -- Todd Miller jmiller@stsci.edu STSCI / SSB

Philip Austin wrote:

Todd Miller writes:

...

Can you send me the Numeric wrapper code or let me know where to get it?

It's now on the cvs main trunk:

cvs -d:pserver:anonymous@cvs.boost.sourceforge.net:/cvsroot/boost login cvs -z3 -d:pserver:anonymous@cvs.boost.sourceforge.net:/cvsroot/boost checkout boost

You'll need all of boost if you want to try to compile/run the test code. If you haven't worked with boost and boost.build (bjam) before, I could send my install notes. If you'd just like to browse the code, the pertinent files are:

boost/libs/python/test/numpy.cpp boost/libs/python/test/numpy.py boost/python/numeric.hpp

and the new tutorial (which doesn't cover numpy yet) is quite good:

boost/boost/libs/python/doc/tutorial

as well as:

boost/libs/python/doc/v2/reference.html (also without numpy documentation)

Regards, Phil

I took a look at the files above trying to see what you're doing. What I thought I would see, was some form of direct coupling between a module in boost and either the Numeric or numarray C-API. However, when I search the whole source tree (*.cpp *.hpp) I can find neither PyArrayObject, arrayobject.h, NDInfo, nor libnumarray.h. So I'm not sure what part of any Numeric/numarray C-API you're using now (if any) and in particular, what isn't working for you. Note: numarray and Numeric are not totally compatible at either the C or Python levels, so either area could have been a source of difficulty. Todd -- Todd Miller jmiller@stsci.edu STSCI / SSB

Philip Austin <paustin@eos.ubc.ca> writes:

...

What I thought I would see, was some form of direct coupling between a module in boost and either the Numeric or numarray C-API. However, when I search the whole source tree (*.cpp *.hpp) I can find neither PyArrayObject, arrayobject.h, NDInfo, nor libnumarray.h. So I'm not sure what part of any Numeric/numarray C-API you're using now (if any)

The answer is -- no part of the C-API if he can get away with it. Your move towards Numeric compatibility for numarray means that he now can get away with it.

I've been getting away with it already. It's easy enough to access all that functionality through the usual Python/C API.

numeric wraps the module pointer, and calls functions through it -- e.g. (from boost/libs/python/src/numeric.cpp):

void array_base::resize(object const& shape) { attr("resize")(shape); }

which invokes PyObject_CallObject to call the array's resize method, with the tuple shape (similar to CXX). If the method isn't part of the library (say for Numeric and getflat) then a python exception is thrown.

Slow but effective. If you want to touch the raw PyObject*, of course, you can do this: void f(numeric::array x) { PyObject* danger = x.ptr(); // Cast to whatever you like. }

...

and in particular, what isn't working for you. Note: numarray and Numeric are not totally compatible at either the C or Python levels, so either area could have been a source of difficulty.

The problem was that, as long as a call to getNDInfo was required to access an array's data pointer, a Python-only approach like the above wouldn't work. Since it is now possible to do this via a cast to PyArrayObject* for a numarray, the problem has disappeared.

Not sure what you're saying here. I /do/ remember when I was writing the Numeric/NumArray support that there seemed no publicly-accessible definition of the structure of the underlying array objects.

(there's nothing stopping you from using the C-API if you want, though. For example you can create a numarray with data copied from a C array with something like:

numeric::array makeNum(int * data, int n=0){ object obj(detail::new_reference(PyArray_FromDims(1, &n, PyArray_INT))); char *arr_data = ((PyArrayObject*) obj.ptr())->data; memcpy(arr_data, data, 4 * n); return numeric::array(obj); }

Well, this uses undocumented implementation details. The supported approach is: handle<> raw_array(PyArray_FromDims(1, &n, PyArray_INT)); object obj(raw_array); Note, though, that you can also write: handle<PyArrayObject> raw_array(PyArray_FromDims(1, &n, PyArray_INT)); Assuming, of course, that PyArray_FromDims returns a PyArrayObject*. And now you have access to the PyArrayObject* through raw_array. Also note that it's probably smarter to use extract<numeric::array>(obj) Than numeric::array(obj) See the note about pitfalls at http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/*checkout*/boost/boost/libs/p... for the reasons why. -- David Abrahams * Boost Consulting dave@boost-consulting.com * http://www.boost-consulting.com

Philip Austin wrote:

David Abrahams writes:

...

Not sure what you're saying here. I /do/ remember when I was writing the Numeric/NumArray support that there seemed no publicly-accessible definition of the structure of the underlying array objects.

Numeric and numarray both have "arrayobject.h" which makes PyArrayObject public. Numarray also has "libnumarray.h" which indirectly exposes NDInfo.

As I understand it, the numarray now in CVS deprecates the NDInfo struct (see section 10.1 of numarray.pdf) in favor of PyArrayObject.

This is true, but is currently neither released nor documented. On the plus side, numarray is becoming more compatible with Numeric, getting faster, and simpler as well. Todd -- Todd Miller jmiller@stsci.edu STSCI / SSB