[Numpy-discussion] Exporting Numpy C functionality to other extension modules

[Francesc Alted]

A Dimecres 15 Gener 2003 23:33, Jack Jansen va escriure:
> Actually, wrt my previous message on cobjects for communicating between
> extension modules, we can do one better!
>
> This is an idea I've been toying with for the MacPython extension
> types, and I think it's applicable to Numeric too. It goes as follows.
>
> Each Numeric object has an attribute with a well-known name, lets call
> it "__Numeric_C_interface". This is a Cobject, and it is shared among
> all Numeric objects of the same type. The value of this C object is a
> pointer to a C structure with pointers to all the C routines you might
> want to call on the object, basically the PyArray_API structure (I
> think). The descr of the C object is a string with the version number
> of this particular PyArray_API structure.
>
> An extension module that knows about this protocol and gets passed an
> object that it think might be a Numeric array checks whether the object
> has an __Numeric_C_interface attribute. If so it retrieves it, checks
> that it is a Cobject, gets the descriptor and tests it for
> compatibility and if it is compatible gets the cobject pointer and
> happily calls all the Numeric routines it needs.

That's a nice idea. But I see two drawbacks:

- numarray needs to be reworked to include the Cobject descriptors, although
I don't know if this would be difficult or not.

- you still need to have Numeric or numarray installed on the client
machine. This could be the usual case, but what about extensions that want
to use Numeric internally (because a number of reasons, like better number
representation, convenient interface to C, etc) without forcing the user to
install it?

However, designing a small library with a minimalist API (I'm thinking in
something similar to zlib) could be very handy in allowing extensions (but
also native python modules) to deal with numarray objects. 

As I said before, this would require the user to install only this small
library, but it can also be included in the application or package. However,
this second alternative can be tricky, as Chris Barker has signaled, because
the different numarray versions coming in the future. But IMO a series of
factors may alleviate this handicap:

- The numarray data structure should be very stable, as improvements are
normally made at the functionality level.

- The library should provide a minimalistic, high level API that, if it is
well designed, should cope with small modifications in the numarray data
structures. 

- Finally, when these differences has to be added, and that would break the
current API, this version should be marked as a major release,
and existing extensions (or whatever software that is embedding the library)
will know that they have to release new versions if they want to support the
newest objects. But, hopefully, that should happen quite unfrequently.

Of course, this small library should cope with both numarray and Numeric (at
least, the not too old versions of it) objects. But I think this shouldn't
pose a big problem as the actual numarray API already can do that.

This logical separation between structure and functionality migth also lead
to a better acceptation by numerical software cratftsmen, as they can be
more confident in that the API to deal with numarray objects will be quite
stable throughout the time.

Well, this is just a thought. I must confess that I'm so interested on that
issue because I really want to support numarray objects in my project, and
I'm just wondering which is the best way to do that without creating too
much nuissance to the users. In fact, I'm pondering to build up such a
library myself, but that can be a waste of time if I've to redone it in
every numarray release.

Cheers,

-- 
Francesc Alted