[Numpy-discussion] Numarray header PEP

[Todd Miller]

On Thu, 2004-07-01 at 02:33, gerard.vermeulen at grenoble.cnrs.fr wrote: 
> On 30 Jun 2004 17:54:19 -0400, Todd Miller wrote
> > 
> > So... you use the "meta" code to provide package specific ordinary
> > (not-macro-fied) functions to keep the different versions of the
> > Present() and isArray() macros from conflicting.
> > 
> > It would be nice to have a standard approach for using the same
> > "extension enhancement code" for both numarray and Numeric.  The PEP
> > should really be expanded to provide an example of dual support for one
> > complete and real function, guts and all, so people can see the process
> > end-to-end;  Something like a simple arrayprint.  That process needs 
> > to be refined to remove as much tedium and duplication of effort as 
> > possible.  The idea is to make it as close to providing one 
> > implementation to support both array packages as possible.  I think it's
> > important to illustrate how to partition the extension module into
> > separate compilation units which correctly navigate the dual
> > implementation mine field in the easiest possible way.
> > 
> > It would also be nice to add some logic to the meta-functions so that
> > which array package gets used is configurable.  We did something like
> > that for the matplotlib plotting software at the Python level with 
> > the "numerix" layer, an idea I think we copied from Chaco.  The kind 
> > of dispatch I think might be good to support configurability looks like
> > this:
> > 
> > PyObject *
> > whatsThis(PyObject *dummy, PyObject *args)
> > {
> >     PyObject *result, *what = NULL;
> >     if (!PyArg_ParseTuple(args, "O", &what))
> >       return 0;
> >     switch(PyArray_Which(what)) {
> >       USE_NUMERIC:
> >          result = Numeric_whatsThis(what); break;
> >       USE_NUMARRAY:
> >          result = Numarray_whatsThis(what); break;
> >       USE_SEQUENCE:
> >          result = Sequence_whatsThis(what); break;
> >     }
> >     Py_INCREF(Py_None);
> >     return Py_None;
> > }
> > 
> > In the above,  I'm picturing a separate .c file for Numeric_whatsThis
> > and for Numarray_whatsThis.  It would be nice to streamline that to one
> > .c and a process which somehow (simply) produces both functions.
> > 
> > Or, ideally, the above would be done more like this:
> > 
> > PyObject *
> > whatsThis(PyObject *dummy, PyObject *args)
> > {
> >     PyObject *result, *what = NULL;
> >     if (!PyArg_ParseTuple(args, "O", &what))
> >        return 0;
> >     switch(Numerix_Which(what)) {
> >        USE_NUMERIX:
> >           result = Numerix_whatsThis(what); break;
> >        USE_SEQUENCE:
> >           result = Sequence_whatsThis(what); break;
> >     }
> >     Py_INCREF(Py_None);
> >     return Py_None;
> > }
> > 
> > Here, a common Numerix implementation supports both numarray and Numeric
> > from a single simple .c.  The extension module would do "#include
> > numerix/arrayobject.h" and "import_numerix()" and otherwise just call
> > PyArray_* functions.
> > 
> > The current stumbling block is that numarray is not binary compatible
> > with Numeric... so numerix in C falls apart.  I haven't analyzed 
> > every symbol and struct to see if it is really feasible... but it 
> > seems like it is *almost* feasible, at least for typical usage.
> > 
> > So, in a nutshell,  I think the dual implementation support you 
> > demoed is important and we should work up an example and kick it 
> > around to make sure it's the best way we can think of doing it.  
> > Then we should add a section to the PEP describing dual support as well.
> > 
> I would never apply numarray code to Numeric arrays and the inverse. It looks
> dangerous and I do not know if it is possible.  

I think that's definitely the marching orders for now... but you gotta
admit, it would be nice.

> The first thing coming
> to mind is that numarray and Numeric arrays refer to different type objects
> (this is what my pep module uses to differentiate them).  So, even if
> numarray and Numeric are binary compatible, any 'alien' code referring the
> the 'Python-standard part' of the type objects may lead to surprises.
> A PEP proposing hacks will raise eyebrows at least. 

I'm a little surprised it took someone to talk me out of it...  I'll
just concede that this was probably a bad idea.

> Secondly, most people use Numeric *or* numarray and not both.

A class of question which will arise for developers is this: "X works
with Numeric,  but X doesn't work with numaray."  The reverse also
happens occasionally.  For this reason, being able to choose would be
nice for developers.

> So, I prefer: Numeric In => Numeric Out or Numarray In => Numarray Out (NINO)
> Of course, Numeric or numarray output can be a user option if NINO does not
> apply.  

When I first heard it, I though NINO was a good idea,  with the
limitation that it doesn't apply when a function produces an array
without consuming any.  But... there is another problem with NINO that
Perry Greenfield pointed out:  with multiple arguments,  there can be a
mix of array types.  For this reason,  it makes sense to be able to
coerce all the inputs to a particular array package.  This form might
look more like:

switch(PyArray_Which(<no_parameter_at_all!>)) {
	case USE_NUMERIC:
		result = Numeric_doit(a1, a2, a3);  break;
	case USE_NUMARRAY:
		result = Numarray_doit(a1, a2, a3);  break;
	case USE_SEQUENCE:
		result = Sequence_doit(a1, a2, a3);  break;
}

One last thing:  I think it would be useful to be able to drive the code
into sequence mode with arrays.  This would enable easy benchmarking of
the performance improvement.

> (explicit safe conversion between Numeric and numarray is possible
> if really needed).
> 
>I'll try to flesh out the demo with real functions in the way you indicated
> (going as far as I consider safe).
> 
> The problem of coding the Numeric (or numarray) functions in more than
> a single source file has also be addressed.
> 
> It may take 2 weeks because I am off to a conference next week.


Excellent.  See you in a couple weeks.

Regards,
Todd