[Numpy-discussion] Follow-up Numarray header PEP

[Todd Miller]

On Sun, 2004-07-18 at 17:24, gerard.vermeulen at grenoble.cnrs.fr wrote:
> Hi Todd,
> 
> This is a follow-up on the 'header pep' discussion.

Great!  I was afraid you were going to disappear back into the ether.

Sorry I didn't respond to this yesterday...  I saw it but accidentally
marked it as "read" and then forgot about it as the day went on.

> The attachment numnum-0.1.tar.gz contains the sources for the
> extension modules pep and numnum.  At least on my systems, both
> modules behave as described in the 'numarray header PEP' when the
> extension modules implementing the C-API are not present (a situation
> not foreseen by the macros import_array() of Numeric and especially
> numarray).  

For numarray,  this was *definitely* foreseen at some point,  so I'm
wondering what doesn't work now...

> IMO, my solution is 'bona fide', but requires further
> testing.

I'll look it over today or tomorrow and comment more then.

> The pep module shows how to handle the colliding C-APIs of the Numeric
> and numarray extension modules and how to implement automagical
> conversion between Numeric and numarray arrays.

Nice;  the conversion code sounds like a good addition to me.  

> For a technical reason explained in the README, the hard work of doing
> the conversion between Numeric and numarray arrays has been delegated
> to the numnum module.  The numnum module is useful when one needs to
> convert from one array type to the other to use an extension module
> which only exists for the other type (eg. combining numarray's image
> processing extensions with pygame's Numeric interface):
> 
> Python 2.3+ (#1, Jan  7 2004, 09:17:35)
> [GCC 3.3.1 (SuSE Linux)] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
> >>> import numnum; import Numeric as np; import numarray as na
> >>> np1 = np.array([[1, 2], [3, 4]]); na1 = numnum.toNA(np1)
> >>> na2 = na.array([[1, 2, 3], [4, 5, 6]]); np2 = numnum.toNP(na2)
> >>> print type(np1); np1; type(np2); np2
> <type 'array'>
> array([[1, 2],
>        [3, 4]])
> <type 'array'>
> array([[1, 2, 3],
>        [4, 5, 6]],'i')
> >>> print type(na1); na1; type(na2); na2
> <class 'numarray.numarraycore.NumArray'>
> array([[1, 2],
>        [3, 4]])
> <class 'numarray.numarraycore.NumArray'>
> array([[1, 2, 3],
>        [4, 5, 6]])
> >>>
> 
> The pep module shows how to implement array processing functions which
> use the Numeric, numarray or Sequence C-API:
> 
> static PyObject *
> wysiwyg(PyObject *dummy, PyObject *args)
> {
>     PyObject *seq1, *seq2;
>     PyObject *result;
> 
>     if (!PyArg_ParseTuple(args, "OO", &seq1, &seq2))
>         return NULL;
> 
>     switch(API) {

We'll definitely need to cover API in the PEP.  There is a design choice
here which needs to be discussed some and any resulting consensus
documented.  I haven't looked at the attachment yet.

>     case NumericAPI:
>     {
>         PyObject *np1 = NN_API->toNP(seq1);
>         PyObject *np2 = NN_API->toNP(seq2);
>         result = np_wysiwyg(np1, np2);
>         Py_XDECREF(np1);
>         Py_XDECREF(np2);
>         break;
>     }
>     case NumarrayAPI:
>     {
>         PyObject *na1 = NN_API->toNA(seq1);
>         PyObject *na2 = NN_API->toNA(seq2);
>         result = na_wysiwyg(na1, na2);
>         Py_XDECREF(na1);
>         Py_XDECREF(na2);
>         break;
>     }
>     case SequenceAPI:
>         result = seq_wysiwyg(seq1, seq2);
>         break;
>     default:
>         PyErr_SetString(PyExc_RuntimeError, "Should never happen");
>         return 0;
>     }
> 
>     return result;
> }
> 
> See the README for an example session using the pep module showing that
> it is possible pass a mix of Numeric and numarray arrays to pep.wysiwyg().
> 
> Notes:
> 
> - it is straightforward to adapt pep and numnum so that the conversion
>   functions are linked into pep instead of imported.
> 
> - numnum is still 'proof of concept'.  I am thinking about methods to
>   make those techniques safer if the numarray (and Numeric?) header
>   files make it never into the Python headers (or make it safer to
>   use those techniques with Python < 2.4).  In particular it would
>   be helpful if the numerical C-APIs export an API version number,
>   similar to the versioning scheme of shared libraries -- see the
>   libtool->versioning info pages. 

I've thought about this a few times;  there's certainly a need for it in
numarray anyway... and I'm always one release too late.  Thanks for the
tip on libtool->versioning.

> I am considering three possibilities to release a more polished
> version of numnum (3rd party extension writers may prefer to link
> rather than import numnum's functionality):
> 
> 1. release it from PyQwt's project page
> 2. register an independent numnum project at SourceForge
> 3. hand numnum over to the Numerical Python project (frees me from
>    worrying about API changes).
> 
> Regards -- Gerard Vermeulen

(3) sounds best to me, for the same reason that numarray is a part of
the numpy project and because numnum is a Numeric/numarray tool.  There
is a small issue of sub-project organization (seperate bug tracking,
etc.),  but I figure if SF can handle Python,  it can handle Numeric,
numarray, and probably a number of other packages as well.  Something
like numnum should not be a problem and so to promote it, it would be
good to keep it where people can find it without having to look too
hard.

For now,  I'm again marking your post as "unread" and will revisit it
later this week.  In the meantime, thanks very much for your efforts
with numnum and the PEP.

Regards,
Todd