[Numpy-discussion] Follow-up Numarray header PEP

[Todd Miller]

Hi Gerard,

I finally got to your numnum stuff today... awesome work!  You've got
lots of good suggestions.  Here are some comments:

1. Thanks for catching the early return problem with numarray's
import_array().  It's not just bad, it's wrong.  It'll be fixed for 1.1.

2. That said,  I think expanding the macros in-line in numnum is a
mistake.  It seems to me that "import_array(); PyErr_Clear();" or
something like it ought to be enough...  after numarray-1.1 anyway.  

3. I think there's a problem in numnum.toNP() because of numarray's
array "behavior" issues.  A test needs to be done to ensure that the
incoming array is not byteswapped or misaligned;  if it is, the easy fix
is to make a numarray copy of the array before copying it to Numeric.

4. Kudos for the LP64 stuff.  numconfig is a thorn in the side of the
PEP,  so I'll put your techniques into numarray for 1.1.  HAS_FLOAT128
is not currently used,  so it might be time to ditch it.  Anyway,
thanks!

5. PyArray_Present() and isArray() are superfluous *now*.  I was
planning to add them to Numeric. 

6. The LGPL may be a problem for us and is probably an issue if we ever
try to get numnum into the Python distribution.  It would be better to
release numnum under the modified BSD license,  same as numarray.

7. Your API struct was very clean.  Eventually I'll regenerate numarray
like that.

8. I logged your comments and bug reports on Source Forge and eventually
they'll get fixed.

A to Z the numnum/pep code is beautiful.  Next stop, header PEP update.

Regards,
Todd

On Sun, 2004-07-18 at 17:24, gerard.vermeulen at grenoble.cnrs.fr wrote: 
> Hi Todd,
> 
> This is a follow-up on the 'header pep' discussion.
> 
> The attachment numnum-0.1.tar.gz contains the sources for the
> extension modules pep and numnum.  At least on my systems, both
> modules behave as described in the 'numarray header PEP' when the
> extension modules implementing the C-API are not present (a situation
> not foreseen by the macros import_array() of Numeric and especially
> numarray).  IMO, my solution is 'bona fide', but requires further
> testing.
> 
> The pep module shows how to handle the colliding C-APIs of the Numeric
> and numarray extension modules and how to implement automagical
> conversion between Numeric and numarray arrays.
> 
> For a technical reason explained in the README, the hard work of doing
> the conversion between Numeric and numarray arrays has been delegated
> to the numnum module.  The numnum module is useful when one needs to
> convert from one array type to the other to use an extension module
> which only exists for the other type (eg. combining numarray's image
> processing extensions with pygame's Numeric interface):
> 
> Python 2.3+ (#1, Jan  7 2004, 09:17:35)
> [GCC 3.3.1 (SuSE Linux)] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
> >>> import numnum; import Numeric as np; import numarray as na
> >>> np1 = np.array([[1, 2], [3, 4]]); na1 = numnum.toNA(np1)
> >>> na2 = na.array([[1, 2, 3], [4, 5, 6]]); np2 = numnum.toNP(na2)
> >>> print type(np1); np1; type(np2); np2
> <type 'array'>
> array([[1, 2],
>        [3, 4]])
> <type 'array'>
> array([[1, 2, 3],
>        [4, 5, 6]],'i')
> >>> print type(na1); na1; type(na2); na2
> <class 'numarray.numarraycore.NumArray'>
> array([[1, 2],
>        [3, 4]])
> <class 'numarray.numarraycore.NumArray'>
> array([[1, 2, 3],
>        [4, 5, 6]])
> >>>
> 
> The pep module shows how to implement array processing functions which
> use the Numeric, numarray or Sequence C-API:
> 
> static PyObject *
> wysiwyg(PyObject *dummy, PyObject *args)
> {
>     PyObject *seq1, *seq2;
>     PyObject *result;
> 
>     if (!PyArg_ParseTuple(args, "OO", &seq1, &seq2))
>         return NULL;
> 
>     switch(API) {
>     case NumericAPI:
>     {
>         PyObject *np1 = NN_API->toNP(seq1);
>         PyObject *np2 = NN_API->toNP(seq2);
>         result = np_wysiwyg(np1, np2);
>         Py_XDECREF(np1);
>         Py_XDECREF(np2);
>         break;
>     }
>     case NumarrayAPI:
>     {
>         PyObject *na1 = NN_API->toNA(seq1);
>         PyObject *na2 = NN_API->toNA(seq2);
>         result = na_wysiwyg(na1, na2);
>         Py_XDECREF(na1);
>         Py_XDECREF(na2);
>         break;
>     }
>     case SequenceAPI:
>         result = seq_wysiwyg(seq1, seq2);
>         break;
>     default:
>         PyErr_SetString(PyExc_RuntimeError, "Should never happen");
>         return 0;
>     }
> 
>     return result;
> }
> 
> See the README for an example session using the pep module showing that
> it is possible pass a mix of Numeric and numarray arrays to pep.wysiwyg().
> 
> Notes:
> 
> - it is straightforward to adapt pep and numnum so that the conversion
>   functions are linked into pep instead of imported.
> 
> - numnum is still 'proof of concept'.  I am thinking about methods to
>   make those techniques safer if the numarray (and Numeric?) header
>   files make it never into the Python headers (or make it safer to
>   use those techniques with Python < 2.4).  In particular it would
>   be helpful if the numerical C-APIs export an API version number,
>   similar to the versioning scheme of shared libraries -- see the
>   libtool->versioning info pages. 
> 
> I am considering three possibilities to release a more polished
> version of numnum (3rd party extension writers may prefer to link
> rather than import numnum's functionality):
> 
> 1. release it from PyQwt's project page
> 2. register an independent numnum project at SourceForge
> 3. hand numnum over to the Numerical Python project (frees me from
>    worrying about API changes).
> 
> 
> Regards -- Gerard Vermeulen



--