[Numpy-discussion] Access to C/C++, typemaps, pyrex...

[Fernando Perez]

Hi all,

there are recurrent questions on the topic of low-level access to numpy, 
wrapping libraries, etc.  Many people have home-brewed SWIG typemaps, the 
cookbook has some (very nice) pyrex examples, etc.  I think that we could use 
a bit of unification work on this front, to make this particular issue easier 
to deal with for newcomers, and to reduce duplication of effort.

1. SWIG
=======

At the last scipy conference, I collected typemaps from a number of people who 
had them around and all I could find on the net.  I grabbed Scott Ransom's, 
Michel Sanner contributed his, Eric Jones gave us some as well, etc.  John 
Hunter worked on updating Eric's (which had some nice features) to work with 
plain C (they were originally C++), and Bill Spotz from Sandia NL agreed to do 
the real work of writing some clean, documented examples of typemap use 
starting with this codebase.

He gave me this code some time ago, and I shamefully just sat on it.  I'm 
finally doing something about it, so today I updated this code to work with 
numpy instead of Numeric, and all the tests pass.

You can grab it from here with an SVN client:

http://wavelets.scipy.org/svn/multiresolution/wavelets/trunk/numpy_swig

It is clearly documented, and while I'm sure it can use improvements and 
extensions, I think it would be a good starting point to have distributed 
/officially/ with numpy.  That way others can improve /this/ code, rather than 
reimplementing numpy typemaps for the millionth time.  D. Beazley (the SWIG 
author) has even indicated that he'd be happy to officially ship numpy 
typemaps with SWIG if something accepted by the community were sent to him.


2. Pyrex
========

After reading the excellent cookbook examples on pyrex usage, which I recently 
needed for a project, I decided to complete the code so it could be used 'out 
of the box'.  This simply meant a bit of organization work, updating the code 
to use the 'modern' cimport statement and .pxd files, and writing a proper 
setup.py file.  The result is here:

http://wavelets.scipy.org/svn/multiresolution/wavelets/trunk/numpyx/

I also think that pyrex should be 'officially' supported.  In particular, I 
think that the c_numpy.pxd file I put there, which is just a copy of that in 
mtrand, should be completed and shipped in the same headers directory as the 
rest of numpy's .h files.  This would make it a LOT easier for newcomers to 
use pyrex for writing numpy extensions, since all headers would be complete 
and available with all numpy installations.


Finally, if anyone is interested, that repository actually contains in the 
full checkout

http://wavelets.scipy.org/svn/multiresolution/wavelets/trunk/

examples of how to wrap a more complex library which requires the creation of 
   C structs from python objects.  Getting this right took some time, as not 
all the details for this kind of construction are clearly documented anywhere. 
  I hope it may be useful to others.


3. Wrapup
=========

I think it would be a good idea to ship the numpy_swig example directory (and 
perhaps also numpyx, if somewhat expanded) listed above with the standard 
distribution, as well as completing the .pxd file to expose all the numpy C 
API for pyrex.

I am not claiming to have done any of the hard work in numpyx or numpy_swig 
(though I did spend a lot of time on the wavelets stuff recently :), but I 
think we would all benefit from having this kind of 'infrastructure' code in 
the core.  Once there, improvements can be made over time, with less effort 
wasted.

If we agree on this, it's as simple as dropping those two directories 
somewhere on the numpy codebase (sandbox?, doc?).  Just having them 
'officially' available will be a benefit, I think.

I'd like to thank Bill for the time he put into the swig module, as well as 
others who contributed code for this.  Hopefully it will benefit everyone.

Regards,

f