[Numpy-discussion] Starting to work on runtime plugin system for plugin (automatic sse optimization, etc...)

[David Cournapeau]

Hi,

    I've just started working on a prototype for a plugin system for 
numpy. The plugin aims at providing a framework for the following user 
cases:
    - runtime selection of blas/lapack/etc...: instead of harcoding in 
the binary one blas/lapack implementation, numpy could choose the SSE 
optimized if the CPU supports SSE, etc...
    - this could also be used for core numpy, for example ufuncs: if we 
want to start implementing some tight loop with aggressively optimized 
code (SSE, etc...), we could again ship with a default pure C 
implementation, and choose the best one at runtime.
    - we could even have a system to choose a different implementation 
(for example, right now, scipy is shipped with a slow fft for licensing 
issues mainly, and people installing fftw could then tell scipy to use 
fftw instead of the included one).

Right now, the prototype does not do much, and only works for linux; I 
mainly focused on automatic generation of the plugin from a list of 
functions, and transparent use from numpy point of view. It provides the 
plugin api through pure function pointers, without the need for the user 
to be aware of it. For example, if you have an api with the following 
functions:

void    foo1();
int     foo2();
int     foo3(int);
int     foo4(double* , double*);
int     foo5(double* , double*, int);

The current implementation would build the boilerplate to load those 
functions, etc... and you would just use those functions in numpy like 
the following:

init_foo();

/* all functions are prefixed with npyw, for numpy wrapper */
npyw_foo1();
npyw_foo2(n);
etc...

The code can be found there:

https://code.launchpad.net/~david-ar/+junk/numplug

And some thinking (pretty low content for now):

http://www.scipy.org/RuntimeOptimization

cheers,

David