David, I briefly took a look at your code, and I have a very, very important observation. Your implementation make uses of low level dlopening. Then, your are going to have to manage all the oddities of runtime loading in the different systems. In this case, 'libtool' could really help. I know, it is GPL, but AFAIK it has some special licencing that let's you ship it with your code in the same licence terms than your code. But, I definitely think that a betther approach would be using a stubs mechanism ala TCL, or wath is currently used in some extension modules in core python, like cStringIO. In short, you access all your functions from a pointer do a struct (statically or heap allocated) where each struct member is filled with a pointer to a function. This is pretty much similar to C++ virtual tables, or the Cython cdef's classes with cdef's methods. And then you just let Python do the work of dynamic loading of extension modules. The numpy C/API uses a similar approach, but uses an array. IMHO, the struct approach is cleaner. What do you think about this? On 4/28/08, David Cournapeau <david@ar.media.kyoto-u.ac.jp> wrote:
Hi,
I've just started working on a prototype for a plugin system for numpy. The plugin aims at providing a framework for the following user cases: - runtime selection of blas/lapack/etc...: instead of harcoding in the binary one blas/lapack implementation, numpy could choose the SSE optimized if the CPU supports SSE, etc... - this could also be used for core numpy, for example ufuncs: if we want to start implementing some tight loop with aggressively optimized code (SSE, etc...), we could again ship with a default pure C implementation, and choose the best one at runtime. - we could even have a system to choose a different implementation (for example, right now, scipy is shipped with a slow fft for licensing issues mainly, and people installing fftw could then tell scipy to use fftw instead of the included one).
Right now, the prototype does not do much, and only works for linux; I mainly focused on automatic generation of the plugin from a list of functions, and transparent use from numpy point of view. It provides the plugin api through pure function pointers, without the need for the user to be aware of it. For example, if you have an api with the following functions:
void foo1(); int foo2(); int foo3(int); int foo4(double* , double*); int foo5(double* , double*, int);
The current implementation would build the boilerplate to load those functions, etc... and you would just use those functions in numpy like the following:
init_foo();
/* all functions are prefixed with npyw, for numpy wrapper */ npyw_foo1(); npyw_foo2(n); etc...
The code can be found there:
https://code.launchpad.net/~david-ar/+junk/numplug
And some thinking (pretty low content for now):
http://www.scipy.org/RuntimeOptimization
cheers,
David _______________________________________________ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
-- Lisandro Dalcín --------------- Centro Internacional de Métodos Computacionales en Ingeniería (CIMEC) Instituto de Desarrollo Tecnológico para la Industria Química (INTEC) Consejo Nacional de Investigaciones Científicas y Técnicas (CONICET) PTLC - Güemes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594