a possible way to implement a plogin system

David, in order to put clear what I was proposing to you in previous mail regarding to implementing plugin systems for numpy, please take a look at the attached tarball. The plugins are in charge of implementing the action of generic foo() and bar() functions in C. The example actually implements two different plugins in Cython. There is also an include file with some support declarations and some macro definitions wrap_foo and wrap_bar Next, in mymodule.pyx, I define python side foo() and bar() functions (calling funtins pointers in a struct via the macros wrap_xxx) and a use() for selecting the plugin at runtime. Finally, the test.py script is pure python code showing all this in action. If you want to give a try, then install Cython, next do 'make', and finally 'python test.py' Please note that I've implemented this is Cython just for convenience, it does not actually depend in any special Cython feature, and could be translated to pure C code with Python C/API calls. IMHO, this is the easier and cleaner way to deal inside python with plugins written in low-level C. It does not depend explicitely on 'dlopen' stuff from your side. Now tell me. What is in your mind that this is not general/robust enough as to have to explicitely deal with dlopen? It even works on systems with no dynload, provided that all is put inside the python executable. -- Lisandro Dalcín --------------- Centro Internacional de Métodos Computacionales en Ingeniería (CIMEC) Instituto de Desarrollo Tecnológico para la Industria Química (INTEC) Consejo Nacional de Investigaciones Científicas y Técnicas (CONICET) PTLC - Güemes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594

Sorry, I forgot to attach the code... On 4/30/08, Lisandro Dalcin <dalcinl@gmail.com> wrote:
David, in order to put clear what I was proposing to you in previous mail regarding to implementing plugin systems for numpy, please take a look at the attached tarball.
The plugins are in charge of implementing the action of generic foo() and bar() functions in C. The example actually implements two different plugins in Cython. There is also an include file with some support declarations and some macro definitions wrap_foo and wrap_bar
Next, in mymodule.pyx, I define python side foo() and bar() functions (calling funtins pointers in a struct via the macros wrap_xxx) and a use() for selecting the plugin at runtime.
Finally, the test.py script is pure python code showing all this in action. If you want to give a try, then install Cython, next do 'make', and finally 'python test.py'
Please note that I've implemented this is Cython just for convenience, it does not actually depend in any special Cython feature, and could be translated to pure C code with Python C/API calls.
IMHO, this is the easier and cleaner way to deal inside python with plugins written in low-level C. It does not depend explicitely on 'dlopen' stuff from your side.
Now tell me. What is in your mind that this is not general/robust enough as to have to explicitely deal with dlopen? It even works on systems with no dynload, provided that all is put inside the python executable.
-- Lisandro Dalcín --------------- Centro Internacional de Métodos Computacionales en Ingeniería (CIMEC) Instituto de Desarrollo Tecnológico para la Industria Química (INTEC) Consejo Nacional de Investigaciones Científicas y Técnicas (CONICET) PTLC - Güemes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594
-- Lisandro Dalcín --------------- Centro Internacional de Métodos Computacionales en Ingeniería (CIMEC) Instituto de Desarrollo Tecnológico para la Industria Química (INTEC) Consejo Nacional de Investigaciones Científicas y Técnicas (CONICET) PTLC - Güemes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594

On Wed, 2008-04-30 at 16:44 -0300, Lisandro Dalcin wrote:
David, in order to put clear what I was proposing to you in previous mail regarding to implementing plugin systems for numpy, please take a look at the attached tarball.
Thanks for looking at this Lisandro. The problem I see with the approach of struct vs direct function pointers is of ABI compatibility: it is easy to mess things up with structures. There is the advantage of using only one dlsym (or equivalent) with the struct, which may be much faster than using hundreds of dlsym for each function. Linux does not seem to have problems with that, but mac os X for example felt slow when I tried doing this for several thousand functions. I did not go really far on that, though. I don't really see why using a struct is cleaner, though. That's really the same thing (function pointers), in both cases the name namespace pollution will be the same, and in both cases there will be a need to generate source code.
IMHO, this is the easier and cleaner way to deal inside python with plugins written in low-level C. It does not depend explicitely on 'dlopen' stuff from your side.
Concerning the loading mechanism, I don't understand the point of using PyCObject_Import. By quickly looking at the code of the function, the plugin needs to be a python module in that case, which is not needed in our case; I don't like the fact that it is not documented either. The code for dlopen/dlclose is really short. For each platform, it is like 50 lines of code, and we have a better control on what we can do (which may be needed; for example, you want want to load symbols from a dll A, but the symbols are only in dll B, which dll A depends on; that's a problem that does not happen for python extensions, I think; I don't really know to be honest). The one thing which may have been useful by using python mechanism is the binary location detection, but that won't be needed in our case, since the plugins will always be distributed at the same time as numpy/scipy, hence we will have tight control on their location.
It even works on systems with no dynload, provided that all is put inside the python executable.
That won't be needed: the point is to have several implementations with the same functions. If you link everything statically, you will get name clashes anyway (if you want to stay portable). On those platforms, plugins should be disabled when building numpy/scipy. cheers, David

On 5/1/08, David Cournapeau <cournapeau@cslab.kecl.ntt.co.jp> wrote:
On Wed, 2008-04-30 at 16:44 -0300, Lisandro Dalcin wrote:
David, in order to put clear what I was proposing to you in previous mail regarding to implementing plugin systems for numpy, please take a look at the attached tarball.
Thanks for looking at this Lisandro.
The problem I see with the approach of struct vs direct function pointers is of ABI compatibility: it is easy to mess things up with structures.
That's a very valid point.
There is the advantage of using only one dlsym (or equivalent) with the struct, which may be much faster than using hundreds of dlsym for each function. Linux does not seem to have problems with that, but mac os X for example felt slow when I tried doing this for several thousand functions. I did not go really far on that, though. I don't really see why using a struct is cleaner, though. That's really the same thing (function pointers), in both cases the name namespace pollution will be the same, and in both cases there will be a need to generate source code.
Perhaps cleaner is not the right word. I actually believe that is far more portable, regarding the oddities of dlopening in different platforms.
Concerning the loading mechanism, I don't understand the point of using PyCObject_Import. By quickly looking at the code of the function, the plugin needs to be a python module in that case, which is not needed in our case; I don't like the fact that it is not documented either.
Well, that is the same mechanism NumPy uses for exporting its C API to extensions modules. Look at generated header '__multiarray_api.h' in function '_import_array()' in your numpy installation. This is the standard, portable, and (as Python doc says) recommended way to expose C API's from extensions modules.
The code for dlopen/dlclose is really short. For each platform, it is like 50 lines of code, and we have a better control on what we can do (which may be needed; for example, you want want to load symbols from a dll A, but the symbols are only in dll B, which dll A depends on; that's a problem that does not happen for python extensions, I think; I don't really know to be honest).
Please note that all my concerns about recommending you not to use dlopen, is just to save you from future headaches!!!. See yourself at row 17 of table here <http://www.fortran-2000.com/ArnaudRecipes/sharedlib.html> . And that table does not cover stuff like RTLD_GLOBAL flags to dlopen (or equivalent). -- Lisandro Dalcín --------------- Centro Internacional de Métodos Computacionales en Ingeniería (CIMEC) Instituto de Desarrollo Tecnológico para la Industria Química (INTEC) Consejo Nacional de Investigaciones Científicas y Técnicas (CONICET) PTLC - Güemes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594

Lisandro Dalcin wrote:
Perhaps cleaner is not the right word. I actually believe that is far more portable, regarding the oddities of dlopening in different platforms.
I am sorry, I don't understand why a struct is less sensitive to platform issues than function pointers when dynamically loading them. Could you give more details ?
Well, that is the same mechanism NumPy uses for exporting its C API to extensions modules. Look at generated header '__multiarray_api.h' in function '_import_array()' in your numpy installation. This is the standard, portable, and (as Python doc says) recommended way to expose C API's from extensions modules.
Yes, but this is not for exposing C python extensions ! This is for pure C plugins, and as such, we have different requirements; in particular, we don't need to hide api (the struct thing is partly useful because iso C does not have a namespace concept outside the compilation unit with static). Again, the use cases I have in mind are multiple implementations of the same core functions, selected at runtime (sse, sse2, mmx, etc...). Adding all the python mechanism just to load symbols does not sound like simplification to me: using the python C API is heavy, you have to start thinking about all kind of issues (ref counting, etc...).
Please note that all my concerns about recommending you not to use dlopen, is just to save you from future headaches!!!
I implemented and tested the API for mac os x, windows and dlopen which covers 99 % of our users I think. Note that we can copy the actual code in python sources if we want to.
. See yourself at row 17 of table here <http://www.fortran-2000.com/ArnaudRecipes/sharedlib.html> . And that table does not cover stuff like RTLD_GLOBAL flags to dlopen (or equivalent).
I do not pretend to know about all the platforms idiosyncrasies, but I am aware of the different aspects. In particular, note that most of the table is related to building, which I already took care of when starting numscons :) Also, I don't think we will use the system outside a few majors platforms (typically, I don'tsee anyone implementing ufunc core loops in alpha assembly for True64 :) ). cheers, David
participants (3)
-
David Cournapeau
-
David Cournapeau
-
Lisandro Dalcin