
I was following the discussion on python-dev about ctypes not being suitable for the standard library for linking to C libraries, and extension modules not suitable for alternative implementations. Ctypes is fragile when it comes linking to libraries whose API might change giving no kind of error. Extension modules are too closely linked to cpython and reference counting. Why not have some sort of intermediate C layer to produce a library which can be immediatedly loaded by any python implementation? You would write a short C program which would be linked to the library to be wrapped. It would automatically expose what it is wrapping to python so no python definitions are necessary. It would be compile-time linked to the library so that ABI changes are apparent at compile-time. The interface would be designed so that no python implementation features leak out. You'd need some sort of loading code (perhaps based on ctypes initially) on each implementation. Imagine something like this #include <python-wrap.h> #include <mylib.h> #include <math.h> /* wrap function int mylibstrlen(char*) and expose as strlen */ WRAP_FUNCTION(mylibstrlen, "strlen", WRAP_INT, WRAP_CHARPTR); /* wrap constant M_PI and expose as PI */ WRAP_CONST(M_PI, "PI", WRAP_DOUBLE); If these macros exposed symbols with some sort of C++-like mangled name they could automatically be interpreted by python on loading the module. Alternatively the macros could generate some sort of table which python interprets. Now you could imagine that it might be possible to write the above code definition in something like Python or XML and then converted to C to be linked against the library in question e.g. import wrap lib = wrap.Library("mlib", "mylib.h") lib.definefunction("mylibstrlen", "strlen", wrap.int, wrap.charptr) lib.writeC("out.c") # compile, etc... I quite like the direct C approach as it is an obvious place to write any additional code to convert between the C library API and a nicer API usable by Python, though you could have both. It would also be able to link to C++ symbols without python knowing about C++ name mangling, if the module was compiled with a C++ compiler. Any opinions? Jeremy

Paul Colomiets wrote:
So what's the problem with cython? It solves your problem well.
I think it works for the cpython case outside the standard library. However, my idea was a simple binary ABI which would be very easy for any python implementation to use, and so would would be suitable for wrapping libraries in the standard library. Maybe there are cython implementations coming soon available for pypy, jython, ironpython, etc...? These would still have to compile to C code to get build time API checking, which is one of the ctypes issues. Jeremy

Jeremy Sanders, 28.08.2011 12:35:
Maybe there are cython implementations coming soon available for pypy, jython, ironpython, etc...?
There are ports to PyPy and IronPython being written, yes. Not sure if a Jython port would be all that competitive, because there is quite some overhead involved in the JNI bridge (AFAIR - this may have changed since the last time I considered it). But it would still be interesting for many projects even with a noticeable performance penalty, I guess.
These would still have to compile to C code to get build time API checking, which is one of the ctypes issues.
Well, the PyPy port actually uses ctypes as a backend for C interaction, so it's not really PyPy specific, but it's supposed to work particularly well on PyPy. The IronPython port uses a mixture of C++ and CLR code, but I haven't been involved with it in any way. It was written to port NumPy to .NET, and that reportedly worked out quite well. Stefan

Paul Colomiets wrote:
So what's the problem with cython? It solves your problem well.
I think it works for the cpython case outside the standard library. However, my idea was a simple binary ABI which would be very easy for any python implementation to use, and so would would be suitable for wrapping libraries in the standard library. Maybe there are cython implementations coming soon available for pypy, jython, ironpython, etc...? These would still have to compile to C code to get build time API checking, which is one of the ctypes issues. Jeremy

Jeremy Sanders, 28.08.2011 12:35:
Maybe there are cython implementations coming soon available for pypy, jython, ironpython, etc...?
There are ports to PyPy and IronPython being written, yes. Not sure if a Jython port would be all that competitive, because there is quite some overhead involved in the JNI bridge (AFAIR - this may have changed since the last time I considered it). But it would still be interesting for many projects even with a noticeable performance penalty, I guess.
These would still have to compile to C code to get build time API checking, which is one of the ctypes issues.
Well, the PyPy port actually uses ctypes as a backend for C interaction, so it's not really PyPy specific, but it's supposed to work particularly well on PyPy. The IronPython port uses a mixture of C++ and CLR code, but I haven't been involved with it in any way. It was written to port NumPy to .NET, and that reportedly worked out quite well. Stefan
participants (3)
-
Jeremy Sanders
-
Paul Colomiets
-
Stefan Behnel