[Python-ideas] ctypes and extensions

[Jeremy Sanders]

I was following the discussion on python-dev about ctypes not being suitable 
for the standard library for linking to C libraries, and extension modules 
not suitable for alternative implementations. Ctypes is fragile when it 
comes linking to libraries whose API might change giving no kind of error. 
Extension modules are too closely linked to cpython and reference counting.

Why not have some sort of intermediate C layer to produce a library which 
can be immediatedly loaded by any python implementation?

You would write a short C program which would be linked to the library to be 
wrapped. It would automatically expose what it is wrapping to python so no 
python definitions are necessary. It would be compile-time linked to the 
library so that ABI changes are apparent at compile-time. The interface 
would be designed so that no python implementation features leak out. You'd 
need some sort of loading code (perhaps based on ctypes initially) on each 
implementation.

Imagine something like this

#include <python-wrap.h>
#include <mylib.h>
#include <math.h>

/* wrap function int mylibstrlen(char*) and expose as strlen */
WRAP_FUNCTION(mylibstrlen, "strlen", WRAP_INT, WRAP_CHARPTR);
/* wrap constant M_PI and expose as PI */
WRAP_CONST(M_PI, "PI", WRAP_DOUBLE);


If these macros exposed symbols with some sort of C++-like mangled name they 
could automatically be interpreted by python on loading the module. 
Alternatively the macros could generate some sort of table which python 
interprets.

Now you could imagine that it might be possible to write the above code 
definition in something like Python or XML and then converted to C to be 
linked against the library in question e.g.

import wrap
lib = wrap.Library("mlib", "mylib.h")
lib.definefunction("mylibstrlen", "strlen", wrap.int, wrap.charptr)
lib.writeC("out.c")
# compile, etc...

I quite like the direct C approach as it is an obvious place to write any 
additional code to convert between the C library API and a nicer API usable 
by Python, though you could have both. It would also be able to link to C++ 
symbols without python knowing about C++ name mangling, if the module was 
compiled with a C++ compiler.

Any opinions?

Jeremy