[Python-Dev] Ctypes and the stdlib

[Stefan Behnel]

Guido van Rossum, 29.08.2011 04:27:
> On Sun, Aug 28, 2011 at 11:23 AM, Stefan Behnel wrote:
>> Terry Reedy, 28.08.2011 06:58:
>>> I think it needs a SWIG-like
>>> companion script that can write at least first-pass ctypes code from the .h
>>> header files. Or maybe it could/should use header info at runtime (with the
>>> .h bundled with a module).
>>
>>  From my experience, this is a "nice to have" more than a requirement. It has
>> been requested for Cython a couple of times, especially by new users, and
>> there are a couple of scripts out there that do this to some extent. But the
>> usual problem is that Cython users (and, similarly, ctypes users) do not
>> want a 1:1 mapping of a library API to a Python API (there's SWIG for that),
>> and you can't easily get more than a trivial mapping out of a script. But,
>> yes, a one-shot generator for the necessary declarations would at least help
>> in cases where the API to be wrapped is somewhat large.
>
> Hm, the main use that was proposed here for ctypes is to wrap existing
> libraries (not to create nicer APIs, that can be done in pure Python
> on top of this).

The same applies to Cython, obviously. The main advantage of Cython over 
ctypes for this is that the Python-level wrapper code is also compiled into 
C, so whenever the need for a thicker wrapper arises in some part of the 
API, you don't loose any performance in intermediate layers.


> In general, an existing library cannot be called
> without access to its .h files -- there are probably struct and
> constant definitions, platform-specific #ifdefs and #defines, and
> other things in there that affect the linker-level calling conventions
> for the functions in the library. (Just like Python's own .h files --
> e.g. the extensive renaming of the Unicode APIs depending on
> narrow/wide build) How does Cython deal with these?

In the CPython backend, the header files are normally #included by the 
generated C code, so they are used at C compilation time.

Cython has its own view on the header files in separate declaration files 
(.pxd). Basically looks like this:

     # file "mymath.pxd"
     cdef extern from "aheader.h":
         double PI
         double E
         double abs(double x)

These declaration files usually only contain the parts of a header file 
that are used in the user code, either manually copied over or extracted by 
scripts (that's what I was referring to in my reply to Terry). The complete 
'real' content of the header file is then used by the C compiler at C 
compilation time.

The user code employs a "cimport" statement to import the declarations at 
Cython compilation time, e.g.

     # file "mymodule.pyx"
     cimport mymath
     print mymath.PI + mymath.E

would result in C code that #includes "aheader.h", adds the C constants 
"PI" and "E", converts the result to a Python float object and prints it 
out using the normal CPython machinery.

This means that declarations can be reused across modules, just like with 
header files. In fact, Cython actually ships with a couple of common 
declaration files, e.g. for parts of libc, NumPy or CPython's C-API.

I don't know that much about the IronPython backend, but from what I heard, 
it uses basically the same build time mechanisms and generates a thin C++ 
wrapper and a corresponding CLI part as glue layer.

The ctypes backend for PyPy works different in that it generates a Python 
module from the .pxd files that contains the declarations as ctypes code. 
Then, the user code imports that normally at Python runtime. Obviously, 
this means that there are cases where the Cython-level declarations and 
thus the generated ctypes code will not match the ABI for a given target 
platform. So, in the worst case, there is a need to manually adapt the 
ctypes declarations in the Python module that was generated from the .pxd. 
Not worse than the current situation, though, and the rest of the Cython 
wrapper will compile into plain Python code that simply imports the 
declarations from the .pxd modules. But there's certainly room for 
improvements here.

Stefan