Guido van Rossum, 29.08.2011 04:27:
On Sun, Aug 28, 2011 at 11:23 AM, Stefan Behnel wrote:
Terry Reedy, 28.08.2011 06:58:
I think it needs a SWIG-like companion script that can write at least first-pass ctypes code from the .h header files. Or maybe it could/should use header info at runtime (with the .h bundled with a module).
From my experience, this is a "nice to have" more than a requirement. It has been requested for Cython a couple of times, especially by new users, and there are a couple of scripts out there that do this to some extent. But the usual problem is that Cython users (and, similarly, ctypes users) do not want a 1:1 mapping of a library API to a Python API (there's SWIG for that), and you can't easily get more than a trivial mapping out of a script. But, yes, a one-shot generator for the necessary declarations would at least help in cases where the API to be wrapped is somewhat large.
Hm, the main use that was proposed here for ctypes is to wrap existing libraries (not to create nicer APIs, that can be done in pure Python on top of this).
The same applies to Cython, obviously. The main advantage of Cython over ctypes for this is that the Python-level wrapper code is also compiled into C, so whenever the need for a thicker wrapper arises in some part of the API, you don't loose any performance in intermediate layers.
In general, an existing library cannot be called without access to its .h files -- there are probably struct and constant definitions, platform-specific #ifdefs and #defines, and other things in there that affect the linker-level calling conventions for the functions in the library. (Just like Python's own .h files -- e.g. the extensive renaming of the Unicode APIs depending on narrow/wide build) How does Cython deal with these?
In the CPython backend, the header files are normally #included by the generated C code, so they are used at C compilation time. Cython has its own view on the header files in separate declaration files (.pxd). Basically looks like this: # file "mymath.pxd" cdef extern from "aheader.h": double PI double E double abs(double x) These declaration files usually only contain the parts of a header file that are used in the user code, either manually copied over or extracted by scripts (that's what I was referring to in my reply to Terry). The complete 'real' content of the header file is then used by the C compiler at C compilation time. The user code employs a "cimport" statement to import the declarations at Cython compilation time, e.g. # file "mymodule.pyx" cimport mymath print mymath.PI + mymath.E would result in C code that #includes "aheader.h", adds the C constants "PI" and "E", converts the result to a Python float object and prints it out using the normal CPython machinery. This means that declarations can be reused across modules, just like with header files. In fact, Cython actually ships with a couple of common declaration files, e.g. for parts of libc, NumPy or CPython's C-API. I don't know that much about the IronPython backend, but from what I heard, it uses basically the same build time mechanisms and generates a thin C++ wrapper and a corresponding CLI part as glue layer. The ctypes backend for PyPy works different in that it generates a Python module from the .pxd files that contains the declarations as ctypes code. Then, the user code imports that normally at Python runtime. Obviously, this means that there are cases where the Cython-level declarations and thus the generated ctypes code will not match the ABI for a given target platform. So, in the worst case, there is a need to manually adapt the ctypes declarations in the Python module that was generated from the .pxd. Not worse than the current situation, though, and the rest of the Cython wrapper will compile into plain Python code that simply imports the declarations from the .pxd modules. But there's certainly room for improvements here. Stefan