[Python-Dev] Ctypes and the stdlib (was Re: LZMA compression support in 3.3)

[Stefan Behnel]

Hi,

sorry for hooking in here with my usual Cython bias and promotion. When the 
question comes up what a good FFI for Python should look like, it's an 
obvious reaction from my part to throw Cython into the game.

Terry Reedy, 28.08.2011 06:58:
> Dan, I once had the more or less the same opinion/question as you with
> regard to ctypes, but I now see at least 3 problems.
>
> 1) It seems hard to write it correctly. There are currently 47 open ctypes
> issues, with 9 being feature requests, leaving 38 behavior-related issues.
> Tom Heller has not been able to work on it since the beginning of 2010 and
> has formally withdrawn as maintainer. No one else that I know of has taken
> his place.

Cython has an active set of developers and a rather large and growing user 
base.

It certainly has lots of open issues in its bug tracker, but most of them 
are there because we *know* where the development needs to go, not so much 
because we don't know how to get there. After all, the semantics of Python 
and C/C++, between which Cython sits, are pretty much established.

Cython compiles to C code for CPython, (hopefully soon [1]) to 
Python+ctypes for PyPy and (mostly [2]) C++/CLI code for IronPython, which 
boils down to the same build time and runtime kind of dependencies that the 
supported Python runtimes have anyway. It does not add dependencies on any 
external libraries by itself, such as the libffi in CPython's ctypes 
implementation.

For the CPython backend, the generated code is very portable and is 
self-contained when compiled against the CPython runtime (plus, obviously, 
libraries that the user code explicitly uses). It generates efficient code 
for all existing CPython versions starting with Python 2.4, with several 
optimisations also for recent CPython versions (including the upcoming 3.3).


> 2) It is not trivial to use it correctly.

Cython is basically Python, so Python developers with some C or C++ 
knowledge tend to get along with it quickly.

I can't say yet how easy it is (or will be) to write code that is portable 
across independent Python implementations, but given that that field is 
still young, there's certainly a lot that can be done to aid this.


> I think it needs a SWIG-like
> companion script that can write at least first-pass ctypes code from the .h
> header files. Or maybe it could/should use header info at runtime (with the
> .h bundled with a module).

 From my experience, this is a "nice to have" more than a requirement. It 
has been requested for Cython a couple of times, especially by new users, 
and there are a couple of scripts out there that do this to some extent. 
But the usual problem is that Cython users (and, similarly, ctypes users) 
do not want a 1:1 mapping of a library API to a Python API (there's SWIG 
for that), and you can't easily get more than a trivial mapping out of a 
script. But, yes, a one-shot generator for the necessary declarations would 
at least help in cases where the API to be wrapped is somewhat large.


> 3) It seems to be slower than compiled C extension wrappers. That, at
> least, was the discovery of someone who re-wrote pygame using ctypes. (The
> hope was that using ctypes would aid porting to 3.x, but the time penalty
> was apparently too much for time-critical code.)

Cython code can be as fast as C code, and in some cases, especially when 
developer time is limited, even faster than hand written C extensions. It 
allows for a straight forward optimisation path from regular Python code 
down to the speed of C, and trivial interaction with C code itself, if the 
need arises.

Stefan


[1] The PyPy port of Cython is currently being written as a GSoC project.

[2] The IronPython port of Cython was written to facility a NumPy port to 
the .NET environment. It's currently not a complete port of all Cython 
features.