[Python-Dev] Ctypes and the stdlib (was Re: LZMA compression support in 3.3)

[Guido van Rossum]

On Sun, Aug 28, 2011 at 11:23 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Hi,
>
> sorry for hooking in here with my usual Cython bias and promotion. When the
> question comes up what a good FFI for Python should look like, it's an
> obvious reaction from my part to throw Cython into the game.
>
> Terry Reedy, 28.08.2011 06:58:
>>
>> Dan, I once had the more or less the same opinion/question as you with
>> regard to ctypes, but I now see at least 3 problems.
>>
>> 1) It seems hard to write it correctly. There are currently 47 open ctypes
>> issues, with 9 being feature requests, leaving 38 behavior-related issues.
>> Tom Heller has not been able to work on it since the beginning of 2010 and
>> has formally withdrawn as maintainer. No one else that I know of has taken
>> his place.
>
> Cython has an active set of developers and a rather large and growing user
> base.
>
> It certainly has lots of open issues in its bug tracker, but most of them
> are there because we *know* where the development needs to go, not so much
> because we don't know how to get there. After all, the semantics of Python
> and C/C++, between which Cython sits, are pretty much established.
>
> Cython compiles to C code for CPython, (hopefully soon [1]) to Python+ctypes
> for PyPy and (mostly [2]) C++/CLI code for IronPython, which boils down to
> the same build time and runtime kind of dependencies that the supported
> Python runtimes have anyway. It does not add dependencies on any external
> libraries by itself, such as the libffi in CPython's ctypes implementation.
>
> For the CPython backend, the generated code is very portable and is
> self-contained when compiled against the CPython runtime (plus, obviously,
> libraries that the user code explicitly uses). It generates efficient code
> for all existing CPython versions starting with Python 2.4, with several
> optimisations also for recent CPython versions (including the upcoming 3.3).
>
>
>> 2) It is not trivial to use it correctly.
>
> Cython is basically Python, so Python developers with some C or C++
> knowledge tend to get along with it quickly.
>
> I can't say yet how easy it is (or will be) to write code that is portable
> across independent Python implementations, but given that that field is
> still young, there's certainly a lot that can be done to aid this.

Cythin does sound attractive for cross-Python-implementation use. This
is exciting.

>> I think it needs a SWIG-like
>> companion script that can write at least first-pass ctypes code from the .h
>> header files. Or maybe it could/should use header info at runtime (with the
>> .h bundled with a module).
>
> From my experience, this is a "nice to have" more than a requirement. It has
> been requested for Cython a couple of times, especially by new users, and
> there are a couple of scripts out there that do this to some extent. But the
> usual problem is that Cython users (and, similarly, ctypes users) do not
> want a 1:1 mapping of a library API to a Python API (there's SWIG for that),
> and you can't easily get more than a trivial mapping out of a script. But,
> yes, a one-shot generator for the necessary declarations would at least help
> in cases where the API to be wrapped is somewhat large.

Hm, the main use that was proposed here for ctypes is to wrap existing
libraries (not to create nicer APIs, that can be done in pure Python
on top of this). In general, an existing library cannot be called
without access to its .h files -- there are probably struct and
constant definitions, platform-specific #ifdefs and #defines, and
other things in there that affect the linker-level calling conventions
for the functions in the library. (Just like Python's own .h files --
e.g. the extensive renaming of the Unicode APIs depending on
narrow/wide build) How does Cython deal with these? I wonder if for
this particular purpose SWIG isn't the better match. (If SWIG weren't
universally hated, even by its original author. :-)

>> 3) It seems to be slower than compiled C extension wrappers. That, at
>> least, was the discovery of someone who re-wrote pygame using ctypes. (The
>> hope was that using ctypes would aid porting to 3.x, but the time penalty
>> was apparently too much for time-critical code.)
>
> Cython code can be as fast as C code, and in some cases, especially when
> developer time is limited, even faster than hand written C extensions. It
> allows for a straight forward optimisation path from regular Python code
> down to the speed of C, and trivial interaction with C code itself, if the
> need arises.
>
> Stefan
>
>
> [1] The PyPy port of Cython is currently being written as a GSoC project.
>
> [2] The IronPython port of Cython was written to facility a NumPy port to
> the .NET environment. It's currently not a complete port of all Cython
> features.

-- 
--Guido van Rossum (python.org/~guido)