[pypy-dev] Python FFI

[Armin Rigo]

Hi all,

Fijal and me would like to raise interest among various groups of
people about building a better ctypes replacement for Python.

The general background first, at least as far as we know it.  People
generally agree that CPython extension modules are not extremely hard
to write, but also not the ultimate solution in the matter of
interfacing with C code.  Obviously you need to know and write C code,
and learn the large CPython C API.  Moreover a CPython extension
module is rather specific to CPython or even a particular version of
it.  Even if you are lucky and your CPython extensions work on all 2.x
versions, it won't work unmodified on Python 3.x, and be slow (or not
work at all) on PyPy.  (I'm leaving Jython and IronPython out of the
picture here but it's very far from ideal for them too.)

The working alternatives nowadays are Cython and ctypes.  For Cython
you have to learn a whole new language; we can argue infinitely about
preferences, but from my point of view it's mostly just a nicer way to
write CPython C extensions using --- instead of C --- a custom
language.  About ctypes, I would say that the fact that its use
remains marginal is enough to show that its mission kind of failed.
Its problems are more than one, but most importantly the level at
which you describe the information is slightly too low - it's
ABI-level instead of API-level. It does not make a difference for some
libraries, but for some it does make a huge difference, like when you
use macros.  We would like something that can be potentially used to
wrap almost all C libraries.

So we would like to propose something (a bit more publically than the
two subprojects of PyPy attempting to do that --- did you know about
their existence?).  The goal would be to come up with a
better-designed project.  "Better designed" in my vocabulary implies
"simpler".  I'm starting in this mailing list but the idea is to try
to move it in its own project, and to develop at the same time at
least a CPython 2.7 and 3.x and PyPy implementation.

The simplest FFI we know of for a high-level language is LuaJIT's FFI.
 If you are interested, read the four pages starting at
http://luajit.org/ext_ffi.html .  A crucial design decision was to
avoid making use of tons of objects representing the C types; instead,
the user mostly specifies C types as plain strings, simplifying the
basic API a lot.

So we would like to propose some design very similar to that one.  The
basic public API looks more or less fine to me.  There are of course
preferences and topics to discuss, but let's leave that for later.
Most importantly, we would welcome a system where the C declarations
can be parsed either internally or (for more complicated cases)
externally, with the help of a real C compiler.  This might include
cases like macros, where the user would need to write (as C code) a
small function calling the macro, and have it compiled when he imports
his Python module.

Also, because it would parse strings, we can also add support for
parsing strings in the format of Cython declarations.  This would
still lower the entry barrier, and have the nice effect that the user
can sometimes copy-and-paste Cython declarations, instead of
copy-and-pasting carefully chosen pieces of a potentially messy .h
file full of custom macros (see for example /usr/include/zlib.h).  The
benefit would be that, unlike Cython, he gets to call the declared
functions directly from his regular Python code.

Opinions?  Interests?  This mail is deliberately low on details about
how we think we can do it.  Instead I'm seeking general reactions for
now and would like to move this soon to its own project, independent
of PyPy.


A bientôt,

Armin.