[Python-3000] Cross-interpreter FFI for Python 3000?
Phillip J. Eby
pje at telecommunity.com
Tue Aug 8 21:30:30 CEST 2006
[Note: Discussion should move to the python-3000 list]
At 11:28 AM 8/8/2006 -0700, Paul Prescod wrote:
>I'll use up a little bit of my post-conference goodwill to push a
>long-term obsession of mine...using a Python variant as the "standard"
>extension/FFI model for Python (3000). I've heard variants of this idea
>from many people I respect, some of who are cc:ed.
>I want to guage interest before doing any next steps. If it's
>preemptively -1 then I won't bother. Therefore I would like to poll the
>assembled brains about the feasibility of using something like
>RPython/Pyrex as an abstraction layer to be compiled to Py2.5 PyObjects,
>Py3000 PyObjects, JNI, .NET, ...
>Each Python implementation needs an FFI. Any Python without a C-oriented
>FFI lacks compatibility with C modules like Numeric and PIL. For this
>reason, PyPy re-invented something like Pyrex as RPython.
Just FYI, but if I understand correctly, PyPy is now using the ctypes API
for its FFI. Also, RPython is entirely unrelated to Pyrex. RPython is
Python with restrictions on how it's used, and doesn't include an FFI of
I would suggest that PyPy's use of ctypes, coupled with the inclusion of
ctypes in the Python 2.5 stdlib, means that ctypes could reasonably be
considered a defacto standard for a C FFI in Python at this point. While I
*like* Pyrex a lot and use it for most extension modules I write, it is
currently heavily tied to the CPython API, lacks many Python features that
even RPython allows, it invents its own object model for C inheritance and
imports, and has a lot of quirks due to being "not quite Python" in syntax
or semantics. These characteristics are undesirable for a
cross-interpreter FFI, IMO.
A major advantage of using ctypes as the FFI, however, is that ctypes is a
library, and thus does not require language or interpreter changes. This
means, for example, that a third party could implement a ctypes clone for
Jython or IronPython without burdening the core developers of those
> The two are
>obviously not identical but I'm looking at the core idea of a language
>that merges Python and C concepts to achieve a usable extension
>mechanism. I overheard Jim musing about something similar for
>But most important: Python 3000 needs something like Pyrex. Python 3000
>and Python 2.6, 2.7, 2.8 may be arbitrarily different internally. If the
>goal is for it to be "just a bit" incompatible then Guido's design space
>is quite constrained. If it is allowed to be massively incompatible then
>extension authors will scream. The Python 2.x line will co-exist with
>the Python 3000 line for a while, and both with co-exist with
>IronPython, Jython, PyPy and others.
It would probably be best if you catch up on the current work by the PyPy
team in this area, since my understanding is that PyPy is now able to
compile "RPython+ctypes" code to create CPython extensions in C. This
suggests that it should be possible to backends for C# and Java, because
(again, if I understand correctly) the ctypes handling is done at a
relatively high level of the translation tool chain, such that the backend
code generators don't need to know anything about ctypes. Hopefully Armin
or somebody else will jump in on this point if I'm getting something wrong
about all that.
> * it would be simpler to write competitive Python interpreters to test
>out different design ideas...one wouldn't have to worry that such an
>interpreter would be inherently a toy because of the unavailability of
Note that this is also a goal of the PyPy project, and they have many such
options now, such as "pure" GC and refcounted variants, even if you
entirely ignore the part where backends can generate code for a variety of
More information about the Python-3000