[Python-3000] Cross-interpreter FFI for Python 3000?
Phillip J. Eby
pje at telecommunity.com
Wed Aug 9 00:40:15 CEST 2006
At 02:45 PM 8/8/2006 -0700, Paul Prescod wrote:
>As you said elsewhere, PyPy can compile an Rpython+rctypes program to a
>C file, just as Pyrex does. So I don't understand why you see them as
Disclaimer again: I like and use Pyrex; I even built additional support for
it into setuptools. Conversely, I've only used ctypes once and am not sure
I care for its API. But as a practical matter, these preferences are
irrelevant; I will end up learning to use ctypes and liking it, and so will
everybody else, because ctypes' *dynamic* advantage will clean Pyrex's
clock at the very moment that extcompiler is as easy to use as Pyrex is now.
To summarize the differences, Pyrex is:
* A *Python-like* language, rather than Python
* Invents new inheritance/import facilities
* Imports various bits of syntax from C, including operators, pointers, etc.
* Inherently tied to the CPython API in its implementation
* Has its own system of "header" files for compile-time import/include
* Generates C code directly from Pyrex
* Cannot be executed by standard Python
PyPy's RPython+rctypes is:
* 100% Python, with certain dynamicity constraints
* Is not tied to any particular back end -- it can be translated to C, LLVM
and optimization machinery is backend-independent
* Code can be run in a normal Python interpreter if a ctypes library is
The only relationship I see between the two are some overlap in use cases,
and the letters "R", "P" and "Y" in the names. :) In particular, Pyrex
cannot be used in the interpreter, and I can't see Guido allowing Pyrex's C
syntax to infect Python-the-language, so this is likely to be a stable
barrier keeping Pyrex from having this feature, unless Greg or somebody
else decides to create a Pyrex interpreter, or perhaps an import hook to
translates Pyrex source code to Python bytecode that invokes the ctypes
(Note, by the way, that such an import hook/translator would be equally
usable in PyPy, instantly making it possible to compile Pyrex to any
backend supported by PyPy! I suggest you let that idea sink in for a
little bit, as it helps to illustrate why making ctypes the standard FFI is
the One Obvious Way To Do It.)
>There are different syntaxes, but the goals are very similar.
Well, you could say that about Python and Ruby, to name just two. Syntax
is important. :)
But that's also entirely ignoring the wide range of practical issues
alluded to above, and some more I'll dig into below.
>Pyrex uses optional type declarations (which are planned
>for Python 3000). RPython infers types from rctypes API calls (which
>will also be available in Python 3000).
They're available in Python 2.5, which means code can be written for them
today. The dynamic usability of ctypes from interpreted Python means that
Pyrex will become a historical footnote as soon as the
RPython+rctypes->CPython translator is practically usable; i.e., when it
can compete with Pyrex for code generation speed (and speed of generated
code), installability, documentation, and user community.
At that point, the advantage of being able to debug your C interface using
the interpreter's ctypes library, and then to compile the code only if/when
you need to, will be a killer advantage.
IMO, it doesn't make sense to fight that now-inevitable future, either on
behalf of Pyrex or some imagined "better" alternative; instead, we might as
well hasten that future's arrival. We can always provide better syntax for
ctypes at a later date, the way 'classmethod' and friends arrived in Python
2.2 but didn't get syntax until 2.4. If you can't wait that long, write
that import hook to turn Pyrex source into Python bytecode. :)
> Perhaps it would be better if I
>dropped the reference to Rpython and merely talked about "extcompiler"
>there tool which is very parallel to the Pyrex compiler?
I'm at a bit of a loss as to how to explain how very not-useful that
comparison is. I would suggest reading up on PyPy architecture and Pyrex
architecture a bit. From an end-user perspective you can compare them as
things that take Python-looking stuff in and spit C code out, but the devil
is definitely in the details. See also the lists I gave above.
>You make some good points about Pyrex and ctypes. I'd rather explore the
>design space after I've heard whether this design direction has the
>potential to be fruitful. I infer that you think "yes".
See http://dirtsimple.org/2005/10/children-of-lesser-python.html for what I
In that article, I highlighted the absence of a standard Python FFI as
being a stumbling block to the future evolution of the language, but noted
that PyPy would likely end up with a solution. The subsequent emergence of
ctypes as an FFI shared by CPython and PyPy has already solved this
problem; it is merely a question of recognizing the fact.
As of Python 2.5, anything else is going to have a serious uphill battle to
fight -- even if it's something like Pyrex, that at least already *exists*
and has at least *one* part-time maintainer. A brand-new FFI invented by
committee and with nobody yet stepping up to implement or maintain it,
really has no chance at all.
(This is all IMO, of course, but I find it hard to imagine how anything
else could succeed.)
More information about the Python-3000