[Python-3000] Cross-interpreter FFI for Python 3000?

[Phillip J. Eby]

[Note: Discussion should move to the python-3000 list]

At 11:28 AM 8/8/2006 -0700, Paul Prescod wrote:
>I'll use up a little bit of my post-conference goodwill to push a
>long-term obsession of mine...using a Python variant as the "standard"
>extension/FFI model for Python (3000). I've heard variants of this idea
>from many people I respect, some of who are cc:ed.
>
>I want to guage interest before doing any next steps. If it's
>preemptively -1 then I won't bother. Therefore I would like to poll the
>assembled brains about the feasibility of using something like
>RPython/Pyrex as an abstraction layer to be compiled to Py2.5 PyObjects,
>Py3000 PyObjects, JNI, .NET, ...
>
>Rationale:
>
>Each Python implementation needs an FFI. Any Python without a C-oriented
>FFI lacks compatibility with C modules like Numeric and PIL. For this
>reason, PyPy re-invented something like Pyrex as RPython.

Just FYI, but if I understand correctly, PyPy is now using the ctypes API 
for its FFI.  Also, RPython is entirely unrelated to Pyrex.  RPython is 
Python with restrictions on how it's used, and doesn't include an FFI of 
its own.

I would suggest that PyPy's use of ctypes, coupled with the inclusion of 
ctypes in the Python 2.5 stdlib, means that ctypes could reasonably be 
considered a defacto standard for a C FFI in Python at this point.  While I 
*like* Pyrex a lot and use it for most extension modules I write, it is 
currently heavily tied to the CPython API, lacks many Python features that 
even RPython allows, it invents its own object model for C inheritance and 
imports, and has a lot of quirks due to being "not quite Python" in syntax 
or semantics.  These characteristics are undesirable for a 
cross-interpreter FFI, IMO.

A major advantage of using ctypes as the FFI, however, is that ctypes is a 
library, and thus does not require language or interpreter changes.  This 
means, for example, that a third party could implement a ctypes clone for 
Jython or IronPython without burdening the core developers of those 
interpreters.


>  The two are
>obviously not identical but I'm looking at the core idea of a language
>that merges Python and C concepts to achieve a usable extension
>mechanism. I overheard Jim musing about something similar for
>IronPython.
>
>But most important: Python 3000 needs something like Pyrex. Python 3000
>and Python 2.6, 2.7, 2.8 may be arbitrarily different internally. If the
>goal is for it to be "just a bit" incompatible then Guido's design space
>is quite constrained. If it is allowed to be massively incompatible then
>extension authors will scream. The Python 2.x line will co-exist with
>the Python 3000 line for a while, and both with co-exist with
>IronPython, Jython, PyPy and others.

It would probably be best if you catch up on the current work by the PyPy 
team in this area, since my understanding is that PyPy is now able to 
compile "RPython+ctypes" code to create CPython extensions in C.  This 
suggests that it should be possible to backends for C# and Java, because 
(again, if I understand correctly) the ctypes handling is done at a 
relatively high level of the translation tool chain, such that the backend 
code generators don't need to know anything about ctypes.  Hopefully Armin 
or somebody else will jump in on this point if I'm getting something wrong 
about all that.


>  * it would be simpler to write competitive Python interpreters to test
>out different design ideas...one wouldn't have to worry that such an
>interpreter would be inherently a toy because of the unavailability of
>third-party software

Note that this is also a goal of the PyPy project, and they have many such 
options now, such as "pure" GC and refcounted variants, even if you 
entirely ignore the part where backends can generate code for a variety of 
languages.