[Cython] [GSoC] Python backend for Cython using PyPy's FFI

[Dag Sverre Seljebotn]

On 04/07/2011 05:01 PM, Romain Guillebert wrote:
> Hi
>
> I proposed the Summer of Code project regarding the Python backend for
> Cython.
>
> As I said in my proposal this would translate Cython code to Python +
> FFI code (I don't know yet if it will use ctypes or something specific
> to PyPy). PyPy's ctypes is now really fast and this will allow people to
> port their Cython code to PyPy.
>
> For the moment I've been mostly in touch with the PyPy people and they
> seem happy with my proposal.
>
> Of course I'm available for questions.

Disclaimer: I haven't read the proposal (don't have access yet but will 
soon). So perhaps the below is redundant.

This seems similar to Carl Witty's port of Cython to .NET/IronPython. An 
important insight from that project is that Cython code does NOT specify 
an ABI, only an API which requires a C compiler to make sense. That is; 
many wrapped C libraries have plenty of macros, we only require partial 
definition of struct, we only require approximate typedef's, and so on.

In the .NET port, the consequence was that rather than the original idea 
of generating C# code (with FFI specifications) was dropped, and one 
instead went with C++/CLR (which is a proper C++ compiler that really 
understands the C side on an API level, in addition to giving access to 
the .NET runtime).

There are two ways around this:

  a) In addition to Python code, generate C code that can take (the 
friendlest) APIs and probe for the ABIs (such as, for instance, getting 
the offset of each struct field from the base pointer). Of course, this 
must really be rerun for each platform/build of the wrapped library.

Essentially, you'd use Cython to generate C code that, in a target 
build, would generate Python code...

  b) Create a subset of the Cython language ("RCython" :-)), where you 
require explicit ABIs (essentially this means either disallowing "cdef 
extern from ...", or creating some new form of it). Most Cython 
extensions I know about would not work with this though, so there would 
need to be porting in each case. Ideally one should then have a similar 
mode for Cython+CPython so that one can debug with CPython as well.


Dag Sverre