[pypy-dev] Bringing Cython and PyPy closer together

[Stefan Behnel]

Stefan Behnel, 15.02.2012 12:32:
> The current state of the discussion seems to be that PyPy provides ways to
> talk to C code, but nothing as complete as CPython's C-API in the sense
> that it allows efficient two-way communication between C code and Python
> objects. Thus, we need to either improve this or look for alternatives.
> 
> In order to get us more focussed on what can be done and what the
> implications are, so that we may eventually be able to decide what should
> be done, I started a Wiki page for a PyPy backend CEP (Cython Enhancement
> Proposal).
> 
> http://wiki.cython.org/enhancements/pypy

The discussion so far makes me rather certain that the most promising
short-term solution is to make Cython generate C code that PyPy's cpyext
can handle. This should get us a rather broad set of running code somewhat
quickly, while requiring the least design-from-scratch type of work in a
direction that does not yet allow us to see if it will really make existing
code work or not.

On top of the basic cpyext interface, it should then be easy to implement
obvious optimisations like native C level calls to Cython wrapped functions
from PyPy (and potentially also the other direction) and otherwise avoid
boxing/unboxing where unnecessary, e.g. for builtins. After all, it all
boils down to native code at some point and I'm sure there are various ways
to exploit that.

Also, going this route will help both projects to get to know each other
better. I think that's a required basis if we really aim for designing a
more high-level interface at some point.

The first steps I see are:

- get Cython's test suite to run on PyPy
- analyse the failing tests and decide how to fix them
- adapt the Cython generated C code accordingly, special casing for PyPy
  where required

Here is a "getting started" guide that tells you how testing works in Cython:

http://wiki.cython.org/HackerGuide

Once we have the test suite runnable, we can set up a PyPy instance on our
CI server to get feed-back on any advances.

https://sage.math.washington.edu:8091/hudson/

So, any volunteers or otherwise interested parties to help in getting this
to work? Anyone in for financial support?

Stefan