[Cython] Cython on PyPy is (mostly) functional

[Stefan Behnel]

Hi,

Cython is now mostly functional on the latest PyPy nightly builds.

https://sage.math.washington.edu:8091/hudson/job/cython-scoder-pypy-nightly/

There are still crashers and a number of tests are disabled for that
reason, but the number of passing tests makes it fair to consider it usable
(if it works, it works). Most of the failing tests are due to bugs in
PyPy's cpyext (the C-API compatibility layer), and most of the crashers as
well. Some doctests just fail due to different exception messages, PyPy has
a famous history of that. Also, basically any test for __dealloc__()
methods is bound to fail because PyPy's garbage collector has no way of
making sure that they have been called at a given point.

Still, it's worth taking another look through the test results, because
Cython can sometimes work around problems in cpyext more easily than it
would be to really fix them on PyPy side. One major source of problems are
borrowed references, because PyPy cannot easily guarantee that they stay
alive in C space when all owned references are in Python space. Their
memory management can move objects around, for example, and cpyext can't
block that because it can't know when a borrowed reference dies. That means
that something as ubiquitous in Cython as PyTuple_GET_ITEM() may not always
work well, and is also far from being as fast in cpyext as in CPython.

The crashers can be seen in a forked complete test run in addition to the
stripped test job above:

https://sage.math.washington.edu:8091/hudson/job/cython-scoder-pypy-nightly-safe/lastBuild/consoleFull

Interestingly, specifically the new features, i.e. memory views and fused
functions, current account for a number of crashes. Likely not that hard to
fix on our side, but needs investigation.

I put up a pull request with the current changes:

https://github.com/cython/cython/pull/110

Nothing controversial, really, but worth looking through to see what kind
of problems to expect when writing code for PyPy. Once these are merged,
I'll copy over my PyPy test jobs in Jenkins to the cython-devel jobs. That
means that you should start taking a look at those after pushing your
changes and get used to either writing your code in a portable way or
providing a fallback code path for PyPy. Use the
CYTHON_COMPILING_IN_(PYPY|CPYTHON) macros for that.

At some point, it'll become interesting to revisit these specialisations
and to start benchmarking them in PyPy. However, given that PyPy's entire
cpyext hasn't received any major optimisation yet, it'd be a waste of time
to optimise for it on our side right now. The emphasis is clearly on making
it safely work at all.

Stefan