[pypy-dev] cpyext performance

Stefan Behnel stefan_ml at behnel.de
Thu Aug 30 10:06:53 CEST 2012

Hi Armin,

Armin Rigo, 30.08.2012 09:10:
> On Wed, Aug 29, 2012 at 10:29 PM, Stefan Behnel wrote:
>>> Ok, so where would this have to be done? Is there a way to implement it
>>> generically in that ubiquitous space.wrap() kind of call (whatever that
>>> does internally), or would it have to be done explicitly whenever objects
>>> pass the boundary?
>> Any comments? I think this would solve a lot of problems.
> You need to do it at the level of the boundary --- space.wrap() is at
> a different level.  We can't slow it down by doing dictionary lookups
> there, because it would be a big hit of performance for the whole
> PyPy.


This may not even be that hard to do, but it's unlikely that I'll give this
a try myself. The turn-over times of building PyPy for an hour or so on a
remote machine (RAM!) for most non-trivial changes are just way too long to
make friends with the code base. Even waiting for minutes to get an
interpreted PyPy in a state where it can start building (let alone running)
Cython extensions is way beyond the seconds in my usual work flow.

> Fwiw I'm now thinking about a possible rewrite of cpyext, based on (an
> extension of) CFFI that would let us declare a C API and implement it
> in Python.  This would mean basically moving cpyext to a regular
> app-level module.  Doing so in general would be useful to write C
> extensions or wrappers for existing libraries: they would not
> necessarily be using the CPython C API but instead any custom C API
> that is most suitable for that particular extension.  For Cython, it
> would mean that you would be free to invent whatever API is most
> suitable --- starting, I'm sure, with some subset of the CPython C
> API, but possibly evolving over time towards something that offers
> better performance in the context of PyPy.

Assuming that's doable in a reasonable time frame (including proper JITting
of the API implementation to make it reasonably fast), that sounds like a
good idea. Cython already uses quite a bit of special casing for cpyext by
now, so replacing at least some of the more involved code blocks with a
more direct C-API call would be nice.

Actually the subset of the CPython C-API that Cython uses is not even all
that large. Instead, we inline and special case a lot of cases that CPython
needs to broaden its API for and tend to use the most low-level functions
only. So Cython should prove a suitable target for such an API implementation.

Funny enough, I had to change many low-level calls in CPython (PyTuple_*)
to high-level calls in cpyext (PySequence_*), both because they are
substantially faster (less useless error testing) and less likely to crash
(no borrowed reference handling).

It also sounds like this approach would actually enable a workable test
work-flow for people like me.


More information about the pypy-dev mailing list