[pypy-dev] Bringing Cython and PyPy closer together

Stefan Behnel stefan_ml at behnel.de
Sat Feb 18 12:23:26 CET 2012

Maciej Fijalkowski, 18.02.2012 10:56:
> On Sat, Feb 18, 2012 at 11:48 AM, Stefan Behnel wrote:
>> Maciej Fijalkowski, 18.02.2012 10:35:
>>> On Sat, Feb 18, 2012 at 11:27 AM, Stefan Behnel wrote:
>>>> Given that XML processing is currently slower in PyPy than in CPython, I
>>>> don't think that's all that bad. Users can still switch their imports to
>>>> ElementTree if they only want to push XML out and I imagine that lxml would
>>>> still be at least as fast as ElementTree under PyPy for the way in.
>>> Are you sure actually?
>> I'm sure it's currently much slower, see here:
>> http://blog.behnel.de/index.php?p=210
>> I'm not sure the quickly patched lxml is as fast in PyPy as it is in
>> CPython, but there is certainly room for improvements, as I mentioned
>> before. A substantial part of it runs in properly hand tuned C, after all,
>> and thus doesn't need to go through cpyext or otherwise talk to PyPy.
> Can you please send me or post somewhere numbers? I'm fairly bad at
> trying to deduce them from the graph (although that doesn't change
> that the graph is very likely more readable).

You can get the code and the input files from here:


Note that this only compares the parser performance, but given how much
faster CPython is here (we are talking seconds!), it'll be hard enough for
PyPy to catch up with anything after such a head start.

> I'm not sure there are easy ways to optimize. Sure cpyext is slower
> than ctypes, but we cannot achieve much more than that. Certainly we
> cannot do unboxing (boxes might be only produced to make cpyext happy
> for example). Unless I'm misunderstanding your intentions, can you
> elaborate?

If you are referring to my comments on a faster interconnection towards
Cython, I think it should be quite easy for PyPy to bypass Cython's Python
function wrapper implementation (of type "CyFunction") to call directly
into the underlying C function, with unboxed parameters. Cython could
provide some sort of C signature introspection feature that PyPy could
analyse and optimise for.

But that's only that direction. Calling from Cython code back into PyPy
compiled code efficiently is (from what I've heard so far) likely going to
be trickier because Cython would have to know how to do that at C
compilation time at the latest, while PyPy decides about theses things at
runtime. Still, there could be a way for Cython to tell PyPy about the
signature it wants to use for a specific unboxed call, and PyPy could
generate an optimised wrapper for that and eventually a specialised
function implementation for the statically known set of argument types in a
specific call. A simple varargs approach may work here, imagine something
like this:

error = PyPy_CallFunctionWithSignature(
    func_obj_ptr, "(int, object, list, option=bint) -> int",
    i, object_ptr, list_obj_ptr, 0, int_result*)

And given that the constant signature string would be interned by the C
compiler, a simple pointer comparison should suffice for branching inside
of PyPy. That would in many cases drop the need to convert all parameters
to objects and to pack them into an argument tuple. Even keyword arguments
could often be folded into positional varargs, as I indicated above.

I can well imagine that PyPy could render such a calling convention quite

> I somehow doubt it's possible to make this run fast using cpyext
> (although there are definitely some ways). Maybe speeding up
> ElementTree would be the way if all we want to get is a fast XML
> processor? I doubt this is the case though.

cElementTree, being younger, may contain a couple of optimisations that
ElementTree lacks, but I doubt that there is still so much to get out of
it. ET has already received quite a bit of tuning in the past (although not
for PyPy). The main problem seems to be that the SAX driven callbacks from
the expat parser inject too much overhead. And any decently sized XML
document will trigger a lot of those.


More information about the pypy-dev mailing list