[pypy-dev] Findings in the Cython test suite

[Stefan Behnel]

Amaury Forgeot d'Arc, 03.04.2012 22:49:
> 2012/4/3 Stefan Behnel:
>> The basic C code that gets executed in the test_append() function is simply
>>
>>    PyObject* m = PyObject_GetAttrString(L, "append");
>>    r = PyObject_CallFunctionObjArgs(m, x, NULL);
>>
>> And that's where the error gets raised (returning r==NULL). Specifically,
>> it does not enter into the actual append() method but fails before that,
>> right at the call.
> 
> The issue is with CyFunctionType, which looks like a subclass of
> PyCFunction_Type.

Yes. It's required to make C implemented functions compatible with Python
functions. Otherwise, they look and behave rather different in CPython,
especially when used as methods (e.g. when assigned to class attributes
after class creation). It's also faster for many things.


> (it's a hack: normally this type is not subclassable, booo)

Yep, that's a problem - works in CPython, though...


> L.append is such a CyFunctionType.

Ah, right - I should have tested with a class created at the Python level
as well - that works.


> Its tp_call slot is called, but this is defined to __Pyx_PyCFunction_Call
> which is #defined to PyObject_Call, which itself invokes the tp_call slot...

Interesting. Then that's the wrong thing to do in PyPy. I guess you just
put it there in your original patch because PyPy doesn't expose
PyCFunction_Call() and it seemed to be the obvious replacement.


> A solution would be to access the "base" tp_call, the one that CPython
> exposes as PyCFunction_Call.
> Unfortunately cpyext only defines one tp_call shared by all types, one
> which simply delegates to self.__call__.

Makes sense for PyPy objects.


> This means that "calling the base slot" does not work very well with cpyext.
> There is a solution though, which I implemented a long time ago for
> the tp_setattro slot.
> It can be easily expanded to [...]
> all slots but I'm a bit scared of the explosion of code this could generate.

I consider it a rather special case that Cython subtypes PyCFunction_Type,
so a general solution may not be necessary. Is there anything we can do on
Cython side? We control the type and its tp_call slot, after all. You could
also implement a fake PyCFunction_Call function specifically for this
purpose. Or even just a PyPyCFunction_Call().

OTOH, I had suggested before that PyPy could eventually learn about
Cython's function type and optimise for it. We could implement PEP 362 to
include C type information in the annotations of the signature object, and
PyPy could use that to pack and execute a direct C call to the underlying C
function. By caching the signature mapping, that could bring the
PyPy-to-Cython call overhead down to that of ctypes.

Stefan