[pypy-dev] Bringing Cython and PyPy closer together

[Stefan Behnel]

Stefan Behnel, 22.02.2012 21:39:
> adding to the list.
> 
>> Some more errors that I see in the logs up to that point, which all hint at
>> missing bits of the C-API implementation:
>>
>> specialfloatvals.c:490: error: ‘Py_HUGE_VAL’ undeclared
> 
> CPython simply defines this as
> 
> #ifndef Py_HUGE_VAL
>   #define Py_HUGE_VAL HUGE_VAL
> #endif
> 
> to allow users to override it on buggy platforms.
> 
>> buffmt.c:2589: warning: implicit declaration of function ‘PyUnicode_Replace’
> 
> Some more missing parts of the C-API:
> 
> - PyUnicode_Tailmatch
> 
> - PyFrozenSet_Type

- PyUnicode_GetMax

- the Unicode character type functions: Py_UNICODE_ISTITLE(),
Py_UNICODE_ISALPHA(), Py_UNICODE_ISDIGIT(), Py_UNICODE_ISNUMERIC()

Our exec/eval implementation is broken because these are missing:

PyCode_Check(), PyCode_GetNumFree(), PyEval_EvalCode(),
PyEval_MergeCompilerFlags(), PyCF_SOURCE_IS_UTF8, PyRun_StringFlags()

I doubt that they will be all that trivial to implement, so I can live with
not having them for a while. Code that uses exec/eval will just fail to
compile for now.

I had to disable the following tests from Cython's test suite because they
either crash PyPy or at least corrupt it in a way that infects subsequent
tests:

bufaccess
cascadedassignment
control_flow_except_T725
exarkun
exceptions_nogil
extended_unpacking_T235
fused_def
fused_cpdef
literal_lists
memoryview
moduletryexcept
purecdef
property_decorator_T593
setjmp
special_methods_T561_py2
tupleassign
tryexcept
tuple_unpack_string
type_slots_nonzero_bool

With those taken out, I get an otherwise complete test run:

https://sage.math.washington.edu:8091/hudson/view/dev-scoder/job/cython-scoder-pypy-nightly/23/

Here are the test results:

https://sage.math.washington.edu:8091/hudson/view/dev-scoder/job/cython-scoder-pypy-nightly/23/testReport/

It obviously runs longer than a CPython run (22 vs. 7 minutes), even though
the runtime is normally dominated by the C compiler runs. However, having
learned a bit about the difficulties that PyPy has in emulating the C-API,
I'm actually quite impressed how much of this just works at all. Well done.

And last but not least, over on python-dev, MvL proposed these two simple
functions for accessing the tstate->exc_* fields:

- PyErr_GetExcInfo(PyObject** type, PyObject** value, PyObject** tb)

- PyErr_SetExcInfo(PyObject* type, PyObject* value, PyObject* tb)

http://thread.gmane.org/gmane.comp.python.devel/129787/focus=129792

Makes sense to me. Getting those in would fix our generator/coroutine
implementation, amongst other things.

Stefan