[pypy-dev] Findings in the Cython test suite

[Stefan Behnel]

Hi,

the new exception handling functions enabled a lot more tests in Cython's
test suite, with still lots and lots of failures. I created tickets for a
couple of them where it's clear that the problem is in PyPy/cpyext.

The test results are collected here:

https://sage.math.washington.edu:8091/hudson/view/dev-scoder/job/cython-scoder-pypy-nightly/lastCompletedBuild/testReport/

Here's the general log (which I find easier to read in this case):

https://sage.math.washington.edu:8091/hudson/view/dev-scoder/job/cython-scoder-pypy-nightly/89/consoleFull

For C compiler errors ("CompileError" exceptions) you'll have to look in
the log at the place where the test was executed, but those are getting
few. The test results are also recorded in the log, at the end (which,
given the amount of failures, starts at about one third through the log).

Some failures are due to wrong expectations in test code, e.g. we sometimes
use the fact that small integers are cached in CPython. Others fail due to
different exception messages in the doctests. Both can be handled on Cython
side, given that CPython's own error messages aren't exactly carved in
stone either.

There is one major problem that accounts for the bulk of the test failures,
somehow related to frame handling. You can tell by the huge amount of long
traceback sequences that run into StackOverflowErrors and equivalent
RuntimeErrors. When you look closer (e.g. repeatedly searching the log for
the "classkwonlyargs" test), you will notice that the traceback refers to
more than one test, i.e. the next doctest execution somehow picks up the
frame of a previous test and continues from it. Funny enough, the frame
leaking tests that have run (and failed) before the current one appear
*below* the current test in the stack trace. This makes it likely that this
is due to the way exception stack frames are constructed in Cython. They
are only instantiated when an exception is being propagated, and then
registered using PyTraceBack_Here(). It looks similarly likely that this is
a bug in cpyext as it being due to problematic code in Cython. In case it's
PyPy's fault, maybe there's something like an off-by-one when cleaning up
the traceback frames somewhere?

Another minor thing I noticed, PyPy seems to check the object type of
keyword argument names at the caller side, whereas CPython does it as part
of the function argument unpacking. Since Cython also does it on the
function side and therefore expects its own error messages in the tests
(which mimic CPython's, including the name of the function), these tests
fail now (look for "keywords must be strings" in the test log). Not sure if
it's worth doing anything about this - checking the type at call time isn't
really wrong, just different. However, this is one of the cases where it
would be nice if PyPy simply included the function name in the error
message. The same applies to the "got multiple values for keyword argument"
error.

Apart from the frame handling bug, the remaining problems look minor enough
to say that we are really close to a point where Cython on PyPy becomes usable.

Stefan