[Cython] Test runner changes for Python/PyPy backend

[Robert Bradshaw]

On Thu, Jun 16, 2011 at 11:07 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Hi,
>
> Romain wants to start writing tests for his Python/ctypes backend branch, so
> we need to extend the test runner.
>
> We'll eventually want to merge his branch back into normal mainline, so I
> suggest actually just running the complete test suite in a new, separate
> configuration, just like we do with C and C++ now.

Makes sense.

> So we'd have a switch
> "--no-ctypes" that disables these tests, and (I assume) a new test class
> (inheriting from the normal compile test case) that properly runs them in
> plain CPython. If the implementation becomes PyPy specific, "--no-pypy" will
> make more sense, but it currently appears to target ctypes in general.
>
> I'm also considering making the backend selection a positive opt-in, i.e.
> switch from "--no-cpp" and friends to "--backends=c,cpp,ctypes". That would
> have avoided the future configuration change in all existing Jenkins test
> jobs (which will then need to disable the new backend tests).
>
> Does that fit everyone's expectations?

+1, though it might makes sense for no specification -> all options.

> There's one problem I see: the ctypes backend will likely have most
> optimisations disabled, so many (most?) of the test tree assertions will
> make no sense. I think it's best to just keep that out of the backend
> specific pipeline completely for now. We have the C/C++ test modes for that.

Probably makes more sense to disable the tree assertions in these
cases, but probably even more important to test them, as the code
paths will be different.

> And, obviously, switching from duplicating to triplicating the test cases
> will substantially increase the time for a complete test run, although a lot
> of the running time is currently spent in the C compiler more than the
> actual test code runs.

I've noticed that running the test suite already takes a while--any
ideas what we can do to improve here? As you mentioned, it's the pyx
-> c and c -> so that's taking up the bulk of the time.

- Robert