[Python-Dev] Matrix testing against CPython releases (was: Re: Assertion in _PyManagedBuffer_FromObject())

[Stefan Behnel]

Stefan Krah, 04.03.2012 12:33:
> Thomas Wouters wrote:
>> Do you test against pydebug builds of Python, or otherwise a build that
>> actually enables asserts?
> 
> Yes, I do (and much more than that):
> 
> http://hg.python.org/features/cdecimal/file/40917e4b51aa/Modules/_decimal/python/runall-memorydebugger.sh
> http://hg.python.org/features/cdecimal/file/40917e4b51aa/Modules/_decimal/python/runall.bat
> 
> It's automated, so it's not a big deal. You get 100% coverage, with and without
> threads, all machine configurations, pydebug, refleaks, release build and
> release build with Valgrind.

Same for Cython. We continuously test against the debug builds of all
CPython branches since 2.4 (the oldest we support), as well as the latest
developer branch, using both our own test suite and Python's regression
test suite.

https://sage.math.washington.edu:8091/hudson/

https://sage.math.washington.edu:8091/hudson/view/python/

BTW, I can warmly recommend Jenkins' matrix builds for this kind of
compatibility testing. Here's an example:

https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests/

Basically, you write a build script and Jenkins configures it using
environment variables that define the specific setup, e.g. Python 2.7 with
C backend. It'll then run all combinations in parallel (optionally
filtering out nonsense combinations or preferring combinations that should
fail the build early) and present the results both as an aggregated view
and in separate per-setup views. It also uses file hashes to remember where
the dependencies came from, e.g. which build created the CPython
installation that was used for testing, so that you can jump right to the
build log of the dependency to check for relevant changes that may have
triggered a test failure. Oh, and you can just copy such a job config to
set up a separate set of test jobs for a developer's branch, for example. A
huge help in distributed developer settings, or when you want to get a GSoC
student up and running.

Stefan