[pypy-dev] py.test bisection algorithm

Ondrej Certik ondrej at certik.cz
Fri Oct 12 02:29:23 CEST 2007


Hi,

we are using py.test in SymPy and unfortunately, we have some bugs in
SymPy, that occurs when certain tests are run and don't occur when the
tests are run in different order. It's very time consuming to
determine which tests cause the problem (I did that several times by
hand). It occured to me that it should be possible to enhance py.test
by this facility to do it automatically.

Example:

1) This passes:

$ py.test sympy/series/tests/test_series.py -k issue409

This doesn't

2) $ py.test

One problem is, that 2) runs for several minutes, another problem is
that I need to specify all tests on the command line and then deleting
some of them it see if it still fails, until I narrow the issue down,
usually quite nicely. Then I need to play with the "-k" parameter to
try different test cases in the file, until I determine the minimal
set of tests that, when executed in order, produce the error.

This can by done automatically - the py.test will be given a set of
files (or just tests) that pass and another set that fails and it will
narrow the problem down, by bisecting.

I would like to implement this in py.test. So I'll try my best and
send you a patch. If you have some ideas, that could help me, I am
interested.

Thanks a lot,
Ondrej



More information about the Pypy-dev mailing list