[pypy-dev] py.test bisection algorithm

[Maciek Fijalkowski]

Ondrej Certik wrote:
> Hi,
>
> we are using py.test in SymPy and unfortunately, we have some bugs in
> SymPy, that occurs when certain tests are run and don't occur when the
> tests are run in different order. It's very time consuming to
> determine which tests cause the problem (I did that several times by
> hand). It occured to me that it should be possible to enhance py.test
> by this facility to do it automatically.
>
> Example:
>
> 1) This passes:
>
> $ py.test sympy/series/tests/test_series.py -k issue409
>
> This doesn't
>
> 2) $ py.test
>
> One problem is, that 2) runs for several minutes, another problem is
> that I need to specify all tests on the command line and then deleting
> some of them it see if it still fails, until I narrow the issue down,
> usually quite nicely. Then I need to play with the "-k" parameter to
> try different test cases in the file, until I determine the minimal
> set of tests that, when executed in order, produce the error.
>
> This can by done automatically - the py.test will be given a set of
> files (or just tests) that pass and another set that fails and it will
> narrow the problem down, by bisecting.
>
> I would like to implement this in py.test. So I'll try my best and
> send you a patch. If you have some ideas, that could help me, I am
> interested.
>
> Thanks a lot,
> Ondrej
Hi Ondrej.

Sorry for late reply. This mail should go to py-dev instead or at least 
also. This would cause faster reply (still my fault, I was offline for a 
while :)

Anyway - very nice idea. The long-term goal for py.test is to support 
this algorithm ie. in respect to revisions (give me revision that broke 
test X) etc. So if you go for implementing this, would be very cool for 
many things. Lately I didn't invest too much time in py.test 
developement, but I would be really happy to review your patches and/or 
help you. You can find help most of the time on #pylib somehow (#pypy 
might work as well in case noone hangs around #pylib).

Cheers,
fijal


:.