Ondrej Certik wrote:
Hi,
we are using py.test in SymPy and unfortunately, we have some bugs in SymPy, that occurs when certain tests are run and don't occur when the tests are run in different order. It's very time consuming to determine which tests cause the problem (I did that several times by hand). It occured to me that it should be possible to enhance py.test by this facility to do it automatically.
Example:
1) This passes:
$ py.test sympy/series/tests/test_series.py -k issue409
This doesn't
2) $ py.test
One problem is, that 2) runs for several minutes, another problem is that I need to specify all tests on the command line and then deleting some of them it see if it still fails, until I narrow the issue down, usually quite nicely. Then I need to play with the "-k" parameter to try different test cases in the file, until I determine the minimal set of tests that, when executed in order, produce the error.
This can by done automatically - the py.test will be given a set of files (or just tests) that pass and another set that fails and it will narrow the problem down, by bisecting.
I would like to implement this in py.test. So I'll try my best and send you a patch. If you have some ideas, that could help me, I am interested.
Thanks a lot, Ondrej Hi Ondrej.
Sorry for late reply. This mail should go to py-dev instead or at least also. This would cause faster reply (still my fault, I was offline for a while :) Anyway - very nice idea. The long-term goal for py.test is to support this algorithm ie. in respect to revisions (give me revision that broke test X) etc. So if you go for implementing this, would be very cool for many things. Lately I didn't invest too much time in py.test developement, but I would be really happy to review your patches and/or help you. You can find help most of the time on #pylib somehow (#pypy might work as well in case noone hangs around #pylib). Cheers, fijal :.