[pypy-dev] py.test bisection algorithm

[Ondrej Certik]

Awesome, thanks a lot. Let's do it soon. There are more things -
Kirill (another developer of SymPy) implemented a new feature:

http://code.google.com/p/sympy/issues/detail?id=389

the relevant file is here:

http://sympy.googlecode.com/svn/trunk/sympy/utilities/pytest.py

and we would like to get it integrated to py.test. It works really
well for us in the serial mode, but it doesn't yet work in the
"py.test -d" mode. And now I found it doesn't work in the boxed mode,
so it still needs some work.

Another problem: "py.test -w" doesn't work in Debian, I reported a bug
long time ago:

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=434226

But got no response. I am CCing to the Debian maintainer - if you
don't have time to maintain the package, I can take over it, I have
several python related packages in Debian already and I need the
py.test to work correctly if possible.

Another problem with "py.test -d" is that it fails for sympy, but
"py.test" only works. This is related to my last email, so by
implementing bisect at least in serial mode should help me to trace
all those nasty bugs in sympy down.  One nice feature would be for
py.test to remember the order of tests in "py.test -d" - and if it
fails, but succeeds in the serial mode, it should automatically bisect
and tell me - hey, this test works fine, but if executed just after
that test, it fails.

Ondrej

On 10/18/07, Maciek Fijalkowski <fijal at genesilico.pl> wrote:
> Ondrej Certik wrote:
> > Hi,
> >
> > we are using py.test in SymPy and unfortunately, we have some bugs in
> > SymPy, that occurs when certain tests are run and don't occur when the
> > tests are run in different order. It's very time consuming to
> > determine which tests cause the problem (I did that several times by
> > hand). It occured to me that it should be possible to enhance py.test
> > by this facility to do it automatically.
> >
> > Example:
> >
> > 1) This passes:
> >
> > $ py.test sympy/series/tests/test_series.py -k issue409
> >
> > This doesn't
> >
> > 2) $ py.test
> >
> > One problem is, that 2) runs for several minutes, another problem is
> > that I need to specify all tests on the command line and then deleting
> > some of them it see if it still fails, until I narrow the issue down,
> > usually quite nicely. Then I need to play with the "-k" parameter to
> > try different test cases in the file, until I determine the minimal
> > set of tests that, when executed in order, produce the error.
> >
> > This can by done automatically - the py.test will be given a set of
> > files (or just tests) that pass and another set that fails and it will
> > narrow the problem down, by bisecting.
> >
> > I would like to implement this in py.test. So I'll try my best and
> > send you a patch. If you have some ideas, that could help me, I am
> > interested.
> >
> > Thanks a lot,
> > Ondrej
> Hi Ondrej.
>
> Sorry for late reply. This mail should go to py-dev instead or at least
> also. This would cause faster reply (still my fault, I was offline for a
> while :)
>
> Anyway - very nice idea. The long-term goal for py.test is to support
> this algorithm ie. in respect to revisions (give me revision that broke
> test X) etc. So if you go for implementing this, would be very cool for
> many things. Lately I didn't invest too much time in py.test
> developement, but I would be really happy to review your patches and/or
> help you. You can find help most of the time on #pylib somehow (#pypy
> might work as well in case noone hangs around #pylib).
>
> Cheers,
> fijal
>
>
> :.
>
>

On 10/18/07, Maciek Fijalkowski <fijal at genesilico.pl> wrote:
> Ondrej Certik wrote:
> > <snip>
> >
> >
>
> Oh, and by the way. We've got boxed version of py.test (--box or sth,
> read --help), which forks for every test, so you're sure that every test
> is run in separate environment. (That doesn't solve your problem, but
> helps the other way around, when tests are run ok only when they're
> together).
>
> Cheers,
> fijal
>
>
> :.
>
>