[Python-Dev] New work-in-progress bisection tool for the Python test suite (in regrtest)

[Victor Stinner]

On Python 2, the addition of Lib/test/bisect.py caused conflict with
Lib/bisect.py when running the Python test suite :-( I chose to rename
Lib/test/bisect.py to Lib/test/bisectcmd.py to reduce changes caused
by this new debug tool. So only in Python 2.7, you have to run:

./python -m test.bisectcmd ...

instead of

./python -m tes.bisect ...

See http://bugs.python.org/issue30843 for more information.

Victor

2017-06-16 18:05 GMT+02:00 Victor Stinner <victor.stinner at gmail.com>:
> Hi,
>
> Last weeks, I worked on a new tool to bisect failing tests because
> it's painful to bisect manually reference leaks (I remove as much code
> as possible until the code is small enough to be reviewable manually).
>
> See the bisect_test.py script attached to this issue:
> http://bugs.python.org/issue29512
>
> With the help of Louie Lu, I added new --list-cases option to "python
> -m test", so you can now list all test cases and write it into a text
> file:
>
>    ./python -m test --list-cases test_os > tests
>
> I also added a new --matchfile option, to filter tests using a text
> file which contains one pattern per line:
>
>    ./python -m test --matchfile=tests test_os
>
> fnmatch is used to match test names, so "*" joker character can be
> used in test names.
>
>
> My bisection tool takes a text file with the --matchfile format (one
> pattern per line) and creates a random subset of tests with half of
> the tests. If tests still fail, use the subset. Otherwise, create a
> new random subset. Loop until the subset contains a single test
> (configurable threshold, -n command line option).
>
> The long term plan is to integrate the bisection feature directly into regrtest.
>
>
>
> Right now, my script is hardcoded to bisect reference leak bugs, but
> it should be easy to modify it to bisect other test issues like test
> creating files without removing it ("ENV_CHANGED" failure in
> regrtest).
>
> For example, a core file is dumped when running test_subprocess on
> FreeBSD buildbots:
>
>    http://bugs.python.org/issue30448
>
> But I'm unable to reproduce the issue on my FreeBSD. It would be nice
> to be able to automate the bisection on the buildbot directly.
>
>
> --list-cases and --matchfile options are now available in 2.7, 3.5,
> 3.6 and master (3.7) branches.
>
> TODO: doctest tests are only partially supported, see:
>
>   http://bugs.python.org/issue30683
>
> Victor