[Python-Dev] New work-in-progress bisection tool for the Python test suite (in regrtest)

Victor Stinner victor.stinner at gmail.com
Fri Jun 16 16:35:48 EDT 2017

2017-06-16 19:00 GMT+02:00 Brett Cannon <brett at python.org>:
> Great idea! I had thought about doing something similar to this years ago to
> help diagnose when a test passes in isolation but somehow fails due to one
> of the bazillion of other tests that ran previously (which can be hard to
> narrow down if the failure is not early on in a test run).

Yeah, sometimes a test only fails if it is run after other tests. The
well known issue of side effects ;-)

Each time that my script bisect_test.py script succeed to reduce the
number of tests, it writes the test list on disk. So you can interrupt
it the script is stuck and you consider that the output is small
enough to a manual review.


> On Fri, 16 Jun 2017 at 09:06 Victor Stinner <victor.stinner at gmail.com>
> wrote:
>> Hi,
>> Last weeks, I worked on a new tool to bisect failing tests because
>> it's painful to bisect manually reference leaks (I remove as much code
>> as possible until the code is small enough to be reviewable manually).
>> See the bisect_test.py script attached to this issue:
>> http://bugs.python.org/issue29512
>> With the help of Louie Lu, I added new --list-cases option to "python
>> -m test", so you can now list all test cases and write it into a text
>> file:
>>    ./python -m test --list-cases test_os > tests
>> I also added a new --matchfile option, to filter tests using a text
>> file which contains one pattern per line:
>>    ./python -m test --matchfile=tests test_os
>> fnmatch is used to match test names, so "*" joker character can be
>> used in test names.
>> My bisection tool takes a text file with the --matchfile format (one
>> pattern per line) and creates a random subset of tests with half of
>> the tests. If tests still fail, use the subset. Otherwise, create a
>> new random subset. Loop until the subset contains a single test
>> (configurable threshold, -n command line option).
>> The long term plan is to integrate the bisection feature directly into
>> regrtest.
>> Right now, my script is hardcoded to bisect reference leak bugs, but
>> it should be easy to modify it to bisect other test issues like test
>> creating files without removing it ("ENV_CHANGED" failure in
>> regrtest).
>> For example, a core file is dumped when running test_subprocess on
>> FreeBSD buildbots:
>>    http://bugs.python.org/issue30448
>> But I'm unable to reproduce the issue on my FreeBSD. It would be nice
>> to be able to automate the bisection on the buildbot directly.
>> --list-cases and --matchfile options are now available in 2.7, 3.5,
>> 3.6 and master (3.7) branches.
>> TODO: doctest tests are only partially supported, see:
>>   http://bugs.python.org/issue30683
>> Victor
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/brett%40python.org

More information about the Python-Dev mailing list