[pytest-dev] Advice for writing a failure bisection plugin

[Craig de Stigter]

Hi there

I'm really enjoying using pytest (and currently porting a large OSS project
<https://github.com/OSGeo/gdal/pull/963> to it).

The project I'm porting has a large number of tests that depend on other
tests. In particular, combining certain tests in the same run as other
tests appears to cause segfaults sometimes, or double-free errors at
process exit (particularly annoying since it's very hard to pinpoint what
caused it)

My current workflow for finding these dependencies involves:
 * run all the tests, note a failure.
 * run half the tests, see if it still fails. If not, run the other half
 * run half of *those *tests, etc.

This is pretty similar to the `git bisect` workflow when debugging when a
bug was introduced in a codebase.

I'd like to find or write a similar plugin for pytest, that would
repeatedly run teh tests with smaller and smaller subsets of tests until
the dependent tests are identified.

Since sometimes the failure mode I'm tracking is a crash at the end of the
process (after all tests have succeeded), I will need to run tests as a
subprocess of the main process (so I can restart it when it fails.)

The simplest implementation I can think of is a plugin that completely
takes over the parent pytest process, runs pytest itself repeatedly in a
`subprocess.check_call()`, then calls `sys.exit(0)` when done. This sounds
pretty wrong but it could be a good starting point?

However, I note the excellent pytest-xdist plugin is good at spawning
worker subprocesses and sending tests to them, and rather than re-invent
the wheel I wondered if I could make my plugin depend on xdist, and
*control* the xdist workers (e.g. run one worker with all the tests; if it
crashes, run a new worker with half the tests, etc).

Could someone knowledgeable in this area give me some pointers on whether
this is a good idea; what exactly is possible; and where to look for code
to re-use? Particularly if you know of any plugins that might do similar
things that I could take inspiration from.

Thanks for reading :)

Craig de Stigter
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pytest-dev/attachments/20181103/bff3a6f2/attachment.html>