[Python-Dev] [python-committers] (Windows) buildbots on 3.x

[R. David Murray]

On Wed, 04 Aug 2010 19:53:55 +0200, =?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?= <martin at v.loewis.de> wrote:
> >>> It happens when running test_smtplib before test_smtpb:
> >>
> >> Nice! How did you work that out? I'd like to learn how to diagnose
> >> this sort of thing, because it seems to come up a lot, and I'm not
> >> much use at the moment :-)
> > 
> > I simply tried to run test_smtplib before test_smtpd.
> 
> A more deterministic (and more tedious) way is this: if you
> suspect that some failure might be due to the order of test cases,
> take a build log from the build bot where it fails, and run the tests
> in the exact same order. See whether the problem reproduces.
> 
> If it does, drop tests from the test sequence until you end up with
> the minimum number of tests that you need to run in sequence (and yes,
> I had interworkings of three test modules at some point).
> 
> Of course, educated guessing can accelerate the process.

Two bits of info to help you implement this process:

The buildbots have regrtest print out the random seed used for the
test run.  The --randseed option of regtest can be used to repeat
this run.  That's the first step, which proves that the specific order
produced the test failure.

Then you cut and paste the test list into a file, and use the
regrtest -f option to run the tests from that file, and do
your binary search by breaking up the list into pieces.

Hmm.  If we added a 'binsearch' option to regrtest, you could just
pass it the random seed and that option and off it would go...

--
R. David Murray                                      www.bitdance.com