[Python-Dev] Spurious Buildbot Reports

[Titus Brown]

On Wed, Dec 19, 2007 at 05:58:35PM -0800, Brett Cannon wrote:
-> On Dec 19, 2007 4:33 PM, Raymond Hettinger <python at rcn.com> wrote:
-> > The bots are kicking-off so many false alarms that it is becoming difficult to tell whether a check-in genuinely broke a build.
-> >
-> > At the root of the problem is a number of tests in the test suite that randomly blow-up.  I now tend to automatically dismiss failures in test_logging and test_threading for example.
-> 
-> Yeah, certain tests need some TLC to make them more predictable and
-> less prone to throw a failure because of some touch race condition or
-> something on the machine was not available to make the test work.
-> 
-> As I have stated on my blog, once I am done with importlib and the
-> stdlib reorg I plan on working on dev docs and then attack the whole
-> structure of the unit tests.
-> 
-> But who knows when that will happen.  =)

OK, but this isn't really a structural problem, right? This is
non-determinism in certain tests ;)

How broken are these tests?  Can we point a 17-yr-old at them and tell
them to fix 'em (yes think "GHOP")?  (If the problem is reproducible on a
1-in-10 basis then I think the answer is "yes".)

And are test_threading and test_logging the two that need the most
work?

Hmm, perhaps a good starting task would be to run the tests 10-100
times, in random order, on a single machine, to get a statistical
picture of of the problem.

cheers,
--titus
(always on the lookout for core => GHOP tasks)