On Dec 19, 2007 4:33 PM, Raymond Hettinger wrote:

The bots are kicking-off so many false alarms that it is becoming difficult to tell whether a check-in genuinely broke a build.

At the root of the problem is a number of tests in the test suite that randomly blow-up. I now tend to automatically dismiss failures in test_logging and test_threading for example.

Yeah, certain tests need some TLC to make them more predictable and less prone to throw a failure because of some touch race condition or something on the machine was not available to make the test work. As I have stated on my blog, once I am done with importlib and the stdlib reorg I plan on working on dev docs and then attack the whole structure of the unit tests. But who knows when that will happen. =) -Brett

Raymond Hettinger wrote:

The bots are kicking-off so many false alarms that it is becoming difficult to tell whether a check-in genuinely broke a build.

It would also be nice if the checkins list only got spammed for actual compile or test failures. I'm not all that interested in getting an email just because a box got rebooted or lost its net connection for a while. In terms of checking the buildbot status page itself, it would be a lot more convenient if the "current/last build" field on the buildbot slave details page was a hyperlink to that build's results page rather than a mere number as it is now.

At the root of the problem is a number of tests in the test suite that randomly blow-up. I now tend to automatically dismiss failures in test_logging and test_threading for example.

I haven't noticed that so much as the fact that a few of the buildbots have been pretty much permanently red for quite some time. Some examples: alpha tru64: Checking the build results for this machine usually shows quite a few different errors, although the latest addition to the collection is a segfault in test_ctypes. Do we really support tru64 well enough to have it as a build slave? ppc debian unstable: test_xmlrpc failing with connection reset by peer errors. I think I've seen that on other platforms as well, but this one seems to do it nearly constantly. (I don't know enough about the structure of that test to know if a GHOP student could figure out the problem without access to a machine where the test fails consistently - might be worth a try though) MIPS debian: test_urllib2net FTP tests are failing, apparently complaining that SocketType expects two arguments rather than three. Probably a hard one to figure out without access to a machine where the test is actually failing (even my description of the problem involves some guesswork based on reading the code near where the failure occurs). I think a few of the others (such as hppa Ubuntu) also have issues. It's gotten to the point where I only pay any real attention to about half the trunk buildbots (looking at the status summary page, I would probably investigate further if sparc solaris, g4 osx 10, XP-3, XP-4 or x86 FreeBSD turned red after one of my checkins) and pretty much ignore the rest. Would it be possible to distinguish the reliable buildbots from the problematic ones in the build master? If such a distinction can be made, it might then be possible to arrange for email to be sent to the checkin list only if one of the reliable buildbots fail. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org