[Python-Dev] Status of the Buildbot fleet and related bugs

[R. David Murray]

The buildbot waterfall is much greener now.  Thanks to all who have
contributed to making it so (and it hasn't just been Mark and Antoine
and I, though we've been the most directly active (and yes, Mark, you
did contribute several fixes!)).

The 'stable builders' fleet is green now except for:

     (1) issue 7269: occasional 2.6/trunk bsddb3 failures on windows
     (2) issue 6748: random 3.1/3.x failures on most buidbots.
     (3) the klose-debian-ppc builder being offline

Of these, (2) is by _far_ the biggest issue, and the one that causes the
most flap (success-failure-success).  And flap is the thing that most
harms the buildbot usefulness.

Anyone who wants to debug this on a platform where it is consistently
reproducible please email me your public key and I'll give you a shell
account on my buildbot (Antoine already has one).

In the 'unstable' builder fleet, Antoine's new builder seems to be
stable across the board, while mine fails consistently on 3.1 and 3.x
because of the test_telnetlib bug.  Thomas Heller's OS X buildbot is
consistently having lots of test failures (the same ones each time
I've checked).  The master claims the klose-debian-alpha buildbot
doesn't know about branches, which is odd since it was working not
too long ago.  The remaining buildslaves appear to have been offline
for some time.

Open issues here are:

     (1) issue 3864: FreeBSD testing hangs consistently.  According to the
         ticket this is a FreeBSD bug fixed in 6.4, so an OS upgrade
         on the buildslave would probably solve it.
     (2) issue 4970: consistent signal 32 error on the norwitz-x86 Gentoo
         buildslave in 3.1 and 3.x.  This may be due to the box
         running an old threading library, but it does make one wonder what
         changed in 3.x that exposed it.

Another issue that I've seen on the buildbots but that doesn't
seem to be showing up right now (is it fixed?) is issue 7251, which
Mark is working on.

So, overall I think the buildbot fleet is in good shape, and if
we can nail issue 6748 I think it will be back to being an
important resource for sanity checking our checkins.

By the way, Georg set up the IRC interface on the #python-dev channel,
so you can hang out there if you want to get realtime reports of which
buildbots have going from success to failure and vice versa.

--David