[Python-Dev] Possible language summit topic: buildbots

["Martin v. Löwis"]

Mark Dickinson wrote:
> Would it be worth spending some time discussing the buildbot situation
> at the PyCon 2010 language summit?  In the past, I've found the
> buildbots to be an incredibly valuable resource;  especially when
> working with aspects of Python or C that tend to vary significantly
> from platform to platform (for me, this usually means floating-point,
> and platform math libraries, but there are surely many other things it
> applies to).  But more recently there seem to have been some
> difficulties keeping a reasonable number of buildbots up and running.
> A secondary problem is that it can be awkward to debug some of the
> more obscure test failures on buildbots without having direct access
> to the machine.  From conversations on IRC, I don't think I'm alone in
> wanting to find ways to make the buildbots more useful.

These are actually two issues:
a) where do we get buildbot hardware and operators?
b) how can we reasonably debug problems occurring on buildbots

For a), I think we can solve this only by redundancy, i.e. create
more build slaves, hoping that a sufficient number would be up at
any point in time.

So: what specific kinds of buildbots do you think are currently lacking?
A call for volunteers will likely be answered quickly.

> So the question is: how best to invest time and possibly money to
> improve the buildbot situation (and as a result, I hope, improve the
> quality of Python)?

I don't think money will really help (I'm skeptical in general that
money helps in open source projects). As for time: "buildbot scales",
meaning that the buildbot slave admins will all share the load, being
responsible only for their own slaves.

On the master side: would you be interested in tracking slave admins?

> What could be done to make maintenance of build
> slaves easier?

This is something that only the slave admins can answer. I don't think
it's difficult - it's just that people are really unlikely to contribute
to the same thing over a period of five years at a steady rate. So we
need to make sure to find replacements when people drop out.

> Or to encourage interested third parties to donate
> hardware and time?

Again: I think a call for volunteers would do (Steve, if you are
reading this, please hold back just a few days before actually
making such a call :-)

> Are there good alternatives to Buildbot that might
> make a difference?

I think people have started working on such a thing. There are
certainly alternatives; I'm fairly skeptical that they are *good*
alternatives (but then, I'm the one who set up the buildbot
installation in the first place).

> What do other projects do?

I think that's really difficult to compare, since their testing
often has a very different scope. I think CruiseControl is widely
used.

> These are probably the wrong questions;  I'm hoping that a discussion
> would help produce the right questions, and possibly some answers.

I think these are good questions - just not for the summit. Setting
up such a system is, conceptually, easy. It's also just a little work
to set it up initially; the difficult part then is to keep it running
(and no, a system where anybody can just post test results at any time
without prior registration is *still* difficult to keep running).

The source of the problem is that such a system can degrade without
anybody taking action. If the web server's hard disk breaks down,
people panic and look for a solution quickly. If the source control
is down, somebody *will* "volunteer" to fix it. If the automated
build system produces results less useful, people will worry, but
not take action.

Regards,
Martin