[Python-Dev] Automated Python testing (was Re: status of development documentation)

[Brett Cannon]

On 12/25/05, Tim Peters <tim.peters at gmail.com> wrote:
> Take a look at:
>
>     http://buildbot.zope.org/
>
> That runs code from:
>
>     http://buildbot.sourceforge.net/
>
> Someone sets up a "buildbot master" (that's what the Zope URL points
> at), and then any number of people can volunteer to set up their boxes
> as "buildbot slaves".

As in some machine I might personally have left on?  That would
require a static IP which I don't know how common that will be.  But
then again I am willing to bet that the Python community is big enough
that people who do have machines that are idle enough that we should
be able to get good coverage.  Wonder if we would have to worry about
result pollution from someone who thought it was funny to send in
false negatives?

>  From time to time the buildbot master asks the
> slaves to do the checkout/build/test dance (or any other code dance
> you like), the slaves report results back to the master, and the
> master displays the slaves' results.
>
> If you look at the 2nd-leftmost column, you can see that the master
> knows when checkins have been done.  Checkins can trigger asking the
> slaves to run tests, and if the tests fail on some slave the master
> can send email saying so, including the list of checkins ("the
> blamelist") done since the last time that slave ran tests:
>
>     The guilty developer can be identified and harassed without human
> intervention.
>
> :-)
>
> This really helps at Zope Corp.  One downside is that we seem unable
> to get an in-house Windows buildbot slave to work reliably, and so far
> don't even know whether that's because of Windows, the buildbot code,
> or flakiness in our internal network.  It seems quite reliable on
> Linux, though.

Well, it is written in Python so *someone* here should either be able
to fix it or properly blame it on Windows.  =)

The idea of the PSF paying to have some machines set up to run
consistent tests has come up multiple times.  I know Neal has said he
would be willing to host the machines at his house before (but I think
this may have been before his relocation to California).  This whole
situation of going two months without knowing that a major platform is
broken shows that this is a real problem and ignoring it is probably
not a good thing.  =)

If we ask for volunteer machines we could offer to put up company or
personal names on the buildbot page of those who have volunteered CPU
cycles.  I am sure that will help motivate companies and people to
install the software on a spare machine.  Heck, I would have no
problem giving a specific company sole sponsorship kudos if they gave
us boxes that covered enough core operating systems.

Maybe this is something to bring up at the PSF meeting and to hash out
at the sprints?

-Brett