[Python-Dev] Looking after the buildbots (in general)

[Steve Holden]

On 8/4/2010 12:42 PM, exarkun at twistedmatrix.com wrote:
> On 03:31 pm, barry at python.org wrote:
>> On Aug 04, 2010, at 03:15 PM, exarkun at twistedmatrix.com wrote:
>>> On 02:51 pm, barry at python.org wrote:
>>>> On Aug 04, 2010, at 11:16 AM, Antoine Pitrou wrote:
>>>>> I think the issue is that many core developers don't have the reflex
>>>>> to check buildbot state after they commit some changes (or at least
>>>>> on a regular, say weekly, basis), and so gradually the buildbots have
>>>>> a tendency to turn from green to red, one after another.
>>>>
>>>> I'd classify this as a failure of the tools, not of the developers.
>>>>> These post-commit verification steps should be proactive, and scream
>>>>> really >loud (or
>>>> even prevent future commits) until everything is green again.
>>>>> Buildbots themselves can be unstable, so this may or may not be
>>>>> workable, and >changing
>>>> any of this will take valuable volunteer time.  It's also unsexy work.
>>>
>>> How hard is it to look at a web page?
>>
>> That's not the right question :)
>>
>> The real questions are: how hard is it to remember how to find the
>> appropriate
>> web page
> 
> Oh, come on.  I don't believe this.
>> how hard is it to know which buildbots are *actually* stable enough
>> to rely on,
> 
> This is more plausible.  But it's not the tools' fault that the test
> suite has intermittent failures.  Developers choose to add new features
> or change existing ones instead of fixing bugs in existing code or
> tests.  I'd call that a developer failure.
>> how hard is it to decipher the results to know what they're
>> telling you?
> 
> Red bad, green good.
> 
> A much more plausible explanation, to me, is that most developers don't
> really care if things are completely working most of the time.  They're
> happy to push the work onto other developers and onto the release team.
> And as long as other developers let them get away with that, it's not
> likely to stop.
> 
> But perhaps the people picking up the slack here don't mind and are
> happy to keep doing it, in which case nothing needs to change.

This whole discussion seems to make it clear that the release manager
procedures are still ill-defined in certain areas. Otherwise a release
manager could proceed by reading a web page an even, heaven help us,
following specific links to ensure particular actions were taken.

But I see rules being established ("there's a language moratorium: no
changes!", "no release should be made unless the buildbots are *all*
green") and then ignored apparently on a whim. This doesn't give people
any confidence that the rules actually mean much, and I think ignoring
the latter rule can negatively affect quality.

Establishing comprehensive procedures can be as difficult as
programming, though, and requires skills that have eluded me through a
fairly lengthy technical career. So it also boils down to shortage of
manpower of a particular kind. People with programming skill would,
understandably, rather invest their time in something they are good at.

regards
 Steve


-- 
Steve Holden           +1 571 484 6266   +1 800 494 3119
DjangoCon US September 7-9, 2010    http://djangocon.us/
See Python Video!       http://python.mirocommunity.org/
Holden Web LLC                 http://www.holdenweb.com/