[Python-Dev] Community buildbots and Python release quality metrics

[Stephen J. Turnbull]

glyph at divmod.com writes:
 > On 01:02 am, grig.gheorghiu at gmail.com wrote:
 > >To bring my $0.02 to this discussion: the Pybots 'community buildbots'
 > >turned out largely to be a failure.
 > 
 > Let's not say it's a failure.  Let's instead say that it hasn't yet 
 > become a success :-).

+1

 > >I still haven't given up, and I hope this thread will spur project
 > >leaders into donating time, or resources, to the Pybots project.

 > I think this list is the wrong place to go to reach the people who need 
 > to be reached.  It's python core developers and other people already 
 > involved in and aware of core development.  That said I'm not sure what 
 > the *right* place is; I think your blog is syndicated on the unofficial 
 > planet python, so maybe that's a good place to start.  Sadly, the right 
 > thing to do in terms of drumming up support is to get someone interested 
 > in PR and have them go to each project individually, but that might be 
 > more effort than setting up the buildbots themselves, at least 
 > initially...

Exactly, and that's why nobody should be "bitter" about it.  The
problem is that the while overall the effort and rewards look to be
balanced in favor of the rewards, the startup costs for individuals
are quite high.

I think this *is* the place to start, though.  The project leaders
"should" be, and probably generally are, "here".  They have the
strongest interest in any individual 'bot, while Guido is quite
correct in saying python-dev can't afford to have strong interest in
all the bots.

 > However, let's say that this were tremendously successful, and lots of 
 > people start paying attention.  I think pybots.org needs to be updated 
 > to say exactly what a participant interested in python testing needs to 
 > do, beyond "here's how you set up a buildbot" (a page that is actually a 
 > daunting-looking blog post which admits it may be somewhat outdated), 
 > because setting up a buildbot might not be the only thing that the 
 > project needs.  It's one thing to tell people that they need to be 
 > helping out (and I'm sure you're right) but it's much more useful to get 
 > the message out that "we really need people to do X, Y, and Z".  One 
 > thing I will definitely commit to is that if you make a "cry for help" 
 > page, I'll blog about it to drive attention to it, and I'll encourage 
 > the other, perhaps better-read Python bloggers I know to do so as
 > well.

Two suggestions in this vein: First, I think it's established that
some but not all "red community bots" *are* of interest to Python core
development.  While I'm not aware of the technical details, I estimate
that triaging the community 'bot failures is probably similar to
reviewing bugs in the Python tracker.  Perhaps Martin van Loewis and
others who have offered the 5-for-1 review deal would be willing to
extend the definition of "review" to include initial bug reports based
on a red community bot (ie, you review the community 'bot failure and
decide it is something that should concern Python core development).
Perhaps that's not appropriate, but a similar system could be set up.

Second, something intermediate between the occasional half-hour of
triaging bugs and a full-blown PR campaign at the projects would be
documenting the criteria for reporting a failure on a community 'bot
to the Python tracker as a bug, etc.  This would also serve as a basis
for talking to project lurker who might have the odd half-hour to do
some "red bot" triaging.  (By criteria I mean the kinds of things that
Python core considers necessary breakage in new versions that
downstream must address in their own code, vs. regressions in a x.y.z
patchlevel, etc.  The kind of thing that glyph and Guido were
discussing earlier.)

 > It would be good to remove the perception that it's somebody else's 
 > problem as much as possible.

To the extent that a 'bot is running prerelease project against
prerelease Python, this is probably not very doable.  If Python is
stable and the project version is prerelease, it's the project's bug
until proven otherwise, and vice versa.  If both are stable, again
some expertise is probably needed for triage.

I guess that means that one important task is to classfy the bots in a
two-by-two matrix according to stability of project and Python
respectively.