[Python-Dev] Snakebite, buildbot and low hanging fruit -- feedback wanted! (Was Re: SSH access against buildbot boxes)

[Trent Nelson]

On 07-Nov-10 1:55 AM, Nick Coghlan wrote:
> On Sun, Nov 7, 2010 at 3:53 AM, Giampaolo Rodolà<g.rodola at gmail.com>  wrote:
>> In such cases I would find more easy to be able to connect to the
>> machine and test myself rather than create a separate branch, commit,
>> schedule a buildbot run, wait for it to complete and see whether
>> everything is "green".
>>
>> On the other side I perfectly understand how opening up blanket ssh
>> access is not something everyone is comfortable with doing.
>> AFAICR there was someone who was setting up an evironment to solve
>> exactly this problem but I'm not sure whether this is already usable.
>
> Dealing with exactly this problem is one of the goals of the Snakebite project.
>
> As far as I know, the folks behind that project are still working on
> it - I've cc'ed Trent Nelson to see if he can provide any additional
> info on the topic.

Thanks for the ping Nick, I might have missed this otherwise.  Good 
timing, too, as Titus and I were just discussing which low hanging 
fruit/pain points Snakebite should tackle first (now that all the server 
room stuff has finally been taken care of).

Luckily, the problems that we faced 2.5 years ago when I came up with 
the idea of Snakebite are still just as ever present today ;-)

1.  Not having access to buildbots is a pain when something doesn't work 
right.  Doing dummy debug commits against trunk to try and coerce some 
more information out of a failing platform is painful.  Losing a build 
slave entirely due to a particularly hard crash and requiring the 
assistance of the owner is also super frustrating.

2.  The buildbot web interface for building non-(trunk|2.x|py3k) 
branches is also crazy unfriendly.  Per-activity branches are a great 
way to isolate development, even with Subversion, but it kinda' blows 
that you don't *really* get any feedback about how your code behaves on 
other platforms until you re-integrate your changes back into a mainline 
branch.  (I'm sure none of us have been masochistic enough to manually 
kick off individual builds for every platform via the buildbot web page 
after every commit to a non-standard branch.)

So, enter Snakebite.  We've got three racks filled with way more 
hardware than I should have ever purchased.  Ignoring the overhead of 
having to set machines up and whatnot, let's just assume that over the 
next couple of months, if there's a platform we need a stable buildbot 
for, Snakebite can provide it.  (And if we feel like bringing IRIX/MIPS 
and Tru64/Alphas back as primary platforms, we've got the hardware to do 
that, too ;-).)

Now, the fact that they're all in the one place and under my complete 
control is a big advantage, as I can start addressing some of the pain 
points that lead me down this twisted path 2.5 years ago.

I'd like to get some feedback from the development community on what 
they'd prefer.  In my mind, I could take one of the following two steps:

1.  Set up standard build slaves on all the platforms, but put something 
in place that allowed committers to ssh/mstsc in to said slaves when 
things go wrong in order to aid with debugging and/or maintaining 
general buildbot health (OK'ing modal crash dialogues on Windows, for 
example).

2.  Address the second problem of the buildbot web interface sucking for 
non-standard branches.  I'm thinking along the lines of a hack to 
buildbot, such that upon creation of new per-activity branches off a 
mainline, something magically runs in the background and sets up a 
complete buildbot view at 
python.snakebite.org/dev/buildbot/<your-branch-name>, just as if you 
were looking at a trunk buildbot page.

I'm not sure how easy the second point will be when we switch to hg; and 
I'll admit if there have been any python-dev discussions about buildbot 
once we're on hg, I've missed them.

Of course there's a third option, which is using the infrastructure I've 
mentioned to address a similarly annoying pain point I haven't thought 
of -- so feel free to mention anything else you'd like to see first 
instead of the above two things.

Titus, for example, alluded to some nifty way for a committer to push 
his local hg branch/changes somewhere, such that it would kick off 
builds on multiple platforms in the same sorta' vein as point 2, but 
able to leverage cloud resources like Amazon's EC2, not just Snakebite 
hardware.

Look forward to hearing some feedback!

Regards,

	Trent.