[core-workflow] Use buildbot to test tracker patches

[anatoly techtonik]

On Sat, Jul 5, 2014 at 3:46 PM, Ezio Melotti <ezio.melotti at gmail.com> wrote:
>
> This is how it should work:
>   1) a new "Test with buildbot" (or whatever) button is added to the
> tracker next to each patch, and developers/triagers can press it to
> trigger the build;
>   2) the buildbot builder receives the URL of the issue and of the
> patch and run all the usual steps with two differences:
>     a) a new step to run "hg import --no-c url_of_the_patch" is added
> after the cloning step;
>     b) an email with the results is sent back to the bug tracker once
> the slaves are done;

How about a separate CI service that automatically testes all patches
submitted and Roundup server just fetches their status. Similar how
Travis CI works on Github to check the status of commits in pull
requests.

> Unless I'm missing something, both these changes should be trivial.
> On the tracker side is enough to add a form with a couple of (hidden)
> fields with the URLs and a submit button.  On the buildbot side it
> should be enough to create a new builder (by copying one of the
> existing ones in the buildbot config file and adding the two steps
> mentioned above).  Putting together a mail summary with the results of
> all the slaves might require a bit of extra code, but it shouldn't be
> anything too difficult.

Yep. It is feasible.

> Some of the issues with this proposal are:
>   1) users uploading patches with malicious code -- this shouldn't be
> a problem if we only allow developers/triagers to trigger the build
> manually and tell them to review the code before doing so;

No. Make environment safe from abuse. People will be tempted to
press button ASAP, because process takes time to complete. For an
example of safe service see drone.io (open source, Go).

>   2) overloading of the slaves -- cloning/compiling/testing/cleaning
> takes quite a lot of time, and if this new feature starts to be used
> heavily it might start to give too much work for the slaves.  I don't
> know what the current load is though, so I'm not sure this will be an
> issue;

The public information on http://speed.python.org tells me that it is
still unused.

>   3) testing/implementing the feature -- I think it should be possible
> to test the feature locally before actually applying the changes on
> the real buildbot master, but unless someone is going to apply them on
> my behalf I would probably need access to the buildbot repo (is there
> a public repo?);

Making it a separate patch tracker service will allow to reuse it for
different projects and more easy customization. I don't want people
to learn Roundup, TAL and other ancient technologies just to improve
this code. If you don't mind against starting from other's code:

1. Google Chromium Try Server
http://www.chromium.org/developers/testing/try-server-usage
2. Mercurial Patch Tracker
http://patchwork.serpentine.com/project/hg/list/

>   4) determine which branch the patch should be applied to -- this
> could be done by adding a new "branch" field on the tracker; rietveld
> could use it too;

patch can be valid for multiple branches, properly exported patch
has a header with parent revision.

>   5) if the patch fails to apply there's no need to launch the build
> on all the slaves -- a single slave (or maybe even the master?) could
> check if the patch applies cleanly, and if it does trigger all the
> other slaves.  More fine-grained options could be added to the request
> form too (e.g. run only on specific slaves/operating systems);

More useful thing IMHO is to detect which commit broke which patch.

> If the idea is reasonable and I didn't miss anything I can try to find
> some time to work on it.

The idea is awesome.
-- 
anatoly t.