[core-workflow] Use buildbot to test tracker patches

[Ezio Melotti]

On Sat, Jul 5, 2014 at 5:45 PM, anatoly techtonik <techtonik at gmail.com> wrote:
> On Sat, Jul 5, 2014 at 3:46 PM, Ezio Melotti <ezio.melotti at gmail.com> wrote:
>>
>> This is how it should work:
>>   1) a new "Test with buildbot" (or whatever) button is added to the
>> tracker next to each patch, and developers/triagers can press it to
>> trigger the build;
>>   2) the buildbot builder receives the URL of the issue and of the
>> patch and run all the usual steps with two differences:
>>     a) a new step to run "hg import --no-c url_of_the_patch" is added
>> after the cloning step;
>>     b) an email with the results is sent back to the bug tracker once
>> the slaves are done;
>
> How about a separate CI service that automatically testes all patches
> submitted and Roundup server just fetches their status. Similar how
> Travis CI works on Github to check the status of commits in pull
> requests.
>

That would be better, but I'm not able to do that currently (due to
lack of knowledge/time).
My suggestion shouldn't take more than a weekend of work, so I might
give it a shot.

>> Unless I'm missing something, both these changes should be trivial.
>> On the tracker side is enough to add a form with a couple of (hidden)
>> fields with the URLs and a submit button.  On the buildbot side it
>> should be enough to create a new builder (by copying one of the
>> existing ones in the buildbot config file and adding the two steps
>> mentioned above).  Putting together a mail summary with the results of
>> all the slaves might require a bit of extra code, but it shouldn't be
>> anything too difficult.
>
> Yep. It is feasible.
>
>> Some of the issues with this proposal are:
>>   1) users uploading patches with malicious code -- this shouldn't be
>> a problem if we only allow developers/triagers to trigger the build
>> manually and tell them to review the code before doing so;
>
> No. Make environment safe from abuse. People will be tempted to
> press button ASAP, because process takes time to complete. For an
> example of safe service see drone.io (open source, Go).
>

This also would be a better solution but I'm unable to do that (also
because I think this should be done for the individual slaves, and
might be different depending on the OS and other things).  FWIW I know
of projects that trust the contributors and just test all the patches
that get submitted and so far they never had problems, but I
understand that is better to err on the safe side.  Requiring the
build to be manually triggered by devs after a review sounds like a
good compromise until we have a better solution.

>>   2) overloading of the slaves -- cloning/compiling/testing/cleaning
>> takes quite a lot of time, and if this new feature starts to be used
>> heavily it might start to give too much work for the slaves.  I don't
>> know what the current load is though, so I'm not sure this will be an
>> issue;
>
> The public information on http://speed.python.org tells me that it is
> still unused.
>
>>   3) testing/implementing the feature -- I think it should be possible
>> to test the feature locally before actually applying the changes on
>> the real buildbot master, but unless someone is going to apply them on
>> my behalf I would probably need access to the buildbot repo (is there
>> a public repo?);
>
> Making it a separate patch tracker service will allow to reuse it for
> different projects and more easy customization.

Are you suggesting that other projects reuse our code to test their
own patches on their own buildbots or that they reuse our buildbots to
test their things by using our code?
In both cases I'm not sure there's much to reuse, since they might
have a different bug tracker and they will probably want different
building steps.

> I don't want people
> to learn Roundup, TAL and other ancient technologies just to improve
> this code. If you don't mind against starting from other's code:
>

That would requires us to switch to more recent technologies, and is
also something that can't be done in a weekend :)

> 1. Google Chromium Try Server
> http://www.chromium.org/developers/testing/try-server-usage
> 2. Mercurial Patch Tracker
> http://patchwork.serpentine.com/project/hg/list/
>
>>   4) determine which branch the patch should be applied to -- this
>> could be done by adding a new "branch" field on the tracker; rietveld
>> could use it too;
>
> patch can be valid for multiple branches, properly exported patch
> has a header with parent revision.
>

Patches created with "hg diff" that don't use the --git format do, but
they have other limitations.
Patches created with "hg export" might have all the required
informations, though I'm not sure if they are automatically applied to
the tip of the branch or somewhere else.

>>   5) if the patch fails to apply there's no need to launch the build
>> on all the slaves -- a single slave (or maybe even the master?) could
>> check if the patch applies cleanly, and if it does trigger all the
>> other slaves.  More fine-grained options could be added to the request
>> form too (e.g. run only on specific slaves/operating systems);
>
> More useful thing IMHO is to detect which commit broke which patch.
>

That would require to re-test all the tracker patches every time a
commit is done, right?
What I was thinking is that whenever someone is going to work on an
old patch, he could retrigger the build manually from the bug tracker
and see if anything changed.

>> If the idea is reasonable and I didn't miss anything I can try to find
>> some time to work on it.
>
> The idea is awesome.
> --
> anatoly t.