[python-committers] LCA2014 presentation on Zuul, OpenStack's merge gating system

[Alex Gaynor]

While I'm not a particularly active python-developer, I have spent quite a
bit of time using OpenStack's Zuul/gerrit system/workflow and all I can say
is it: It is the right way to write software. Huge +1 from me on every
project using it :-)

Alex


On Sat, Jan 18, 2014 at 11:34 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> One of the items I have on the language summit agenda is the core
> reviewer workflow used by the OpenStack project, specifically Zuul,
> the merge gating system that powers that workflow. I know some other
> folks here are working on OpenStack these days so this will already be
> familiar to them, but many aren't. I'm not either, but my day job
> involves developing CI workflow automation tools for Red Hat, so I'm
> paying very close attention to what the OpenStack infrastructure team
> are up to.
>
> We don't necessarily have to wait until the language summit to discuss
> it though, so I figured I'd pass along a link to James Blair's
> presentation at linux.conf.au 2014:
>
> Stream: https://www.youtube.com/watch?v=sLD9LHc1QFM
> Download:
> http://mirror.linux.org.au/linux.conf.au/2014/Friday/106-How_OpenStack_Improves_Code_Quality_with_Project_Gating_and_Zuul_-_James_Blair.mp4
>
> The Zuul docs are at: http://ci.openstack.org/zuul/
> The OpenStack Zuul dashboard is at: http://status.openstack.org/zuul/
>
> The key difference, relative to our current workflow, is to take the
> human out of the loop for the final pre-merge test run. Instead, once
> a core reviewer gives a patch a +2, the process of taking that
> reviewed patch, checking it still merges correctly, checking it passes
> all the tests on the stable buildbots and then merging it into source
> control would all be handled automatically by Zuul.
>
> This will require and/or enable eliminating several of the current
> annoyances in the core development workflow:
>
> - the annoyingly manual "download patch, apply patch, run tests
> locally, commit change, push change, watch the buildbots for problems"
> approach goes away entirely (while what we have now is substantially
> better than what we had a few years ago, times move on and projects
> like OpenStack raise the bar when it comes to figuring out how to make
> the most effective of scarce contributor time)
> - spurious conflicts on NEWS will go away (as we'll have to finally
> address this in order to avoid breaking Zuul's gating)
> - push races will go away (since merging would now be handled by Zuul,
> non conflicting changes would be rebased automatically, and only
> conflicting changes would be bounced back to the submitter for
> updates). Critically, conference sprints might create a backlog in
> Zuul, but they wouldn't completely break a developer's workflow
> through incessant push races relating to non-conflicting changes.
> - POSIX developers breaking the Windows buildbots and vice-versa
> (since Zuul would ensure at least the stable buildbots remain green by
> blocking such changes before they hit the main branches). This means
> even when we get such things wrong, we will no longer have the time
> pressure of needing to unbreak other people's builds.
> - approving and merging pure docs patches should become largely
> trivial, as it should be possible to configure Zuul to only check that
> the docs build cleanly for those, rather than running the full test
> suite across all the stable buildbots.
> - checking for a contributor licensing agreement in Roundup before
> processing a patch could also be automated, rather than requiring core
> developers to check for it manually.
>
> Zuul, as OpenStack use it, already has plugins for the Gerrit code
> review/git repo management system (at least as customised by the
> OpenStack infrastructure folks), as well as for Jenkins to run the
> automated CI tests.
>
> Rather than suggesting wholesale changes to our own infrastructure,
> what I am suggesting we consider is devoting time (and potentially PSF
> funding) to getting Zuul to play nice with Roundup, Reitveld, BuildBot
> and Mercurial. (Like the rest of our existing infrastructure, Zuul is
> a web service implemented in Python).
>
> The main technical challenges I foresee are:
>
> * dealing with maintenance branches (especially for patches which
> don't merge forward cleanly), since OpenStack currently appear to
> "handle" that limitation by just not providing upstream maintenance
> branches at all, leaving downstream vendors to fill the gap
> * finally cleaning up the way we manage the NEWS file (see
> http://bugs.python.org/issue18967 for discussion)
> * replicating Gerrit's patch approval/voting mechanism in Reitveld
> * replicating Gerrit's merge/rebase capabilities in Mercurial (I'm
> sure Mercurial is functionally capable of this, I just don't know the
> best way to model it).
> * actually writing the new Zuul plugins to talk to the services used
> by the CPython project, rather than those used by OpenStack
>
> There would also be a non-trivial update to the developer guide
> involved in such a change, since it would heavily impact the core
> developer workflows, as well as the way external contributors interact
> with the core development team.
>
> If people find this idea interesting, I would like to invite some of
> the Mercurial devs and OpenStack infrastructure folks (i.e. Zuul devs)
> to the language summit (leaving it up to them if they stay for the
> whole day, or just this part). I believe we already have enough
> Reitveld, Roundup and Buildbot experience amongst the core development
> team to assess the feasibility of the idea from that side of things.
>
> Regards,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
> _______________________________________________
> python-committers mailing list
> python-committers at python.org
> https://mail.python.org/mailman/listinfo/python-committers
>



-- 
"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
GPG Key fingerprint: 125F 5C67 DFE9 4084
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-committers/attachments/20140118/e3353996/attachment.html>