[python-committers] LCA2014 presentation on Zuul, OpenStack's merge gating system

[Nick Coghlan]

One of the items I have on the language summit agenda is the core
reviewer workflow used by the OpenStack project, specifically Zuul,
the merge gating system that powers that workflow. I know some other
folks here are working on OpenStack these days so this will already be
familiar to them, but many aren't. I'm not either, but my day job
involves developing CI workflow automation tools for Red Hat, so I'm
paying very close attention to what the OpenStack infrastructure team
are up to.

We don't necessarily have to wait until the language summit to discuss
it though, so I figured I'd pass along a link to James Blair's
presentation at linux.conf.au 2014:

Stream: https://www.youtube.com/watch?v=sLD9LHc1QFM
Download: http://mirror.linux.org.au/linux.conf.au/2014/Friday/106-How_OpenStack_Improves_Code_Quality_with_Project_Gating_and_Zuul_-_James_Blair.mp4

The Zuul docs are at: http://ci.openstack.org/zuul/
The OpenStack Zuul dashboard is at: http://status.openstack.org/zuul/

The key difference, relative to our current workflow, is to take the
human out of the loop for the final pre-merge test run. Instead, once
a core reviewer gives a patch a +2, the process of taking that
reviewed patch, checking it still merges correctly, checking it passes
all the tests on the stable buildbots and then merging it into source
control would all be handled automatically by Zuul.

This will require and/or enable eliminating several of the current
annoyances in the core development workflow:

- the annoyingly manual "download patch, apply patch, run tests
locally, commit change, push change, watch the buildbots for problems"
approach goes away entirely (while what we have now is substantially
better than what we had a few years ago, times move on and projects
like OpenStack raise the bar when it comes to figuring out how to make
the most effective of scarce contributor time)
- spurious conflicts on NEWS will go away (as we'll have to finally
address this in order to avoid breaking Zuul's gating)
- push races will go away (since merging would now be handled by Zuul,
non conflicting changes would be rebased automatically, and only
conflicting changes would be bounced back to the submitter for
updates). Critically, conference sprints might create a backlog in
Zuul, but they wouldn't completely break a developer's workflow
through incessant push races relating to non-conflicting changes.
- POSIX developers breaking the Windows buildbots and vice-versa
(since Zuul would ensure at least the stable buildbots remain green by
blocking such changes before they hit the main branches). This means
even when we get such things wrong, we will no longer have the time
pressure of needing to unbreak other people's builds.
- approving and merging pure docs patches should become largely
trivial, as it should be possible to configure Zuul to only check that
the docs build cleanly for those, rather than running the full test
suite across all the stable buildbots.
- checking for a contributor licensing agreement in Roundup before
processing a patch could also be automated, rather than requiring core
developers to check for it manually.

Zuul, as OpenStack use it, already has plugins for the Gerrit code
review/git repo management system (at least as customised by the
OpenStack infrastructure folks), as well as for Jenkins to run the
automated CI tests.

Rather than suggesting wholesale changes to our own infrastructure,
what I am suggesting we consider is devoting time (and potentially PSF
funding) to getting Zuul to play nice with Roundup, Reitveld, BuildBot
and Mercurial. (Like the rest of our existing infrastructure, Zuul is
a web service implemented in Python).

The main technical challenges I foresee are:

* dealing with maintenance branches (especially for patches which
don't merge forward cleanly), since OpenStack currently appear to
"handle" that limitation by just not providing upstream maintenance
branches at all, leaving downstream vendors to fill the gap
* finally cleaning up the way we manage the NEWS file (see
http://bugs.python.org/issue18967 for discussion)
* replicating Gerrit's patch approval/voting mechanism in Reitveld
* replicating Gerrit's merge/rebase capabilities in Mercurial (I'm
sure Mercurial is functionally capable of this, I just don't know the
best way to model it).
* actually writing the new Zuul plugins to talk to the services used
by the CPython project, rather than those used by OpenStack

There would also be a non-trivial update to the developer guide
involved in such a change, since it would heavily impact the core
developer workflows, as well as the way external contributors interact
with the core development team.

If people find this idea interesting, I would like to invite some of
the Mercurial devs and OpenStack infrastructure folks (i.e. Zuul devs)
to the language summit (leaving it up to them if they stay for the
whole day, or just this part). I believe we already have enough
Reitveld, Roundup and Buildbot experience amongst the core development
team to assess the feasibility of the idea from that side of things.

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia