LCA2014 presentation on Zuul, OpenStack's merge gating system
One of the items I have on the language summit agenda is the core reviewer workflow used by the OpenStack project, specifically Zuul, the merge gating system that powers that workflow. I know some other folks here are working on OpenStack these days so this will already be familiar to them, but many aren't. I'm not either, but my day job involves developing CI workflow automation tools for Red Hat, so I'm paying very close attention to what the OpenStack infrastructure team are up to.
We don't necessarily have to wait until the language summit to discuss it though, so I figured I'd pass along a link to James Blair's presentation at linux.conf.au 2014:
Stream: https://www.youtube.com/watch?v=sLD9LHc1QFM Download: http://mirror.linux.org.au/linux.conf.au/2014/Friday/106-How_OpenStack_Impro...
The Zuul docs are at: http://ci.openstack.org/zuul/ The OpenStack Zuul dashboard is at: http://status.openstack.org/zuul/
The key difference, relative to our current workflow, is to take the human out of the loop for the final pre-merge test run. Instead, once a core reviewer gives a patch a +2, the process of taking that reviewed patch, checking it still merges correctly, checking it passes all the tests on the stable buildbots and then merging it into source control would all be handled automatically by Zuul.
This will require and/or enable eliminating several of the current annoyances in the core development workflow:
- the annoyingly manual "download patch, apply patch, run tests locally, commit change, push change, watch the buildbots for problems" approach goes away entirely (while what we have now is substantially better than what we had a few years ago, times move on and projects like OpenStack raise the bar when it comes to figuring out how to make the most effective of scarce contributor time)
- spurious conflicts on NEWS will go away (as we'll have to finally address this in order to avoid breaking Zuul's gating)
- push races will go away (since merging would now be handled by Zuul, non conflicting changes would be rebased automatically, and only conflicting changes would be bounced back to the submitter for updates). Critically, conference sprints might create a backlog in Zuul, but they wouldn't completely break a developer's workflow through incessant push races relating to non-conflicting changes.
- POSIX developers breaking the Windows buildbots and vice-versa (since Zuul would ensure at least the stable buildbots remain green by blocking such changes before they hit the main branches). This means even when we get such things wrong, we will no longer have the time pressure of needing to unbreak other people's builds.
- approving and merging pure docs patches should become largely trivial, as it should be possible to configure Zuul to only check that the docs build cleanly for those, rather than running the full test suite across all the stable buildbots.
- checking for a contributor licensing agreement in Roundup before processing a patch could also be automated, rather than requiring core developers to check for it manually.
Zuul, as OpenStack use it, already has plugins for the Gerrit code review/git repo management system (at least as customised by the OpenStack infrastructure folks), as well as for Jenkins to run the automated CI tests.
Rather than suggesting wholesale changes to our own infrastructure, what I am suggesting we consider is devoting time (and potentially PSF funding) to getting Zuul to play nice with Roundup, Reitveld, BuildBot and Mercurial. (Like the rest of our existing infrastructure, Zuul is a web service implemented in Python).
The main technical challenges I foresee are:
- dealing with maintenance branches (especially for patches which don't merge forward cleanly), since OpenStack currently appear to "handle" that limitation by just not providing upstream maintenance branches at all, leaving downstream vendors to fill the gap
- finally cleaning up the way we manage the NEWS file (see http://bugs.python.org/issue18967 for discussion)
- replicating Gerrit's patch approval/voting mechanism in Reitveld
- replicating Gerrit's merge/rebase capabilities in Mercurial (I'm sure Mercurial is functionally capable of this, I just don't know the best way to model it).
- actually writing the new Zuul plugins to talk to the services used by the CPython project, rather than those used by OpenStack
There would also be a non-trivial update to the developer guide involved in such a change, since it would heavily impact the core developer workflows, as well as the way external contributors interact with the core development team.
If people find this idea interesting, I would like to invite some of the Mercurial devs and OpenStack infrastructure folks (i.e. Zuul devs) to the language summit (leaving it up to them if they stay for the whole day, or just this part). I believe we already have enough Reitveld, Roundup and Buildbot experience amongst the core development team to assess the feasibility of the idea from that side of things.
Regards, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
While I'm not a particularly active python-developer, I have spent quite a bit of time using OpenStack's Zuul/gerrit system/workflow and all I can say is it: It is the right way to write software. Huge +1 from me on every project using it :-)
Alex
On Sat, Jan 18, 2014 at 11:34 PM, Nick Coghlan ncoghlan@gmail.com wrote:
One of the items I have on the language summit agenda is the core reviewer workflow used by the OpenStack project, specifically Zuul, the merge gating system that powers that workflow. I know some other folks here are working on OpenStack these days so this will already be familiar to them, but many aren't. I'm not either, but my day job involves developing CI workflow automation tools for Red Hat, so I'm paying very close attention to what the OpenStack infrastructure team are up to.
We don't necessarily have to wait until the language summit to discuss it though, so I figured I'd pass along a link to James Blair's presentation at linux.conf.au 2014:
Stream: https://www.youtube.com/watch?v=sLD9LHc1QFM Download: http://mirror.linux.org.au/linux.conf.au/2014/Friday/106-How_OpenStack_Impro...
The Zuul docs are at: http://ci.openstack.org/zuul/ The OpenStack Zuul dashboard is at: http://status.openstack.org/zuul/
The key difference, relative to our current workflow, is to take the human out of the loop for the final pre-merge test run. Instead, once a core reviewer gives a patch a +2, the process of taking that reviewed patch, checking it still merges correctly, checking it passes all the tests on the stable buildbots and then merging it into source control would all be handled automatically by Zuul.
This will require and/or enable eliminating several of the current annoyances in the core development workflow:
- the annoyingly manual "download patch, apply patch, run tests locally, commit change, push change, watch the buildbots for problems" approach goes away entirely (while what we have now is substantially better than what we had a few years ago, times move on and projects like OpenStack raise the bar when it comes to figuring out how to make the most effective of scarce contributor time)
- spurious conflicts on NEWS will go away (as we'll have to finally address this in order to avoid breaking Zuul's gating)
- push races will go away (since merging would now be handled by Zuul, non conflicting changes would be rebased automatically, and only conflicting changes would be bounced back to the submitter for updates). Critically, conference sprints might create a backlog in Zuul, but they wouldn't completely break a developer's workflow through incessant push races relating to non-conflicting changes.
- POSIX developers breaking the Windows buildbots and vice-versa (since Zuul would ensure at least the stable buildbots remain green by blocking such changes before they hit the main branches). This means even when we get such things wrong, we will no longer have the time pressure of needing to unbreak other people's builds.
- approving and merging pure docs patches should become largely trivial, as it should be possible to configure Zuul to only check that the docs build cleanly for those, rather than running the full test suite across all the stable buildbots.
- checking for a contributor licensing agreement in Roundup before processing a patch could also be automated, rather than requiring core developers to check for it manually.
Zuul, as OpenStack use it, already has plugins for the Gerrit code review/git repo management system (at least as customised by the OpenStack infrastructure folks), as well as for Jenkins to run the automated CI tests.
Rather than suggesting wholesale changes to our own infrastructure, what I am suggesting we consider is devoting time (and potentially PSF funding) to getting Zuul to play nice with Roundup, Reitveld, BuildBot and Mercurial. (Like the rest of our existing infrastructure, Zuul is a web service implemented in Python).
The main technical challenges I foresee are:
- dealing with maintenance branches (especially for patches which don't merge forward cleanly), since OpenStack currently appear to "handle" that limitation by just not providing upstream maintenance branches at all, leaving downstream vendors to fill the gap
- finally cleaning up the way we manage the NEWS file (see http://bugs.python.org/issue18967 for discussion)
- replicating Gerrit's patch approval/voting mechanism in Reitveld
- replicating Gerrit's merge/rebase capabilities in Mercurial (I'm sure Mercurial is functionally capable of this, I just don't know the best way to model it).
- actually writing the new Zuul plugins to talk to the services used by the CPython project, rather than those used by OpenStack
There would also be a non-trivial update to the developer guide involved in such a change, since it would heavily impact the core developer workflows, as well as the way external contributors interact with the core development team.
If people find this idea interesting, I would like to invite some of the Mercurial devs and OpenStack infrastructure folks (i.e. Zuul devs) to the language summit (leaving it up to them if they stay for the whole day, or just this part). I believe we already have enough Reitveld, Roundup and Buildbot experience amongst the core development team to assess the feasibility of the idea from that side of things.
Regards, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
python-committers mailing list python-committers@python.org https://mail.python.org/mailman/listinfo/python-committers
-- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero GPG Key fingerprint: 125F 5C67 DFE9 4084
On dim., 2014-01-19 at 15:34 +1000, Nick Coghlan wrote:
Rather than suggesting wholesale changes to our own infrastructure, what I am suggesting we consider is devoting time (and potentially PSF funding) to getting Zuul to play nice with Roundup, Reitveld, BuildBot and Mercurial. (Like the rest of our existing infrastructure, Zuul is a web service implemented in Python).
Just a minor point: it's Rietveld, not "Reitveld".
Regards
Antoine.
On 19 January 2014 20:59, Antoine Pitrou solipsis@pitrou.net wrote:
On dim., 2014-01-19 at 15:34 +1000, Nick Coghlan wrote:
Rather than suggesting wholesale changes to our own infrastructure, what I am suggesting we consider is devoting time (and potentially PSF funding) to getting Zuul to play nice with Roundup, Reitveld, BuildBot and Mercurial. (Like the rest of our existing infrastructure, Zuul is a web service implemented in Python).
Just a minor point: it's Rietveld, not "Reitveld".
Huh, I think I've been transposing that for years. My confidence in my ability to retrain my fingers is not high, but I'll try to remember the right spelling :)
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
participants (3)
-
Alex Gaynor -
Antoine Pitrou -
Nick Coghlan