[core-workflow] Starting the improved workflow discussion again

[Nick Coghlan]

On 21 July 2015 at 05:49, Brett Cannon <bcannon at gmail.com> wrote:
> In my ideal workflow scenario, these are the steps a patch would take:
>
> Issue is created
> Issue is triaged to have right affected versions, etc.
> Patch is uploaded
> CI kicks the patch off for all branches and OSs that are affected
> CI flags what branches and OSs did (not) pass or apply cleanly to
> If necessary, another patch that works in a specific branch that is affected
> is uploaded (obviously requires some way to flag that a patch applies to a
> specific branch, deciding how to deal with Misc/NEWS, etc.)
> Code review -- with a tool other than Rietveld -- from a core developer with
> feedback
> New version of patch uploaded, usual CI kicked off
> If everything looks good and CI is green, get patch approval from a core dev
> Approval submits the patch(es) to the appropriate branches
> CI triggered yet again, and if tests fail then patch(es) are automatically
> rolled back
>
> Now I realize this is not about to launch immediately. There are changes to
> Roundup in there, a reliable test suite that actually fails only on failures
> and not because it's flaky, etc. But the key point here is that everything
> that can be automated is, and code reviews can occur entirely through a
> browser.

I think you're conflating some different issues here, at least two of
which are worth separating out from each other:

1. Completely online workflow for documentation editing
2. Pre-commit CI for CPython

Both of the current forge.python.org proposals are aimed primarily at
the first problem, since they start out with purely documentation
repos like the developer guide and the PEPs. Hopefully we can also
eventually separate out "version independent" repos for the how to
guides and the tutorial. In addition to a completely online process
for documentation editing, review, and approval, the other key benefit
to these repos would be that *access management* would also be online,
rather than requiring shell access to hg.python.org.

Documentation projects are a good starting point for this side of
things, as they're inherently lower risk. The worst thing
documentation can do is give readers bad advice, it can't force them
to follow it.

This means that for forge.python.org, I think "What about CPython?"
should be something we take into account as a "What's next?" for the
service, but our near term focus should be on making things like the
developer guide and PEPs trivial to suggest edits to, trivial to
approve edits to, and trivial to grant approval rights over. Those
levels of access (who can suggest edits, who can approve edits, who
can approve edit rights for others) should also all be completely
transparent and changes in them should be tracked automatically rather
than requiring manual updates to a text file.

Pre-commit CI for CPython is a different story - as David points out,
it is *not* OK to run code on the Buildbot fleet that hasn't been
approved by a core developer. Folks are trusting *us* to run code on
their systems, not random developers posting patches to
bugs.python.org. Noah (quite sensibly) isn't interested in getting the
PSF Infrastructure team involved in running random code from the
internet either.

That's where the system Kushal set up in collaboration with the CentOS
folks potentially comes in:
https://mail.python.org/pipermail/python-dev/2015-May/140050.html

That's just a simple "smoke test" to say "Does the proposed change
pass on x86_64 systems running CentOS 7?". If we could combine it with
a similar system for running Windows smoke tests in Appveyor, I think
we'd flush out a lot of basic cross-platform compatibility issues
pre-commit, regardless of whether folks are working locally on a *nix
system or a Windows one. (We wouldn't catch *everything*, because
Linux is not FreeBSD is not Mac OS X, etc, but we'd catch a lot of
them).

At the moment, running those requires that we be logged into IRC, be
approved to trigger test runs, and indicate which issue we'd like to
test.

If we instead had a "test" link next to patch files in Roundup, then a
core developer, completely online, could:

1. Read over the patch to see if we think its reasonable to smoke test
2. Trigger the smoke test directly from Roundup
3. Receive the results back as Roundup comments, with links to the run logs

As we gained further familiarity and confidence with the system, we
could extend the trust for running pre-commit test runs to anyone that
has been granted Developer privileges on the issue tracker, rather
than restricting it specifically to core developers. (BTW, we should
probably come up with an icon for folks with elevated tracker
privileges - at the moment they're just marked as having signed the
CLA if they aren't also CPython core developers)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia