On 30 May 2018 at 22:30, Ivan Pozdeev via Python-Dev python-dev@python.org wrote:

What's the big idea of separate buildbots anyway? I thought the purpose of CI is to test everything _before_ it breaks the main codebase. Then it's the job of the contributor rather than maintainer to fix any breakages.

So, maybe making them be driven by Github checks would be a better time investment. Especially since we've got VSTS checks just recently, so whoever was doing that still knows how to interface with this Github machinery.

If the bots cancel a previous build if a new one for the same PR arrives, this will not lead to a significant load difference 'cuz the number of actively developed PRs is stable and roughly equal to the number of merges according to the open/closed tickets dynamics.

There are a few key details here:

1. We currently need to run post-merge CI anyway, as we're not doing linearised commits (where core devs just approve a change without merging it, and then a gating system like Zuul ensures that the tests are run against the latest combination of the target branch and the PR before merging the change) 2. Since the buildbots are running on donated dedicated machines (rather than throwaway instances from a dynamic CI provider), we need to review the code before we let it run on the contributed systems 3. The buildbot instances run *1* build at a time, which would lead to major PR merging bottlenecks during sprints if we made them a gating requirement 4. For the vast majority of PRs, the post-merge cross-platform testing is a formality, since the code being modified is using lower level cross-platform APIs elsewhere in the standard library, so if it works on Windows, Linux, and Mac OS X, it will work everywhere Python runs 5. We generally don't *want* to burden new contributors with the task of dealing with the less common (or harder to target) platforms outside the big 3 - when they do break, it often takes a non-trivial amount of platform knowledge to understand what's different about the platform in question

Cheers, Nick.

P.S. That said, if VSTS or Travis were to offer FreeBSD as an option for pre-merge CI, I'd suggest we enable it, at least in an advisory capacity - it's a better check against Linux-specific assumptions creeping into the code base than Mac OS X, since the latter is regularly different enough from other *nix systems that we need to give it dedicated code paths.

On Wed, May 30, 2018 at 7:36 AM, Nick Coghlan ncoghlan@gmail.com wrote:

There are a few key details here:

1. We currently need to run post-merge CI anyway, as we're not doing

linearised commits (where core devs just approve a change without merging it, and then a gating system like Zuul ensures that the tests are run against the latest combination of the target branch and the PR before merging the change)

This is more of a concern when non-conflicting PRs against the same (or related) code are active at the same time. For the CPython code base this isn't as much of a problem, right? Under normal circumstances [fresh] active PRs typically do not run afoul of each other. Furthermore, during peak-activity events (like sprints) folks tend to keep a closer eye on the buildbots. I suppose old-but-still-active PRs that previously passed CI could cause a problem. However, it would be unlikely for such a PR to sit for a long time without needing changes before merging, whether to address reviewer concerns or to resolve merge conflicts.

So post-merge CI (or merge gating) doesn't seem like much of a factor for us. In that regard I'd consider the buildbots more that sufficient.

2. Since the buildbots are running on donated dedicated machines (rather

than throwaway instances from a dynamic CI provider), we need to review the code before we let it run on the contributed systems 3. The buildbot instances run *1* build at a time,

...where each build incorporates potentially several merged PRs...

which would lead to major PR merging bottlenecks during sprints if we made them a gating requirement

Agreed. There's enough of a delay already when watching the buildbots post-merge (especially some of them). :)

4. For the vast majority of PRs, the post-merge cross-platform testing is a

formality, since the code being modified is using lower level cross-platform APIs elsewhere in the standard library, so if it works on Windows, Linux, and Mac OS X, it will work everywhere Python runs

This is especially true of changes proposed by non-core contributors. It is also very true for buildbots with the OS/hardware combos that match CI. That said, when working with the C-API you can end up breaking things on the less common OSes and hardware platforms. So *those* buildbots are invaluable. I'm dealing with that right now.

5. We generally don't *want* to burden new contributors with the task of

dealing with the less common (or harder to target) platforms outside the big 3 - when they do break, it often takes a non-trivial amount of platform knowledge to understand what's different about the platform in question

As hinted above, I would not expect new contributors to provide patches very often (if ever) that would have the potential to cause buildbot failures but not fail under CI. So this point seems somewhat moot. :)

P.S. That said, if VSTS or Travis were to offer FreeBSD as an option for pre-merge CI, I'd suggest we enable it, at least in an advisory capacity - it's a better check against Linux-specific assumptions creeping into the code base than Mac OS X, since the latter is regularly different enough from other *nix systems that we need to give it dedicated code paths.

+1

-eric