On 15 June 2017 at 00:40, Victor Stinner firstname.lastname@example.org wrote:
The CPython workflow was enhanced to get pre-commit CI checks. That's a huge win, thank you for that... But, sometimes, a change can still break many buildbots, bugs which weren't catched by pre-commit checks (Travis CI/Linux and AppVeyor/Windows). Buildbots cover much more different architectures and platforms.
I spend a significant amount of time to maintain the sanity of our buildbots. Sometimes, it can take me up to 3 days on a week (of 5 working days). It's annoying to see new regressions while I'm trying hard to fix old ones :-(
So I would like to set a new rule: if I'm unable to fix buildbots failures caused by a recent change quickly (say, in less than 2 hours), I propose to revert the change.
I'm not necessarily opposed to such a policy change, but if folks really want guaranteed green post-merge buildbots for all platforms (rather than just guaranteed green for Linux & Windows, sometimes red for everything else), then I think a better place to focus effort would be in making it easier for us to test things across the full BuildBot fleet in pre-merge CI.
For example, something that would be genuinely helpful would be a bot monitoring PR comments that could automate the "custom-build" dance for core developers (i.e. we'd be able to write something like "BuildBot: test custom build", and it would go away, kick off a custom BuildBot run by pushing the PR to the "custom-build" branch, and then report back the links for failed builds like http://buildbot.python.org/all/builders/x86%20Tiger%20custom/builds/5 )
That way, the reversion process would be:
the issue resolution process based on the specific links posted back by the helper bot