Ok, since I spent weeks on fixing buildbots, I'm now more confident that our buildbots are super stable. Since a test_datetime change introduced a *regression* (ARMv7 started to fail), I reverted the first commit: https://github.com/python/cpython/pull/2588
Again, it's not to reject the change, just to repair the buildbot to get more time to design and test the proper fix.
Because of a bug, test_datetime only tested the C implementation of datetime. The change enables tests on Lib/datetime.py, the Python implementation. Suddenly, test_datetime started to take up to 20 minutes to run! The slowest time on some buildbots.
I don't know much more at this point.
Please join http://bugs.python.org/issue30822 if you want to help fixing this issue ;-)
2017-06-14 16:40 GMT+02:00 Victor Stinner email@example.com:
The CPython workflow was enhanced to get pre-commit CI checks. That's a huge win, thank you for that... But, sometimes, a change can still break many buildbots, bugs which weren't catched by pre-commit checks (Travis CI/Linux and AppVeyor/Windows). Buildbots cover much more different architectures and platforms.
I spend a significant amount of time to maintain the sanity of our buildbots. Sometimes, it can take me up to 3 days on a week (of 5 working days). It's annoying to see new regressions while I'm trying hard to fix old ones :-(
So I would like to set a new rule: if I'm unable to fix buildbots failures caused by a recent change quickly (say, in less than 2 hours), I propose to revert the change.
It doesn't mean that the commit is bad and must not be merged ever. No. It would just mean that we need time to work on fixing the issue, and it shouldn't impact other pending changes, to keep a sane master branch.
What do you think? Would you be ok with such rule?
A recent example is Nick Coghlan's implementation of the PEP 538: basically, it broke all buildbots... except of Linux and Windows :-) And it will take a few more days to fix all failures. Well, we are working on fixing these issues, so I don't want to revert this change. It's just an example of a single change which broke many buildbots. The PEP 538 depends a lot on the platform, so I'm not surprised to see different failures per platforms ;-)
By "buildbot failure", I mean a red buildbot failing because of compilation error or failed test suite. But I would prefer to ignore failures of the Refleak buildbots since these ones are not stable (even if there are less and less ref leaks in master, and these buildbots already catched recent regressions!).
If the rule is approved, I plan to announce it on python-dev to be transparent.