[python-committers] Revert changes which break too many buildbots

[Victor Stinner]

Ok, since I spent weeks on fixing buildbots, I'm now more confident
that our buildbots are super stable. Since a test_datetime change
introduced a *regression* (ARMv7 started to fail), I reverted the
first commit:
https://github.com/python/cpython/pull/2588

Again, it's not to reject the change, just to repair the buildbot to
get more time to design and test the proper fix.

Because of a bug, test_datetime only tested the C implementation of
datetime. The change enables tests on Lib/datetime.py, the Python
implementation. Suddenly, test_datetime started to take up to 20
minutes to run! The slowest time on some buildbots.

I don't know much more at this point.

Please join http://bugs.python.org/issue30822 if you want to help
fixing this issue ;-)

Victor

2017-06-14 16:40 GMT+02:00 Victor Stinner <victor.stinner at gmail.com>:
> Hi,
>
> The CPython workflow was enhanced to get pre-commit CI checks. That's
> a huge win, thank you for that... But, sometimes, a change can still
> break many buildbots, bugs which weren't catched by pre-commit checks
> (Travis CI/Linux and AppVeyor/Windows). Buildbots cover much more
> different architectures and platforms.
>
> I spend a significant amount of time to maintain the sanity of our
> buildbots. Sometimes, it can take me up to 3 days on a week (of 5
> working days). It's annoying to see new regressions while I'm trying
> hard to fix old ones :-(
>
> So I would like to set a new rule: if I'm unable to fix buildbots
> failures caused by a recent change quickly (say, in less than 2
> hours), I propose to revert the change.
>
> It doesn't mean that the commit is bad and must not be merged ever.
> No. It would just mean that we need time to work on fixing the issue,
> and it shouldn't impact other pending changes, to keep a sane master
> branch.
>
> What do you think? Would you be ok with such rule?
>
> A recent example is Nick Coghlan's implementation of the PEP 538:
> basically, it broke all buildbots... except of Linux and Windows :-)
> And it will take a few more days to fix all failures. Well, we are
> working on fixing these issues, so I don't want to revert this change.
> It's just an example of a single change which broke many buildbots.
> The PEP 538 depends a lot on the platform, so I'm not surprised to see
> different failures per platforms ;-)
>
> By "buildbot failure", I mean a red buildbot failing because of
> compilation error or failed test suite. But I would prefer to ignore
> failures of the Refleak buildbots since these ones are not stable
> (even if there are less and less ref leaks in master, and these
> buildbots already catched recent regressions!).
>
> If the rule is approved, I plan to announce it on python-dev to be transparent.
>
> Victor