Keeping an eye on Travis CI, AppVeyor and buildbots: revert on regression
Hi, I fixed a few tests which failed randomly. There are still a few, but the most annoying have been fixed. I will continue to keep an eye on our CIs: Travis CI, AppVeyor and buildbots. Sometimes, even when I report a regression, the author doesn't fix the bug. But when a test fails on Travis CI or AppVeyor, we cannot merge a pull request, so it blocks our whole workflow. I will restart the policy that I proposed last year: if a change introduces a regression and I'm unable to fix it quickly (say in less than 2 hours and the author isn't available), I will simply revert the change. Please don't take it personally, the purpose is just to unblock our workflow and be able to detect other regressions. It's just an opportunity for the change author to fix the change without the pressure of the broken CI. Buildbots only send email notifications to buildbot-status@python.org when the state changes from success (green) to failure (red). It's much simpler for me to spot a regression when most buildbots are green. By the way, while miss-lington is really an amazing tool (thanks Mariatta!!!), most core developers (including me!) forgot that 2 years ago, the policy was to check if a change doesn't break buildbots before backporting a change (well, 2 years ago we forward-ported changes, but it's the same idea ;-)). Please remind that we only run tests on Linux and Windows as pre-commit checks on GitHub pull requests, whereas it's very common to spot bugs on buildbots thanks to the diversity of platforms and architectures (and different performance). All buildbot builders can be watched at: http://buildbot.python.org/all/#/builders But I prefer to watch notifications on IRC (#python-dev on Freenode) and the buildbot-status mailing list. https://mail.python.org/mm3/mailman3/lists/buildbot-status.python.org/ -- Night gathers, AND NOW MY WATCH BEGINS. IT SHALL NOT END UNTIL MY DEATH. I shall take no wife, hold no lands, father no children. I shall wear no crowns and win no glory. I shall live and die at my post. I am the sword in the darkness. I am the watcher on the walls. I am the fire that burns against cold, the light that brings the dawn, the horn that wakes the sleepers, the shield that guards the realms of men. I pledge my life and honor to the Night's Watch, for this night and all the nights to come. Victor
2018-05-30 11:33 GMT+02:00 Victor Stinner <vstinner@redhat.com>:
I fixed a few tests which failed randomly. There are still a few, but the most annoying have been fixed.
Quick update a few days later. For an unknown reason, test_multiprocessing_forkserver.TestIgnoreEINTR.test_ignore() started to fail very frequently but only on Travis CI. I have no explanation why only Travis CI. I failed to reproduce the issue on a Ubuntu Trusty container or in a Ubuntu Trusty VM. After hours of debug, I found the bug and wrote a fix. But the fix didn't work in all cases. A second fix and now it seems like the issue is gone! https://bugs.python.org/issue33532 if you are curious about the strange multiprocessing send() which must block but it doesn't :-) Except Windows 7 which has issues with test_asyncio and multiprocessing tests because this buildbot is slow, it seems like most CIs are now stable. Known issues: * PPC64 Fedora 3.x, PPC64LE Fedora 3.x, s390x RHEL 3.x: https://bugs.python.org/issue33630 * AIX: always red * USBan: experimental buildbot * Alpine: platform not supported yet (musl issues) Victor
2018-06-04 18:31 GMT+02:00 Victor Stinner <vstinner@redhat.com>:
Quick update a few days later. (...) Except Windows 7 which has issues with test_asyncio and multiprocessing tests because this buildbot is slow, it seems like most CIs are now stable.
The bug wasn't specific to this buildbot, it was a very old race condition in the Windows ProactorEventLoop which only started recently to trigger test_asyncio failures. See https://bugs.python.org/issue33694 for the details.
Known issues:
* PPC64 Fedora 3.x, PPC64LE Fedora 3.x, s390x RHEL 3.x: https://bugs.python.org/issue33630 * AIX: always red * USBan: experimental buildbot * Alpine: platform not supported yet (musl issues)
Except of these few CIs, the main issue was test__xxsubinterpreters which is known to crash: https://bugs.python.org/issue33615 To fix CIs and give more time to Eric Snow to look at this crash, I decided to skip test__xxsubinterpreters. You may want to offer help to Eric to look into these tricky issues. So again, except of the few known issues listed above, all other CIs (Travis CI, AppVeyor, all other buildbots) should now pass. Victor
Are there APIs we can use to check the status of builbots? Maybe we can have the our bots check for the buildbot status in backport PRs. On Wed, May 30, 2018, 2:33 AM Victor Stinner <vstinner@redhat.com> wrote:
Buildbots only send email notifications to buildbot-status@python.org when the state changes from success (green) to failure (red). It's much simpler for me to spot a regression when most buildbots are green.
On Wed, Jun 6, 2018 at 9:45 PM, Mariatta Wijaya <mariatta.wijaya@gmail.com> wrote:
Are there APIs we can use to check the status of builbots? Maybe we can have the our bots check for the buildbot status in backport PRs.
There is a REST API for buildbot; I have no idea how usable/useful it is though (but I think the new UI interacts with the master mostly via REST calls). I am planning to eventually get buildbot integration with GitHub set up, possibly in September. I think it should be possible to make only stable bots show up as status checks, or even just a select subset of the stable bots. -- Zach
2018-06-07 4:45 GMT+02:00 Mariatta Wijaya <mariatta.wijaya@gmail.com>:
Are there APIs we can use to check the status of builbots?
Buildbots offer different ways to send notifications: emails and IRC bot for example. If you want to *poll* for recent builds, I don't know. I would suggest to use notifications (push) rather than polling.
Maybe we can have the our bots check for the buildbot status in backport PRs.
Right now, I'm not confortable with this idea because there is a risk to scare newcomers with false alarms and real bugs which are not coming from their changes, but may be known or are new but still unrelated to their changes. Moreover, even when a buildbot fails because of a real regression, a build may include multiple changes (I saw builds with more than 25 changes on some buildbots). Buildbot is different than a CI: slow buildbots are able to pack test multiple new changes at once. So again, there is a high risk of false alarms. Maybe I'm too conservative and we can try something with good documentation and proper warnings to explain properly such hypotetical notifications on pull requests. See also my other email which explains this differently: https://mail.python.org/pipermail/python-dev/2018-May/153759.html Victor
participants (3)
-
Mariatta Wijaya -
Victor Stinner -
Zachary Ware