Time to switch off AppVeyor and start requiring VSTS for Windows?
We had to turn off AppVeyor as a requirement today due to it not running any builds for the last 20 hours. Ned Deily suggested that since VSTS on at least Windows has stabilized that maybe we should switch off AppVeyor off entirely and make the Windows VSTS builder required. Since https://python.visualstudio.com/cpython/_build/index?context=mine&path=%5C&definitionId=4&_a=history shows the builder has been stable I would like to make this required for master today/tomorrow and then flip it on for the other 3.x branches on Friday if we don't see any instability. Any objections/opinions about doing this?
There are some updates on the AppVeyor side: https://mail.python.org/pipermail/python-committers/2018-June/005549.html Victor 2018-06-04 20:34 GMT+02:00 Brett Cannon <brett@python.org>:
2018-06-04 20:34 GMT+02:00 Brett Cannon <brett@python.org>:
Just to be pedantic, it's not 100% stable yet :-) test_asyncio strikes back: https://bugs.python.org/issue33694#msg318705 I see two 3.7 and master builds which failed because of this issue. Moreover, it seems like the regrtest -w option is missing: failing tests are not re-run in verbose mode. Victor
On Mon, 4 Jun 2018 at 15:58 Victor Stinner <vstinner@redhat.com> wrote:
Right, but those aren't special to VSTS, correct?
Moreover, it seems like the regrtest -w option is missing: failing tests are not re-run in verbose mode.
Did you want to propose a PR to update the YAML files in the .vsts directory to start using -w?
Steve has no plan to support VSTS on Python 2.7: "Correct, and I wasn't planning on it. The default VSTS machines don’t have the right compiler, and I’m not about to start managing VMs for 2.7 at this stage." So I would suggest to keep AppVeyor for Python 2.7, at least for the short term, and maybe make it mandatory again since it works again we doubled our capacity (1 job => 2 jobs in parallel)! The workflow velocity is less impact for Python 2.7 which gets less changes than 3.x & master branches. Victor 2018-06-04 20:34 GMT+02:00 Brett Cannon <brett@python.org>:
IHMO this issue is a blocker to make VSTS mandatory: https://bugs.python.org/issue33782 VSTS Windows-PR failed with "The request was failed due to an internal service error. Please try again." It would prevent to not be blocked by an internal VSTS error :-( Victor 2018-06-05 20:07 GMT+02:00 Brett Cannon <brett@python.org>:
On 06/06, Victor Stinner wrote:
It would prevent to not be blocked by an internal VSTS error :-(
-- Stéphane Wirtel - http://wirtel.be - @matrixise
On Wed, 6 Jun 2018 at 04:16 Victor Stinner <vstinner@redhat.com> wrote:
While I agree it should be looked into (which I know it is as I'm cc'ed on the internal emails), I don't think making it a blocker is quite fair. If blocking making VSTS required for a couple of failed builds due to an internal error that could be kicked off by closing and re-opening a PR then I don't know if it's reasonable to require AppVeyor again when they were down for all PRs for 20 hours. My point is this is all software which we know isn't perfect and as long as there is a way to kick off a wedged/failed build again I don't think it's a hard blocker. -Brett
2018-06-06 19:37 GMT+02:00 Brett Cannon <brett@python.org>:
Ok, let me elaborate. I'm trying to stabilize Travis CI, AppVeyor and buildbots for 1 month. I got some help on some issues, but sadly too few core devs are working on making CIs more stable, whereas CIs are blocking our workflow when they fail. It's annoying when a PR is blocked by a failure which is unrelated to the proposed change. In 1 month, I already saw significant progress, but there are still a few open issues: https://mail.python.org/pipermail/python-dev/2018-June/153796.html My point is that making VSTS mandatory adds more issues. I would prefer to have more time to fix existing *known* issues first. I'm not talking about VSTS issues in particular, but all CI known issues. In the meanwhile, AppVeyor seems stable again. Would it be possible to re-enable it in all stable branches? Victor
2018-06-07 5:36 GMT+02:00 Brett Cannon <brett@python.org>:
Yes, I mean making it required again... but yesterday I made progress on the super annoying test_asyncio random failure, and it seems like a major regression in ProactorEventLoop on Windows: https://bugs.python.org/issue33694 Calling pause_reading() / resume_reading() of a proactor transport can lead to data loss :-( Right now, I would prefer to not make any Windows CI required, until the test is either fixed or skipped. The failure rate is way too high (25% at least, maybe even 50%?). I'm working on a fix, but it's super complex :-( I would need an IOCP guru and an asyncio guru to help me here. Maybe the short term fix is to first skip the test, since the bug is know well identified, and I'm easily able to reproduce the failure. Victor
On Thu, 7 Jun 2018 at 06:08 Victor Stinner <vstinner@redhat.com> wrote:
I'll leave that call up to you since you're putting in the work. If you disable the test then I'm comfortable with making AppVeyor required again, otherwise with a failure rate that high we shouldn't require any Windows status check.
Update: I fixed the root issue of https://bugs.python.org/issue33694 it was a race condition in ProactorEventLoop, so specific to Windows. test_asyncio *should* now pass on all Windows CIs. I leave the issue open a few days to check if the issue is really fixed. Victor 2018-06-07 18:38 GMT+02:00 Brett Cannon <brett@python.org>:
There are some updates on the AppVeyor side: https://mail.python.org/pipermail/python-committers/2018-June/005549.html Victor 2018-06-04 20:34 GMT+02:00 Brett Cannon <brett@python.org>:
2018-06-04 20:34 GMT+02:00 Brett Cannon <brett@python.org>:
Just to be pedantic, it's not 100% stable yet :-) test_asyncio strikes back: https://bugs.python.org/issue33694#msg318705 I see two 3.7 and master builds which failed because of this issue. Moreover, it seems like the regrtest -w option is missing: failing tests are not re-run in verbose mode. Victor
On Mon, 4 Jun 2018 at 15:58 Victor Stinner <vstinner@redhat.com> wrote:
Right, but those aren't special to VSTS, correct?
Moreover, it seems like the regrtest -w option is missing: failing tests are not re-run in verbose mode.
Did you want to propose a PR to update the YAML files in the .vsts directory to start using -w?
Steve has no plan to support VSTS on Python 2.7: "Correct, and I wasn't planning on it. The default VSTS machines don’t have the right compiler, and I’m not about to start managing VMs for 2.7 at this stage." So I would suggest to keep AppVeyor for Python 2.7, at least for the short term, and maybe make it mandatory again since it works again we doubled our capacity (1 job => 2 jobs in parallel)! The workflow velocity is less impact for Python 2.7 which gets less changes than 3.x & master branches. Victor 2018-06-04 20:34 GMT+02:00 Brett Cannon <brett@python.org>:
IHMO this issue is a blocker to make VSTS mandatory: https://bugs.python.org/issue33782 VSTS Windows-PR failed with "The request was failed due to an internal service error. Please try again." It would prevent to not be blocked by an internal VSTS error :-( Victor 2018-06-05 20:07 GMT+02:00 Brett Cannon <brett@python.org>:
On 06/06, Victor Stinner wrote:
It would prevent to not be blocked by an internal VSTS error :-(
-- Stéphane Wirtel - http://wirtel.be - @matrixise
On Wed, 6 Jun 2018 at 04:16 Victor Stinner <vstinner@redhat.com> wrote:
While I agree it should be looked into (which I know it is as I'm cc'ed on the internal emails), I don't think making it a blocker is quite fair. If blocking making VSTS required for a couple of failed builds due to an internal error that could be kicked off by closing and re-opening a PR then I don't know if it's reasonable to require AppVeyor again when they were down for all PRs for 20 hours. My point is this is all software which we know isn't perfect and as long as there is a way to kick off a wedged/failed build again I don't think it's a hard blocker. -Brett
2018-06-06 19:37 GMT+02:00 Brett Cannon <brett@python.org>:
Ok, let me elaborate. I'm trying to stabilize Travis CI, AppVeyor and buildbots for 1 month. I got some help on some issues, but sadly too few core devs are working on making CIs more stable, whereas CIs are blocking our workflow when they fail. It's annoying when a PR is blocked by a failure which is unrelated to the proposed change. In 1 month, I already saw significant progress, but there are still a few open issues: https://mail.python.org/pipermail/python-dev/2018-June/153796.html My point is that making VSTS mandatory adds more issues. I would prefer to have more time to fix existing *known* issues first. I'm not talking about VSTS issues in particular, but all CI known issues. In the meanwhile, AppVeyor seems stable again. Would it be possible to re-enable it in all stable branches? Victor
On Wed, Jun 6, 2018, 15:48 Victor Stinner, <vstinner@redhat.com> wrote:
Fair enough!
In the meanwhile, AppVeyor seems stable again. Would it be possible to re-enable it in all stable branches?
It was never turned off, but if you mean make it required again then yes.
2018-06-07 5:36 GMT+02:00 Brett Cannon <brett@python.org>:
Yes, I mean making it required again... but yesterday I made progress on the super annoying test_asyncio random failure, and it seems like a major regression in ProactorEventLoop on Windows: https://bugs.python.org/issue33694 Calling pause_reading() / resume_reading() of a proactor transport can lead to data loss :-( Right now, I would prefer to not make any Windows CI required, until the test is either fixed or skipped. The failure rate is way too high (25% at least, maybe even 50%?). I'm working on a fix, but it's super complex :-( I would need an IOCP guru and an asyncio guru to help me here. Maybe the short term fix is to first skip the test, since the bug is know well identified, and I'm easily able to reproduce the failure. Victor
On Thu, 7 Jun 2018 at 06:08 Victor Stinner <vstinner@redhat.com> wrote:
I'll leave that call up to you since you're putting in the work. If you disable the test then I'm comfortable with making AppVeyor required again, otherwise with a failure rate that high we shouldn't require any Windows status check.
Update: I fixed the root issue of https://bugs.python.org/issue33694 it was a race condition in ProactorEventLoop, so specific to Windows. test_asyncio *should* now pass on all Windows CIs. I leave the issue open a few days to check if the issue is really fixed. Victor 2018-06-07 18:38 GMT+02:00 Brett Cannon <brett@python.org>:
participants (4)
-
Brett Cannon -
Ned Deily -
Stephane Wirtel -
Victor Stinner