Time to switch off AppVeyor and start requiring VSTS for Windows?
We had to turn off AppVeyor as a requirement today due to it not running any builds for the last 20 hours. Ned Deily suggested that since VSTS on at least Windows has stabilized that maybe we should switch off AppVeyor off entirely and make the Windows VSTS builder required. Since https://python.visualstudio.com/cpython/_build/index?context=mine&path=%5C&definitionId=4&_a=history shows the builder has been stable I would like to make this required for master today/tomorrow and then flip it on for the other 3.x branches on Friday if we don't see any instability. Any objections/opinions about doing this?
On Jun 4, 2018, at 14:34, Brett Cannon <brett@python.org> wrote:
We had to turn off AppVeyor as a requirement today due to it not running any builds for the last 20 hours. Ned Deily suggested that since VSTS on at least Windows has stabilized that maybe we should switch off AppVeyor off entirely and make the Windows VSTS builder required.
Since https://python.visualstudio.com/cpython/_build/index?context=mine&path=%5C&definitionId=4&_a=history shows the builder has been stable I would like to make this required for master today/tomorrow and then flip it on for the other 3.x branches on Friday if we don't see any instability.
Any objections/opinions about doing this?
LGTM, although since next Monday is the cutoff for 3.7.0rc1 and 3.6.6rc1, perhaps we can wait until next Wednesday to make the change for those branches - just in case. -- Ned Deily nad@python.org -- []
On Mon, 4 Jun 2018 at 11:56 Ned Deily <nad@python.org> wrote:
On Jun 4, 2018, at 14:34, Brett Cannon <brett@python.org> wrote:
We had to turn off AppVeyor as a requirement today due to it not running any builds for the last 20 hours. Ned Deily suggested that since VSTS on at least Windows has stabilized that maybe we should switch off AppVeyor off entirely and make the Windows VSTS builder required.
Since https://python.visualstudio.com/cpython/_build/index?context=mine&path=%5C&definitionId=4&_a=history shows the builder has been stable I would like to make this required for master today/tomorrow and then flip it on for the other 3.x branches on Friday if we don't see any instability.
Any objections/opinions about doing this?
LGTM, although since next Monday is the cutoff for 3.7.0rc1 and 3.6.6rc1, perhaps we can wait until next Wednesday to make the change for those branches - just in case.
Then let's push this out to considering trying VSTS as required on master this Friday and then waiting a week for other branches.
There are some updates on the AppVeyor side: https://mail.python.org/pipermail/python-committers/2018-June/005549.html Victor 2018-06-04 20:34 GMT+02:00 Brett Cannon <brett@python.org>:
We had to turn off AppVeyor as a requirement today due to it not running any builds for the last 20 hours. Ned Deily suggested that since VSTS on at least Windows has stabilized that maybe we should switch off AppVeyor off entirely and make the Windows VSTS builder required.
Since https://python.visualstudio.com/cpython/_build/index?context=mine&path=%5C&definitionId=4&_a=history shows the builder has been stable I would like to make this required for master today/tomorrow and then flip it on for the other 3.x branches on Friday if we don't see any instability.
Any objections/opinions about doing this?
_______________________________________________ core-workflow mailing list -- core-workflow@python.org To unsubscribe send an email to core-workflow-leave@python.org https://mail.python.org/mm3/mailman3/lists/core-workflow.python.org/ This list is governed by the PSF Code of Conduct: https://www.python.org/psf/codeofconduct
2018-06-04 20:34 GMT+02:00 Brett Cannon <brett@python.org>:
Since https://python.visualstudio.com/cpython/_build/index?context=mine&path=%5C&definitionId=4&_a=history shows the builder has been stable
Just to be pedantic, it's not 100% stable yet :-) test_asyncio strikes back: https://bugs.python.org/issue33694#msg318705 I see two 3.7 and master builds which failed because of this issue. Moreover, it seems like the regrtest -w option is missing: failing tests are not re-run in verbose mode. Victor
On Mon, 4 Jun 2018 at 15:58 Victor Stinner <vstinner@redhat.com> wrote:
2018-06-04 20:34 GMT+02:00 Brett Cannon <brett@python.org>:
Since
https://python.visualstudio.com/cpython/_build/index?context=mine&path=%5C&definitionId=4&_a=history
shows the builder has been stable
Just to be pedantic, it's not 100% stable yet :-) test_asyncio strikes back: https://bugs.python.org/issue33694#msg318705
I see two 3.7 and master builds which failed because of this issue.
Right, but those aren't special to VSTS, correct?
Moreover, it seems like the regrtest -w option is missing: failing tests are not re-run in verbose mode.
Did you want to propose a PR to update the YAML files in the .vsts directory to start using -w?
Steve has no plan to support VSTS on Python 2.7: "Correct, and I wasn't planning on it. The default VSTS machines don’t have the right compiler, and I’m not about to start managing VMs for 2.7 at this stage." So I would suggest to keep AppVeyor for Python 2.7, at least for the short term, and maybe make it mandatory again since it works again we doubled our capacity (1 job => 2 jobs in parallel)! The workflow velocity is less impact for Python 2.7 which gets less changes than 3.x & master branches. Victor 2018-06-04 20:34 GMT+02:00 Brett Cannon <brett@python.org>:
We had to turn off AppVeyor as a requirement today due to it not running any builds for the last 20 hours. Ned Deily suggested that since VSTS on at least Windows has stabilized that maybe we should switch off AppVeyor off entirely and make the Windows VSTS builder required.
Since https://python.visualstudio.com/cpython/_build/index?context=mine&path=%5C&definitionId=4&_a=history shows the builder has been stable I would like to make this required for master today/tomorrow and then flip it on for the other 3.x branches on Friday if we don't see any instability.
Any objections/opinions about doing this?
_______________________________________________ core-workflow mailing list -- core-workflow@python.org To unsubscribe send an email to core-workflow-leave@python.org https://mail.python.org/mm3/mailman3/lists/core-workflow.python.org/ This list is governed by the PSF Code of Conduct: https://www.python.org/psf/codeofconduct
On Tue, 5 Jun 2018 at 03:52 Victor Stinner <vstinner@redhat.com> wrote:
Steve has no plan to support VSTS on Python 2.7:
"Correct, and I wasn't planning on it. The default VSTS machines don’t have the right compiler, and I’m not about to start managing VMs for 2.7 at this stage."
So I would suggest to keep AppVeyor for Python 2.7, at least for the short term, and maybe make it mandatory again since it works again we doubled our capacity (1 job => 2 jobs in parallel)!
The workflow velocity is less impact for Python 2.7 which gets less changes than 3.x & master branches.
Yep, that's what I was thinking of doing on Friday. -Brett
Victor
2018-06-04 20:34 GMT+02:00 Brett Cannon <brett@python.org>:
We had to turn off AppVeyor as a requirement today due to it not running any builds for the last 20 hours. Ned Deily suggested that since VSTS on at least Windows has stabilized that maybe we should switch off AppVeyor off entirely and make the Windows VSTS builder required.
Since
https://python.visualstudio.com/cpython/_build/index?context=mine&path=%5C&definitionId=4&_a=history
shows the builder has been stable I would like to make this required for master today/tomorrow and then flip it on for the other 3.x branches on Friday if we don't see any instability.
Any objections/opinions about doing this?
_______________________________________________ core-workflow mailing list -- core-workflow@python.org To unsubscribe send an email to core-workflow-leave@python.org https://mail.python.org/mm3/mailman3/lists/core-workflow.python.org/ This list is governed by the PSF Code of Conduct: https://www.python.org/psf/codeofconduct
IHMO this issue is a blocker to make VSTS mandatory: https://bugs.python.org/issue33782 VSTS Windows-PR failed with "The request was failed due to an internal service error. Please try again." It would prevent to not be blocked by an internal VSTS error :-( Victor 2018-06-05 20:07 GMT+02:00 Brett Cannon <brett@python.org>:
On Tue, 5 Jun 2018 at 03:52 Victor Stinner <vstinner@redhat.com> wrote:
Steve has no plan to support VSTS on Python 2.7:
"Correct, and I wasn't planning on it. The default VSTS machines don’t have the right compiler, and I’m not about to start managing VMs for 2.7 at this stage."
So I would suggest to keep AppVeyor for Python 2.7, at least for the short term, and maybe make it mandatory again since it works again we doubled our capacity (1 job => 2 jobs in parallel)!
The workflow velocity is less impact for Python 2.7 which gets less changes than 3.x & master branches.
Yep, that's what I was thinking of doing on Friday.
-Brett
Victor
2018-06-04 20:34 GMT+02:00 Brett Cannon <brett@python.org>:
We had to turn off AppVeyor as a requirement today due to it not running any builds for the last 20 hours. Ned Deily suggested that since VSTS on at least Windows has stabilized that maybe we should switch off AppVeyor off entirely and make the Windows VSTS builder required.
Since
https://python.visualstudio.com/cpython/_build/index?context=mine&path=%5C&definitionId=4&_a=history shows the builder has been stable I would like to make this required for master today/tomorrow and then flip it on for the other 3.x branches on Friday if we don't see any instability.
Any objections/opinions about doing this?
_______________________________________________ core-workflow mailing list -- core-workflow@python.org To unsubscribe send an email to core-workflow-leave@python.org https://mail.python.org/mm3/mailman3/lists/core-workflow.python.org/ This list is governed by the PSF Code of Conduct: https://www.python.org/psf/codeofconduct
On 06/06, Victor Stinner wrote:
IHMO this issue is a blocker to make VSTS mandatory: https://bugs.python.org/issue33782
VSTS Windows-PR failed with "The request was failed due to an internal service error. Please try again." Victor, do we have a traceback of the build or just this message?
It would prevent to not be blocked by an internal VSTS error :-(
-- Stéphane Wirtel - http://wirtel.be - @matrixise
It's an error intern to VSTS, it's unrelated to Python. There is no log. Don't worry, Steve Dower already passed the bug to the VSTS team. Victor 2018-06-06 15:28 GMT+02:00 Stephane Wirtel <stephane@wirtel.be>:
On 06/06, Victor Stinner wrote:
IHMO this issue is a blocker to make VSTS mandatory: https://bugs.python.org/issue33782
VSTS Windows-PR failed with "The request was failed due to an internal service error. Please try again."
Victor, do we have a traceback of the build or just this message?
It would prevent to not be blocked by an internal VSTS error :-(
-- Stéphane Wirtel - http://wirtel.be - @matrixise
On Wed, 6 Jun 2018 at 04:16 Victor Stinner <vstinner@redhat.com> wrote:
IHMO this issue is a blocker to make VSTS mandatory: https://bugs.python.org/issue33782
VSTS Windows-PR failed with "The request was failed due to an internal service error. Please try again."
It would prevent to not be blocked by an internal VSTS error :-(
While I agree it should be looked into (which I know it is as I'm cc'ed on the internal emails), I don't think making it a blocker is quite fair. If blocking making VSTS required for a couple of failed builds due to an internal error that could be kicked off by closing and re-opening a PR then I don't know if it's reasonable to require AppVeyor again when they were down for all PRs for 20 hours. My point is this is all software which we know isn't perfect and as long as there is a way to kick off a wedged/failed build again I don't think it's a hard blocker. -Brett
Victor
2018-06-05 20:07 GMT+02:00 Brett Cannon <brett@python.org>:
On Tue, 5 Jun 2018 at 03:52 Victor Stinner <vstinner@redhat.com> wrote:
Steve has no plan to support VSTS on Python 2.7:
"Correct, and I wasn't planning on it. The default VSTS machines don’t have the right compiler, and I’m not about to start managing VMs for 2.7 at this stage."
So I would suggest to keep AppVeyor for Python 2.7, at least for the short term, and maybe make it mandatory again since it works again we doubled our capacity (1 job => 2 jobs in parallel)!
The workflow velocity is less impact for Python 2.7 which gets less changes than 3.x & master branches.
Yep, that's what I was thinking of doing on Friday.
-Brett
Victor
2018-06-04 20:34 GMT+02:00 Brett Cannon <brett@python.org>:
We had to turn off AppVeyor as a requirement today due to it not
running
any builds for the last 20 hours. Ned Deily suggested that since VSTS on at least Windows has stabilized that maybe we should switch off AppVeyor off entirely and make the Windows VSTS builder required.
Since
https://python.visualstudio.com/cpython/_build/index?context=mine&path=%5C&definitionId=4&_a=history
shows the builder has been stable I would like to make this required for master today/tomorrow and then flip it on for the other 3.x branches on Friday if we don't see any instability.
Any objections/opinions about doing this?
_______________________________________________ core-workflow mailing list -- core-workflow@python.org To unsubscribe send an email to core-workflow-leave@python.org https://mail.python.org/mm3/mailman3/lists/core-workflow.python.org/ This list is governed by the PSF Code of Conduct: https://www.python.org/psf/codeofconduct
2018-06-06 19:37 GMT+02:00 Brett Cannon <brett@python.org>:
My point is this is all software which we know isn't perfect and as long as there is a way to kick off a wedged/failed build again I don't think it's a hard blocker.
Ok, let me elaborate. I'm trying to stabilize Travis CI, AppVeyor and buildbots for 1 month. I got some help on some issues, but sadly too few core devs are working on making CIs more stable, whereas CIs are blocking our workflow when they fail. It's annoying when a PR is blocked by a failure which is unrelated to the proposed change. In 1 month, I already saw significant progress, but there are still a few open issues: https://mail.python.org/pipermail/python-dev/2018-June/153796.html My point is that making VSTS mandatory adds more issues. I would prefer to have more time to fix existing *known* issues first. I'm not talking about VSTS issues in particular, but all CI known issues. In the meanwhile, AppVeyor seems stable again. Would it be possible to re-enable it in all stable branches? Victor
On Wed, Jun 6, 2018, 15:48 Victor Stinner, <vstinner@redhat.com> wrote:
2018-06-06 19:37 GMT+02:00 Brett Cannon <brett@python.org>:
My point is this is all software which we know isn't perfect and as long as there is a way to kick off a wedged/failed build again I don't think it's a hard blocker.
Ok, let me elaborate.
I'm trying to stabilize Travis CI, AppVeyor and buildbots for 1 month. I got some help on some issues, but sadly too few core devs are working on making CIs more stable, whereas CIs are blocking our workflow when they fail. It's annoying when a PR is blocked by a failure which is unrelated to the proposed change.
In 1 month, I already saw significant progress, but there are still a few open issues: https://mail.python.org/pipermail/python-dev/2018-June/153796.html
My point is that making VSTS mandatory adds more issues. I would prefer to have more time to fix existing *known* issues first. I'm not talking about VSTS issues in particular, but all CI known issues.
Fair enough!
In the meanwhile, AppVeyor seems stable again. Would it be possible to re-enable it in all stable branches?
It was never turned off, but if you mean make it required again then yes.
2018-06-07 5:36 GMT+02:00 Brett Cannon <brett@python.org>:
In the meanwhile, AppVeyor seems stable again. Would it be possible to re-enable it in all stable branches?
It was never turned off, but if you mean make it required again then yes.
Yes, I mean making it required again... but yesterday I made progress on the super annoying test_asyncio random failure, and it seems like a major regression in ProactorEventLoop on Windows: https://bugs.python.org/issue33694 Calling pause_reading() / resume_reading() of a proactor transport can lead to data loss :-( Right now, I would prefer to not make any Windows CI required, until the test is either fixed or skipped. The failure rate is way too high (25% at least, maybe even 50%?). I'm working on a fix, but it's super complex :-( I would need an IOCP guru and an asyncio guru to help me here. Maybe the short term fix is to first skip the test, since the bug is know well identified, and I'm easily able to reproduce the failure. Victor
On Thu, 7 Jun 2018 at 06:08 Victor Stinner <vstinner@redhat.com> wrote:
2018-06-07 5:36 GMT+02:00 Brett Cannon <brett@python.org>:
In the meanwhile, AppVeyor seems stable again. Would it be possible to re-enable it in all stable branches?
It was never turned off, but if you mean make it required again then yes.
Yes, I mean making it required again... but yesterday I made progress on the super annoying test_asyncio random failure, and it seems like a major regression in ProactorEventLoop on Windows: https://bugs.python.org/issue33694
Calling pause_reading() / resume_reading() of a proactor transport can lead to data loss :-(
Right now, I would prefer to not make any Windows CI required, until the test is either fixed or skipped. The failure rate is way too high (25% at least, maybe even 50%?).
I'm working on a fix, but it's super complex :-( I would need an IOCP guru and an asyncio guru to help me here. Maybe the short term fix is to first skip the test, since the bug is know well identified, and I'm easily able to reproduce the failure.
I'll leave that call up to you since you're putting in the work. If you disable the test then I'm comfortable with making AppVeyor required again, otherwise with a failure rate that high we shouldn't require any Windows status check.
Update: I fixed the root issue of https://bugs.python.org/issue33694 it was a race condition in ProactorEventLoop, so specific to Windows. test_asyncio *should* now pass on all Windows CIs. I leave the issue open a few days to check if the issue is really fixed. Victor 2018-06-07 18:38 GMT+02:00 Brett Cannon <brett@python.org>:
On Thu, 7 Jun 2018 at 06:08 Victor Stinner <vstinner@redhat.com> wrote:
2018-06-07 5:36 GMT+02:00 Brett Cannon <brett@python.org>:
In the meanwhile, AppVeyor seems stable again. Would it be possible to re-enable it in all stable branches?
It was never turned off, but if you mean make it required again then yes.
Yes, I mean making it required again... but yesterday I made progress on the super annoying test_asyncio random failure, and it seems like a major regression in ProactorEventLoop on Windows: https://bugs.python.org/issue33694
Calling pause_reading() / resume_reading() of a proactor transport can lead to data loss :-(
Right now, I would prefer to not make any Windows CI required, until the test is either fixed or skipped. The failure rate is way too high (25% at least, maybe even 50%?).
I'm working on a fix, but it's super complex :-( I would need an IOCP guru and an asyncio guru to help me here. Maybe the short term fix is to first skip the test, since the bug is know well identified, and I'm easily able to reproduce the failure.
I'll leave that call up to you since you're putting in the work. If you disable the test then I'm comfortable with making AppVeyor required again, otherwise with a failure rate that high we shouldn't require any Windows status check.
participants (4)
-
Brett Cannon
-
Ned Deily
-
Stephane Wirtel
-
Victor Stinner