We had to turn off AppVeyor as a requirement today due to it not running any builds for the last 20 hours. Ned Deily suggested that since VSTS on at least Windows has stabilized that maybe we should switch off AppVeyor off entirely and make the Windows VSTS builder required.
Since https://python.visualstudio.com/cpython/_build/index?context=mine&path=%... shows the builder has been stable I would like to make this required for master today/tomorrow and then flip it on for the other 3.x branches on Friday if we don't see any instability.
Any objections/opinions about doing this?
On Jun 4, 2018, at 14:34, Brett Cannon brett@python.org wrote:
We had to turn off AppVeyor as a requirement today due to it not running any builds for the last 20 hours. Ned Deily suggested that since VSTS on at least Windows has stabilized that maybe we should switch off AppVeyor off entirely and make the Windows VSTS builder required.
Since https://python.visualstudio.com/cpython/_build/index?context=mine&path=%... shows the builder has been stable I would like to make this required for master today/tomorrow and then flip it on for the other 3.x branches on Friday if we don't see any instability.
Any objections/opinions about doing this?
LGTM, although since next Monday is the cutoff for 3.7.0rc1 and 3.6.6rc1, perhaps we can wait until next Wednesday to make the change for those branches - just in case.
-- Ned Deily nad@python.org -- []
On Mon, 4 Jun 2018 at 11:56 Ned Deily nad@python.org wrote:
On Jun 4, 2018, at 14:34, Brett Cannon brett@python.org wrote:
We had to turn off AppVeyor as a requirement today due to it not running
any builds for the last 20 hours. Ned Deily suggested that since VSTS on at least Windows has stabilized that maybe we should switch off AppVeyor off entirely and make the Windows VSTS builder required.
Since
https://python.visualstudio.com/cpython/_build/index?context=mine&path=%... shows the builder has been stable I would like to make this required for master today/tomorrow and then flip it on for the other 3.x branches on Friday if we don't see any instability.
Any objections/opinions about doing this?
LGTM, although since next Monday is the cutoff for 3.7.0rc1 and 3.6.6rc1, perhaps we can wait until next Wednesday to make the change for those branches - just in case.
Then let's push this out to considering trying VSTS as required on master this Friday and then waiting a week for other branches.
There are some updates on the AppVeyor side: https://mail.python.org/pipermail/python-committers/2018-June/005549.html
Victor
2018-06-04 20:34 GMT+02:00 Brett Cannon brett@python.org:
We had to turn off AppVeyor as a requirement today due to it not running any builds for the last 20 hours. Ned Deily suggested that since VSTS on at least Windows has stabilized that maybe we should switch off AppVeyor off entirely and make the Windows VSTS builder required.
Since https://python.visualstudio.com/cpython/_build/index?context=mine&path=%... shows the builder has been stable I would like to make this required for master today/tomorrow and then flip it on for the other 3.x branches on Friday if we don't see any instability.
Any objections/opinions about doing this?
core-workflow mailing list -- core-workflow@python.org To unsubscribe send an email to core-workflow-leave@python.org https://mail.python.org/mm3/mailman3/lists/core-workflow.python.org/ This list is governed by the PSF Code of Conduct: https://www.python.org/psf/codeofconduct
2018-06-04 20:34 GMT+02:00 Brett Cannon brett@python.org:
Since https://python.visualstudio.com/cpython/_build/index?context=mine&path=%... shows the builder has been stable
Just to be pedantic, it's not 100% stable yet :-) test_asyncio strikes back: https://bugs.python.org/issue33694#msg318705
I see two 3.7 and master builds which failed because of this issue.
Moreover, it seems like the regrtest -w option is missing: failing tests are not re-run in verbose mode.
Victor
On Mon, 4 Jun 2018 at 15:58 Victor Stinner vstinner@redhat.com wrote:
2018-06-04 20:34 GMT+02:00 Brett Cannon brett@python.org:
Since
https://python.visualstudio.com/cpython/_build/index?context=mine&path=%...
shows the builder has been stable
Just to be pedantic, it's not 100% stable yet :-) test_asyncio strikes back: https://bugs.python.org/issue33694#msg318705
I see two 3.7 and master builds which failed because of this issue.
Right, but those aren't special to VSTS, correct?
Moreover, it seems like the regrtest -w option is missing: failing tests are not re-run in verbose mode.
Did you want to propose a PR to update the YAML files in the .vsts directory to start using -w?
Steve has no plan to support VSTS on Python 2.7:
"Correct, and I wasn't planning on it. The default VSTS machines don’t have the right compiler, and I’m not about to start managing VMs for 2.7 at this stage."
So I would suggest to keep AppVeyor for Python 2.7, at least for the short term, and maybe make it mandatory again since it works again we doubled our capacity (1 job => 2 jobs in parallel)!
The workflow velocity is less impact for Python 2.7 which gets less changes than 3.x & master branches.
Victor
2018-06-04 20:34 GMT+02:00 Brett Cannon brett@python.org:
We had to turn off AppVeyor as a requirement today due to it not running any builds for the last 20 hours. Ned Deily suggested that since VSTS on at least Windows has stabilized that maybe we should switch off AppVeyor off entirely and make the Windows VSTS builder required.
Since https://python.visualstudio.com/cpython/_build/index?context=mine&path=%... shows the builder has been stable I would like to make this required for master today/tomorrow and then flip it on for the other 3.x branches on Friday if we don't see any instability.
Any objections/opinions about doing this?
core-workflow mailing list -- core-workflow@python.org To unsubscribe send an email to core-workflow-leave@python.org https://mail.python.org/mm3/mailman3/lists/core-workflow.python.org/ This list is governed by the PSF Code of Conduct: https://www.python.org/psf/codeofconduct
On Tue, 5 Jun 2018 at 03:52 Victor Stinner vstinner@redhat.com wrote:
Steve has no plan to support VSTS on Python 2.7:
"Correct, and I wasn't planning on it. The default VSTS machines don’t have the right compiler, and I’m not about to start managing VMs for 2.7 at this stage."
So I would suggest to keep AppVeyor for Python 2.7, at least for the short term, and maybe make it mandatory again since it works again we doubled our capacity (1 job => 2 jobs in parallel)!
The workflow velocity is less impact for Python 2.7 which gets less changes than 3.x & master branches.
Yep, that's what I was thinking of doing on Friday.
-Brett
Victor
2018-06-04 20:34 GMT+02:00 Brett Cannon brett@python.org:
We had to turn off AppVeyor as a requirement today due to it not running
any
builds for the last 20 hours. Ned Deily suggested that since VSTS on at least Windows has stabilized that maybe we should switch off AppVeyor off entirely and make the Windows VSTS builder required.
Since
https://python.visualstudio.com/cpython/_build/index?context=mine&path=%...
shows the builder has been stable I would like to make this required for master today/tomorrow and then flip it on for the other 3.x branches on Friday if we don't see any instability.
Any objections/opinions about doing this?
core-workflow mailing list -- core-workflow@python.org To unsubscribe send an email to core-workflow-leave@python.org https://mail.python.org/mm3/mailman3/lists/core-workflow.python.org/ This list is governed by the PSF Code of Conduct: https://www.python.org/psf/codeofconduct
IHMO this issue is a blocker to make VSTS mandatory: https://bugs.python.org/issue33782
VSTS Windows-PR failed with "The request was failed due to an internal service error. Please try again."
It would prevent to not be blocked by an internal VSTS error :-(
Victor
2018-06-05 20:07 GMT+02:00 Brett Cannon brett@python.org:
On Tue, 5 Jun 2018 at 03:52 Victor Stinner vstinner@redhat.com wrote:
Steve has no plan to support VSTS on Python 2.7:
"Correct, and I wasn't planning on it. The default VSTS machines don’t have the right compiler, and I’m not about to start managing VMs for 2.7 at this stage."
So I would suggest to keep AppVeyor for Python 2.7, at least for the short term, and maybe make it mandatory again since it works again we doubled our capacity (1 job => 2 jobs in parallel)!
The workflow velocity is less impact for Python 2.7 which gets less changes than 3.x & master branches.
Yep, that's what I was thinking of doing on Friday.
-Brett
Victor
2018-06-04 20:34 GMT+02:00 Brett Cannon brett@python.org:
We had to turn off AppVeyor as a requirement today due to it not running any builds for the last 20 hours. Ned Deily suggested that since VSTS on at least Windows has stabilized that maybe we should switch off AppVeyor off entirely and make the Windows VSTS builder required.
Since
https://python.visualstudio.com/cpython/_build/index?context=mine&path=%... shows the builder has been stable I would like to make this required for master today/tomorrow and then flip it on for the other 3.x branches on Friday if we don't see any instability.
Any objections/opinions about doing this?
core-workflow mailing list -- core-workflow@python.org To unsubscribe send an email to core-workflow-leave@python.org https://mail.python.org/mm3/mailman3/lists/core-workflow.python.org/ This list is governed by the PSF Code of Conduct: https://www.python.org/psf/codeofconduct
On 06/06, Victor Stinner wrote:
IHMO this issue is a blocker to make VSTS mandatory: https://bugs.python.org/issue33782
VSTS Windows-PR failed with "The request was failed due to an internal service error. Please try again."
Victor, do we have a traceback of the build or just this message?
It would prevent to not be blocked by an internal VSTS error :-(
It's an error intern to VSTS, it's unrelated to Python. There is no log. Don't worry, Steve Dower already passed the bug to the VSTS team.
Victor
2018-06-06 15:28 GMT+02:00 Stephane Wirtel stephane@wirtel.be:
On 06/06, Victor Stinner wrote:
IHMO this issue is a blocker to make VSTS mandatory: https://bugs.python.org/issue33782
VSTS Windows-PR failed with "The request was failed due to an internal service error. Please try again."
Victor, do we have a traceback of the build or just this message?
It would prevent to not be blocked by an internal VSTS error :-(
-- Stéphane Wirtel - http://wirtel.be - @matrixise
On Wed, 6 Jun 2018 at 04:16 Victor Stinner vstinner@redhat.com wrote:
IHMO this issue is a blocker to make VSTS mandatory: https://bugs.python.org/issue33782
VSTS Windows-PR failed with "The request was failed due to an internal service error. Please try again."
It would prevent to not be blocked by an internal VSTS error :-(
While I agree it should be looked into (which I know it is as I'm cc'ed on the internal emails), I don't think making it a blocker is quite fair. If blocking making VSTS required for a couple of failed builds due to an internal error that could be kicked off by closing and re-opening a PR then I don't know if it's reasonable to require AppVeyor again when they were down for all PRs for 20 hours.
My point is this is all software which we know isn't perfect and as long as there is a way to kick off a wedged/failed build again I don't think it's a hard blocker.
-Brett
Victor
2018-06-05 20:07 GMT+02:00 Brett Cannon brett@python.org:
On Tue, 5 Jun 2018 at 03:52 Victor Stinner vstinner@redhat.com wrote:
Steve has no plan to support VSTS on Python 2.7:
"Correct, and I wasn't planning on it. The default VSTS machines don’t have the right compiler, and I’m not about to start managing VMs for 2.7 at this stage."
So I would suggest to keep AppVeyor for Python 2.7, at least for the short term, and maybe make it mandatory again since it works again we doubled our capacity (1 job => 2 jobs in parallel)!
The workflow velocity is less impact for Python 2.7 which gets less changes than 3.x & master branches.
Yep, that's what I was thinking of doing on Friday.
-Brett
Victor
2018-06-04 20:34 GMT+02:00 Brett Cannon brett@python.org:
We had to turn off AppVeyor as a requirement today due to it not
running
any builds for the last 20 hours. Ned Deily suggested that since VSTS on
at
least Windows has stabilized that maybe we should switch off AppVeyor off entirely and make the Windows VSTS builder required.
Since
https://python.visualstudio.com/cpython/_build/index?context=mine&path=%...
shows the builder has been stable I would like to make this required
for
master today/tomorrow and then flip it on for the other 3.x branches
on
Friday if we don't see any instability.
Any objections/opinions about doing this?
core-workflow mailing list -- core-workflow@python.org To unsubscribe send an email to core-workflow-leave@python.org https://mail.python.org/mm3/mailman3/lists/core-workflow.python.org/ This list is governed by the PSF Code of Conduct: https://www.python.org/psf/codeofconduct
2018-06-06 19:37 GMT+02:00 Brett Cannon brett@python.org:
My point is this is all software which we know isn't perfect and as long as there is a way to kick off a wedged/failed build again I don't think it's a hard blocker.
Ok, let me elaborate.
I'm trying to stabilize Travis CI, AppVeyor and buildbots for 1 month. I got some help on some issues, but sadly too few core devs are working on making CIs more stable, whereas CIs are blocking our workflow when they fail. It's annoying when a PR is blocked by a failure which is unrelated to the proposed change.
In 1 month, I already saw significant progress, but there are still a few open issues: https://mail.python.org/pipermail/python-dev/2018-June/153796.html
My point is that making VSTS mandatory adds more issues. I would prefer to have more time to fix existing *known* issues first. I'm not talking about VSTS issues in particular, but all CI known issues.
In the meanwhile, AppVeyor seems stable again. Would it be possible to re-enable it in all stable branches?
Victor
On Wed, Jun 6, 2018, 15:48 Victor Stinner, vstinner@redhat.com wrote:
2018-06-06 19:37 GMT+02:00 Brett Cannon brett@python.org:
My point is this is all software which we know isn't perfect and as long
as
there is a way to kick off a wedged/failed build again I don't think
it's a
hard blocker.
Ok, let me elaborate.
I'm trying to stabilize Travis CI, AppVeyor and buildbots for 1 month. I got some help on some issues, but sadly too few core devs are working on making CIs more stable, whereas CIs are blocking our workflow when they fail. It's annoying when a PR is blocked by a failure which is unrelated to the proposed change.
In 1 month, I already saw significant progress, but there are still a few open issues: https://mail.python.org/pipermail/python-dev/2018-June/153796.html
My point is that making VSTS mandatory adds more issues. I would prefer to have more time to fix existing *known* issues first. I'm not talking about VSTS issues in particular, but all CI known issues.
Fair enough!
In the meanwhile, AppVeyor seems stable again. Would it be possible to re-enable it in all stable branches?
It was never turned off, but if you mean make it required again then yes.
2018-06-07 5:36 GMT+02:00 Brett Cannon brett@python.org:
In the meanwhile, AppVeyor seems stable again. Would it be possible to re-enable it in all stable branches?
It was never turned off, but if you mean make it required again then yes.
Yes, I mean making it required again... but yesterday I made progress on the super annoying test_asyncio random failure, and it seems like a major regression in ProactorEventLoop on Windows: https://bugs.python.org/issue33694
Calling pause_reading() / resume_reading() of a proactor transport can lead to data loss :-(
Right now, I would prefer to not make any Windows CI required, until the test is either fixed or skipped. The failure rate is way too high (25% at least, maybe even 50%?).
I'm working on a fix, but it's super complex :-( I would need an IOCP guru and an asyncio guru to help me here. Maybe the short term fix is to first skip the test, since the bug is know well identified, and I'm easily able to reproduce the failure.
Victor
On Thu, 7 Jun 2018 at 06:08 Victor Stinner vstinner@redhat.com wrote:
2018-06-07 5:36 GMT+02:00 Brett Cannon brett@python.org:
In the meanwhile, AppVeyor seems stable again. Would it be possible to re-enable it in all stable branches?
It was never turned off, but if you mean make it required again then yes.
Yes, I mean making it required again... but yesterday I made progress on the super annoying test_asyncio random failure, and it seems like a major regression in ProactorEventLoop on Windows: https://bugs.python.org/issue33694
Calling pause_reading() / resume_reading() of a proactor transport can lead to data loss :-(
Right now, I would prefer to not make any Windows CI required, until the test is either fixed or skipped. The failure rate is way too high (25% at least, maybe even 50%?).
I'm working on a fix, but it's super complex :-( I would need an IOCP guru and an asyncio guru to help me here. Maybe the short term fix is to first skip the test, since the bug is know well identified, and I'm easily able to reproduce the failure.
I'll leave that call up to you since you're putting in the work. If you disable the test then I'm comfortable with making AppVeyor required again, otherwise with a failure rate that high we shouldn't require any Windows status check.
Update: I fixed the root issue of https://bugs.python.org/issue33694 it was a race condition in ProactorEventLoop, so specific to Windows.
test_asyncio *should* now pass on all Windows CIs. I leave the issue open a few days to check if the issue is really fixed.
Victor
2018-06-07 18:38 GMT+02:00 Brett Cannon brett@python.org:
On Thu, 7 Jun 2018 at 06:08 Victor Stinner vstinner@redhat.com wrote:
2018-06-07 5:36 GMT+02:00 Brett Cannon brett@python.org:
In the meanwhile, AppVeyor seems stable again. Would it be possible to re-enable it in all stable branches?
It was never turned off, but if you mean make it required again then yes.
Yes, I mean making it required again... but yesterday I made progress on the super annoying test_asyncio random failure, and it seems like a major regression in ProactorEventLoop on Windows: https://bugs.python.org/issue33694
Calling pause_reading() / resume_reading() of a proactor transport can lead to data loss :-(
Right now, I would prefer to not make any Windows CI required, until the test is either fixed or skipped. The failure rate is way too high (25% at least, maybe even 50%?).
I'm working on a fix, but it's super complex :-( I would need an IOCP guru and an asyncio guru to help me here. Maybe the short term fix is to first skip the test, since the bug is know well identified, and I'm easily able to reproduce the failure.
I'll leave that call up to you since you're putting in the work. If you disable the test then I'm comfortable with making AppVeyor required again, otherwise with a failure rate that high we shouldn't require any Windows status check.
-
Brett Cannon -
Ned Deily -
Stephane Wirtel -
Victor Stinner