The current thinking/plans around VSTS as our CI
Since both Paul Moore and Antoine Pitrou started to ask questions about a side comment I made about VSTS, I might as well start a discussion (Steve has also *just* emailed python-committers about this topic). Basically Steve Dower and I have been working directly with the VSTS team to improve their Python support. Along the way they have announced they are adding anonymous access to build logs which was always a major blocker for using it in open source. VSTS gave us early access to the anonymous access that they are launching along with 20 concurrent builders which is more than what we currently have on Travis and AppVeyor. At this point Steve has added the appropriate YAML files in the .vsts directory that control the build jobs so we can test out the integration and see what the performance is like. But based on what we have been seeing there will be no queue in getting PRs tested and that applies to Windows, macOS, and Linux (IOW better coverage and no more issues about performance or having to wait too long for CI to go green ;) . There is also other benefits like making at least Windows builds easier to release as signing those builds can now be automated. My expectation is that once we are convinced that the VSTS builds are passing consistently and everything is working as everyone wants for e.g. a week, we can plan to turn off Travis and AppVeyor for master and 3.x branches (no one wants to bother with trying to get 2.7 to work on this and we can simply continue to rely on Travis and AppVeyor as necessary for the next 18 months). I don't think we want to run multiple CIs indefinitely on the same branch as it does take time and effort to keep CI green on a provider since they all have their own quirks. I should also mention that this free access is not limited to just CPython and any project can get free accounts and credits now, you just have to talk to Steve about getting in on the anonymous access before it launches to the general public (I have no timeline on when that's happening).
Hm... While I'm grateful for the donation, this sounds a bit like a coup. It looks as if MS trying to poach reference customers (like the well-respected CPython project) from AppVeyor and Travis-CI. Those are two small companies that have donated a lot of resources to many open source projects, and are also dependent on those open source projects as reference customers so they can sell to paying customers. I'm sure Brett and Steve are doing this for Python, but just remember there's no such thing as a free lunch. Who pays for those 20 machines? Is it coming out of MS's PR and marketing budget? --Guido On Thu, May 17, 2018 at 7:54 AM, Brett Cannon <brett@python.org> wrote:
Since both Paul Moore and Antoine Pitrou started to ask questions about a side comment I made about VSTS, I might as well start a discussion (Steve has also *just* emailed python-committers about this topic).
Basically Steve Dower and I have been working directly with the VSTS team to improve their Python support. Along the way they have announced they are adding anonymous access to build logs which was always a major blocker for using it in open source. VSTS gave us early access to the anonymous access that they are launching along with 20 concurrent builders which is more than what we currently have on Travis and AppVeyor.
At this point Steve has added the appropriate YAML files in the .vsts directory that control the build jobs so we can test out the integration and see what the performance is like. But based on what we have been seeing there will be no queue in getting PRs tested and that applies to Windows, macOS, and Linux (IOW better coverage and no more issues about performance or having to wait too long for CI to go green ;) . There is also other benefits like making at least Windows builds easier to release as signing those builds can now be automated.
My expectation is that once we are convinced that the VSTS builds are passing consistently and everything is working as everyone wants for e.g. a week, we can plan to turn off Travis and AppVeyor for master and 3.x branches (no one wants to bother with trying to get 2.7 to work on this and we can simply continue to rely on Travis and AppVeyor as necessary for the next 18 months). I don't think we want to run multiple CIs indefinitely on the same branch as it does take time and effort to keep CI green on a provider since they all have their own quirks.
I should also mention that this free access is not limited to just CPython and any project can get free accounts and credits now, you just have to talk to Steve about getting in on the anonymous access before it launches to the general public (I have no timeline on when that's happening).
_______________________________________________ core-workflow mailing list -- core-workflow@python.org To unsubscribe send an email to core-workflow-leave@python.org https://mail.python.org/mm3/mailman3/lists/core-workflow.python.org/ This list is governed by the PSF Code of Conduct: https://www.python.org/psf/codeofconduct
-- --Guido van Rossum (python.org/~guido)
On Thu, 17 May 2018 at 11:41 Guido van Rossum <guido@python.org> wrote:
Hm... While I'm grateful for the donation, this sounds a bit like a coup. It looks as if MS trying to poach reference customers (like the well-respected CPython project) from AppVeyor and Travis-CI.
This was an idea from Steve and me, not the VSTS team. I'll also mention that while we get some extra builders from Travis we get nothing special from AppVeyor.
Those are two small companies that have donated a lot of resources to many open source projects, and are also dependent on those open source projects as reference customers so they can sell to paying customers.
I'm sure Brett and Steve are doing this for Python, but just remember there's no such thing as a free lunch. Who pays for those 20 machines?
Is it coming out of MS's PR and marketing budget?
The extra credits are coming from VSTS itself (there is also the base free credits that anyone can get, so this whole thing isn't special to us). VSTS asked if we knew of any projects that might be interested in being involved in the preview, Steve and I said we were since we wanted the better performance for CPython, we then asked if we could have extra credits and they said sure, and here we are. IOW no one is pushing anything on us, we're proactively asking for this ourselves to address performance complaints I have received since our transition to GitHub due to Travis and AppVeyor taking a while to queue up builds for us (to the point that we don't depend on macOS builds because they historically take too long). And If people feel I'm not impartial enough then I'm fine with someone else making the decision to either leave around or turn off Travis and AppVeyor (Steve is already purposefully staying out of the final decision). As I said, I just know we have needed to put time into keeping Travis and AppVeyor happy so I figured if we ended up being happy with VSTS we would want to keep things simple. I'll also not quite sure why this is different than the credits we get from e.g. Heroku/Salesforce for the bots or AWS for PyPI? I get the worry of relying on a company that's too big to care if they upset us, but we already do rely on such companies so if this is to be a new policy then we have some de-tangling to do. And I do understand the worry if this was old MS, but this is new MS (i.e. the one I choose to work for ;) . I have been in multiple meetings with VPs who were worried about upsetting the open source community and I had to step in and tell them that they were worrying too much. :) IOW no one has done or is doing a sales pitch here; Steve and I just saw an opportunity to address a problem I have seen with our CI and so we decided to go for it with me being optimistic it's going to pay off. I apologize if people think we are overstepping, but honestly we're just excited to get faster CI and to actually measure the impact we have to do the initial integration and my excitement has made be see the bright side of life for this. ;) -Brett
--Guido
On Thu, May 17, 2018 at 7:54 AM, Brett Cannon <brett@python.org> wrote:
Since both Paul Moore and Antoine Pitrou started to ask questions about a side comment I made about VSTS, I might as well start a discussion (Steve has also *just* emailed python-committers about this topic).
Basically Steve Dower and I have been working directly with the VSTS team to improve their Python support. Along the way they have announced they are adding anonymous access to build logs which was always a major blocker for using it in open source. VSTS gave us early access to the anonymous access that they are launching along with 20 concurrent builders which is more than what we currently have on Travis and AppVeyor.
At this point Steve has added the appropriate YAML files in the .vsts directory that control the build jobs so we can test out the integration and see what the performance is like. But based on what we have been seeing there will be no queue in getting PRs tested and that applies to Windows, macOS, and Linux (IOW better coverage and no more issues about performance or having to wait too long for CI to go green ;) . There is also other benefits like making at least Windows builds easier to release as signing those builds can now be automated.
My expectation is that once we are convinced that the VSTS builds are passing consistently and everything is working as everyone wants for e.g. a week, we can plan to turn off Travis and AppVeyor for master and 3.x branches (no one wants to bother with trying to get 2.7 to work on this and we can simply continue to rely on Travis and AppVeyor as necessary for the next 18 months). I don't think we want to run multiple CIs indefinitely on the same branch as it does take time and effort to keep CI green on a provider since they all have their own quirks.
I should also mention that this free access is not limited to just CPython and any project can get free accounts and credits now, you just have to talk to Steve about getting in on the anonymous access before it launches to the general public (I have no timeline on when that's happening).
_______________________________________________ core-workflow mailing list -- core-workflow@python.org To unsubscribe send an email to core-workflow-leave@python.org https://mail.python.org/mm3/mailman3/lists/core-workflow.python.org/ This list is governed by the PSF Code of Conduct: https://www.python.org/psf/codeofconduct
-- --Guido van Rossum (python.org/~guido)
On 18 May 2018 at 02:44, Brett Cannon <brett@python.org> wrote:
I'll also not quite sure why this is different than the credits we get from e.g. Heroku/Salesforce for the bots or AWS for PyPI? I get the worry of relying on a company that's too big to care if they upset us, but we already do rely on such companies so if this is to be a new policy then we have some de-tangling to do.
Note that those relationships are generally getting formalised as PSF in-kind sponsorships, so that they're not completely dependent on personal relationships with folks that work there, and so there's likely to be some prior warning if credits are being withdrawn and the Infrastructure team needs to make alternate service hosting arrangements (see https://www.python.org/psf/sponsorship/sponsors/ - while that listing doesn't separate out cash sponsorships from in-kind service donations, it's possible to make a reasonably informed guess as to which is which). FWIW, switching from a hybrid Travis/Appveyor setup to a hybrid Travis/VSTS setup seems like it would be a reasonable way to go to me, since we have existing sponsorship arrangements in place with both MS and Travis CI, but no formal relationship with the Appveyor folks that I'm aware of (I believe we're just using their regular free tier). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Fri, May 18, 2018, 21:32 Nick Coghlan, <ncoghlan@gmail.com> wrote:
On 18 May 2018 at 02:44, Brett Cannon <brett@python.org> wrote:
I'll also not quite sure why this is different than the credits we get from e.g. Heroku/Salesforce for the bots or AWS for PyPI? I get the worry of relying on a company that's too big to care if they upset us, but we already do rely on such companies so if this is to be a new policy then we have some de-tangling to do.
Note that those relationships are generally getting formalised as PSF in-kind sponsorships, so that they're not completely dependent on personal relationships with folks that work there, and so there's likely to be some prior warning if credits are being withdrawn and the Infrastructure team needs to make alternate service hosting arrangements (see https://www.python.org/psf/sponsorship/sponsors/ - while that listing doesn't separate out cash sponsorships from in-kind service donations, it's possible to make a reasonably informed guess as to which is which).
FWIW, switching from a hybrid Travis/Appveyor setup to a hybrid Travis/VSTS setup seems like it would be a reasonable way to go to me, since we have existing sponsorship arrangements in place with both MS and Travis CI,
I'm not sure how formal our Travis arrangement is as that came about that is to Donald's personal connections there which have left the company. (E.g. I emailed support to get a sprint bump to not have cryptography cripple us this year and I got the impression they didn't know what I was talking about.) but no formal relationship with the Appveyor folks that I'm aware of (I
believe we're just using their regular free tier).
Correct. We're just like any other project to them. -Brett
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Hi Brett, On Thu, 17 May 2018 10:54:27 -0400 Brett Cannon <brett@python.org> wrote:
Since both Paul Moore and Antoine Pitrou started to ask questions about a side comment I made about VSTS, I might as well start a discussion (Steve has also *just* emailed python-committers about this topic).
The main thing that worries me is that VSTS (the service, perhaps not the basic infrastructure) is a very new thing and we don't have any visibility over its continuity, its capacity, its robustness, etc. Travis-CI and AppVeyor, with all their defaults, have been in this space for some time, have a lot of existing customers and we can expect some amount of stability in the future (of course, there's no guarantee). So I'd like it if didn't ditch *all* of the non-VSTS CI. Not for now anyway, we can of course revisit the decision in 2 years :-) Regards Antoine.
On Thu, 17 May 2018 at 12:57 Antoine Pitrou <solipsis@pitrou.net> wrote:
Hi Brett,
On Thu, 17 May 2018 10:54:27 -0400 Brett Cannon <brett@python.org> wrote:
Since both Paul Moore and Antoine Pitrou started to ask questions about a side comment I made about VSTS, I might as well start a discussion (Steve has also *just* emailed python-committers about this topic).
The main thing that worries me is that VSTS (the service, perhaps not the basic infrastructure) is a very new thing and we don't have any visibility over its continuity, its capacity, its robustness, etc.
Actually, depending on how you measure, VSTS is older than Travis. :) So originally there was TFS which was an on-premise thing from MS released in 2005 (https://en.wikipedia.org/wiki/Team_Foundation_Server). The cloud-hosted version is VSTS and that was released in 2013 ( https://en.wikipedia.org/wiki/Microsoft_Visual_Studio#Team_Services). Travis it turns out was founded in 2011 ( https://en.wikipedia.org/wiki/Travis_CI). And if it makes you feel any better, all of MS runs on VSTS, e.g. Windows, Visual Studio, Office, etc. So it's only "new" from the perspective of anonymous access and thus being usable for open source projects in general.
Travis-CI and AppVeyor, with all their defaults, have been in this space for some time, have a lot of existing customers and we can expect some amount of stability in the future (of course, there's no guarantee).
So I'd like it if didn't ditch *all* of the non-VSTS CI. Not for now anyway, we can of course revisit the decision in 2 years :-)
If that's how people end up feeling in general then that's fine and we can leave Travis and/or AppVeyor in place and simply not make them required so they don't hold us up if VSTS turns out to run faster.
Hi all,
On May 17, 2018, at 11:29 AM, Brett Cannon <brett@python.org <mailto:brett@python.org>> wrote:
On Thu, 17 May 2018 at 12:57 Antoine Pitrou <solipsis@pitrou.net <mailto:solipsis@pitrou.net>> wrote:
Hi Brett,
On Thu, 17 May 2018 10:54:27 -0400 Brett Cannon <brett@python.org <mailto:brett@python.org>> wrote:
Since both Paul Moore and Antoine Pitrou started to ask questions about a side comment I made about VSTS, I might as well start a discussion (Steve has also *just* emailed python-committers about this topic).
The main thing that worries me is that VSTS (the service, perhaps not the basic infrastructure) is a very new thing and we don't have any visibility over its continuity, its capacity, its robustness, etc.
Actually, depending on how you measure, VSTS is older than Travis. :) So originally there was TFS which was an on-premise thing from MS released in 2005 (https://en.wikipedia.org/wiki/Team_Foundation_Server <https://en.wikipedia.org/wiki/Team_Foundation_Server>). The cloud-hosted version is VSTS and that was released in 2013 (https://en.wikipedia.org/wiki/Microsoft_Visual_Studio#Team_Services <https://en.wikipedia.org/wiki/Microsoft_Visual_Studio#Team_Services>). Travis it turns out was founded in 2011 (https://en.wikipedia.org/wiki/Travis_CI <https://en.wikipedia.org/wiki/Travis_CI>).
And if it makes you feel any better, all of MS runs on VSTS, e.g. Windows, Visual Studio, Office, etc. So it's only "new" from the perspective of anonymous access and thus being usable for open source projects in general.
Travis-CI and AppVeyor, with all their defaults, have been in this space for some time, have a lot of existing customers and we can expect some amount of stability in the future (of course, there's no guarantee).
So I'd like it if didn't ditch *all* of the non-VSTS CI. Not for now anyway, we can of course revisit the decision in 2 years :-)
If that's how people end up feeling in general then that's fine and we can leave Travis and/or AppVeyor in place and simply not make them required so they don't hold us up if VSTS turns out to run faster.
I had a long discussion with Steve at PyCon about VSTS. The benefits seem promising. Perhaps a reasonable step forward is to run Travis, AppVeyor, and VSTS as required for 3 months or 6 months. This would give everyone a view into using VSTS. After the initial period, we could then revisit what makes sense in the longer term. Which could be to continue using and requiring all; use all but perhaps relax the requirement for passing; or to choose one preferred combination of tools.
_______________________________________________ core-workflow mailing list -- core-workflow@python.org <mailto:core-workflow@python.org> To unsubscribe send an email to core-workflow-leave@python.org <mailto:core-workflow-leave@python.org> https://mail.python.org/mm3/mailman3/lists/core-workflow.python.org/ <https://mail.python.org/mm3/mailman3/lists/core-workflow.python.org/> This list is governed by the PSF Code of Conduct: https://www.python.org/psf/codeofconduct
OK, clearly I worry too much. :-) Faster tests are certainly most welcome, and I am super grateful for all the work that Steve and Brett have put into this. --Guido On Thu, May 17, 2018 at 3:41 PM, Carol Willing < willingc@willingconsulting.com> wrote:
Hi all,
On May 17, 2018, at 11:29 AM, Brett Cannon <brett@python.org> wrote:
On Thu, 17 May 2018 at 12:57 Antoine Pitrou <solipsis@pitrou.net> wrote:
Hi Brett,
On Thu, 17 May 2018 10:54:27 -0400 Brett Cannon <brett@python.org> wrote:
Since both Paul Moore and Antoine Pitrou started to ask questions about a side comment I made about VSTS, I might as well start a discussion (Steve has also *just* emailed python-committers about this topic).
The main thing that worries me is that VSTS (the service, perhaps not the basic infrastructure) is a very new thing and we don't have any visibility over its continuity, its capacity, its robustness, etc.
Actually, depending on how you measure, VSTS is older than Travis. :) So originally there was TFS which was an on-premise thing from MS released in 2005 (https://en.wikipedia.org/wiki/Team_Foundation_Server). The cloud-hosted version is VSTS and that was released in 2013 ( https://en.wikipedia.org/wiki/Microsoft_Visual_Studio#Team_Services). Travis it turns out was founded in 2011 (https://en.wikipedia.org/ wiki/Travis_CI).
And if it makes you feel any better, all of MS runs on VSTS, e.g. Windows, Visual Studio, Office, etc. So it's only "new" from the perspective of anonymous access and thus being usable for open source projects in general.
Travis-CI and AppVeyor, with all their defaults, have been in this space for some time, have a lot of existing customers and we can expect some amount of stability in the future (of course, there's no guarantee).
So I'd like it if didn't ditch *all* of the non-VSTS CI. Not for now anyway, we can of course revisit the decision in 2 years :-)
If that's how people end up feeling in general then that's fine and we can leave Travis and/or AppVeyor in place and simply not make them required so they don't hold us up if VSTS turns out to run faster.
I had a long discussion with Steve at PyCon about VSTS. The benefits seem promising.
Perhaps a reasonable step forward is to run Travis, AppVeyor, and VSTS as required for 3 months or 6 months. This would give everyone a view into using VSTS.
After the initial period, we could then revisit what makes sense in the longer term. Which could be to continue using and requiring all; use all but perhaps relax the requirement for passing; or to choose one preferred combination of tools.
_______________________________________________ core-workflow mailing list -- core-workflow@python.org To unsubscribe send an email to core-workflow-leave@python.org https://mail.python.org/mm3/mailman3/lists/core-workflow.python.org/ This list is governed by the PSF Code of Conduct: https://www.python.org/psf/codeofconduct
_______________________________________________ core-workflow mailing list -- core-workflow@python.org To unsubscribe send an email to core-workflow-leave@python.org https://mail.python.org/mm3/mailman3/lists/core-workflow.python.org/ This list is governed by the PSF Code of Conduct: https://www.python.org/psf/codeofconduct
-- --Guido van Rossum (python.org/~guido)
Hi all,
I had a long discussion with Steve at PyCon about VSTS. The benefits seem promising.
Perhaps a reasonable step forward is to run Travis, AppVeyor, and VSTS as required for 3 months or 6 months. This would give everyone a view into using VSTS.
But that isn't what happened. Someone turned off our Travis and AppVeyor CI systems and made the unreliable VSTS system a PR merge blocker. :( While we're in the middle of a release cycle!
After the initial period, we could then revisit what makes sense in the longer term. Which could be to continue using and requiring all; use all but perhaps relax the requirement for passing; or to choose one preferred combination of tools.
That would be the only sane way to do this. instead, we have: https://python.visualstudio.com/cpython/_build?buildId=522 https://python.visualstudio.com/cpython/_build?buildId=523 both of which are entirely useless, providing zero information about what went wrong. and are clearly infrastructure failures. The UI has ZERO LOGS, and doesn't even link back to the github thing that triggered it. for reference, those runs came from these issues: https://github.com/python/cpython/pull/6938#issuecomment-389908094 https://github.com/python/cpython/pull/6939 Those PRs can't be merged into to 3.6 or 3.7 because of VSTS and whomever made the choice to hold the CPython repo hostage. I am not impressed. Bring back Travis and AppVeyor. Microsoft's currently alpha-quality toy must not be a merge blocker until after it has proven itself stable for at least a month. Even after that, if it continues to have a UI that doesn't put the logs up front and center when opening a failure, I will never want to open a link to it.
On Fri, 18 May 2018 at 01:09 Gregory P. Smith <greg@krypto.org> wrote:
Hi all,
I had a long discussion with Steve at PyCon about VSTS. The benefits seem promising.
Perhaps a reasonable step forward is to run Travis, AppVeyor, and VSTS as required for 3 months or 6 months. This would give everyone a view into using VSTS.
But that isn't what happened.
Someone turned off our Travis and AppVeyor CI systems and made the unreliable VSTS system a PR merge blocker. :(
No one has turned anything off or changed what is required. Looking at the two latest PRs they both have Travis and AppVeyor running ( https://github.com/python/cpython/pull/6971 is for master, https://github.com/python/cpython/pull/6972 is for 2.7). Did you happen to not scroll down the status check section in your browser? GitHub puts the required checks at the bottom (which has also not changed).
While we're in the middle of a release cycle!
Yep, and nothing has changed except we have *added* more CI; nothing else has changed.
After the initial period, we could then revisit what makes sense in the longer term. Which could be to continue using and requiring all; use all but perhaps relax the requirement for passing; or to choose one preferred combination of tools.
That would be the only sane way to do this.
instead, we have: https://python.visualstudio.com/cpython/_build?buildId=522 https://python.visualstudio.com/cpython/_build?buildId=523
both of which are entirely useless, providing zero information about what went wrong. and are clearly infrastructure failures. The UI has ZERO LOGS, and doesn't even link back to the github thing that triggered it.
for reference, those runs came from these issues: https://github.com/python/cpython/pull/6938#issuecomment-389908094 https://github.com/python/cpython/pull/6939
Those PRs can't be merged into to 3.6 or 3.7 because of VSTS and whomever made the choice to hold the CPython repo hostage.
No, Travis and AppVeyor ran and they both say the PR passed. The issue is only that the Linux build on VSTS failed and miss-islington doesn't distinguish between *required* status checks and *any* status check. It can still be merged by clicking the "Squash and merge" button.
I am not impressed.
Bring back Travis and AppVeyor. Microsoft's currently alpha-quality toy must not be a merge blocker until after it has proven itself stable for at least a month. Even after that, if it continues to have a UI that doesn't put the logs up front and center when opening a failure, I will never want to open a link to it.
I'm going to choose to assume the tone used in this email is due to a misunderstanding of not noticing that Travis and AppVeyor are in fact still running and are required along with an oversight -- that I can take blame for if that's necessary -- that miss-islington doesn't distinguish between required and optional status checks when deciding whether to automatically merge. We can temporarily flip off VSTS if automatic merging is viewed as critical right now until after the RC, but otherwise all that has happened is we have introduced more CI which isn't as stable as Travis and AppVeyor yet due to working out testing kinks which did not come up when Steve was testing this on his fork of CPython (which we had to do for both Travis and AppVeyor as well). But in the future, please give us the benefit of the doubt. I'm sitting in MSP trying to deal with this, stressed up the wazoo from worrying that somehow someone had changed something without speaking with me or a release manager, all while about to get on a plane after frankly being yelled at by a friend. :(
On Fri, May 18, 2018 at 8:38 AM Brett Cannon <brett@python.org> wrote:
On Fri, 18 May 2018 at 01:09 Gregory P. Smith <greg@krypto.org> wrote:
Hi all,
I had a long discussion with Steve at PyCon about VSTS. The benefits seem promising.
Perhaps a reasonable step forward is to run Travis, AppVeyor, and VSTS as required for 3 months or 6 months. This would give everyone a view into using VSTS.
But that isn't what happened.
Someone turned off our Travis and AppVeyor CI systems and made the unreliable VSTS system a PR merge blocker. :(
No one has turned anything off or changed what is required. Looking at the two latest PRs they both have Travis and AppVeyor running ( https://github.com/python/cpython/pull/6971 is for master, https://github.com/python/cpython/pull/6972 is for 2.7). Did you happen to not scroll down the status check section in your browser? GitHub puts the required checks at the bottom (which has also not changed).
While we're in the middle of a release cycle!
Yep, and nothing has changed except we have *added* more CI; nothing else has changed.
After the initial period, we could then revisit what makes sense in the longer term. Which could be to continue using and requiring all; use all but perhaps relax the requirement for passing; or to choose one preferred combination of tools.
That would be the only sane way to do this.
instead, we have: https://python.visualstudio.com/cpython/_build?buildId=522 https://python.visualstudio.com/cpython/_build?buildId=523
both of which are entirely useless, providing zero information about what went wrong. and are clearly infrastructure failures. The UI has ZERO LOGS, and doesn't even link back to the github thing that triggered it.
for reference, those runs came from these issues: https://github.com/python/cpython/pull/6938#issuecomment-389908094 https://github.com/python/cpython/pull/6939
Those PRs can't be merged into to 3.6 or 3.7 because of VSTS and whomever made the choice to hold the CPython repo hostage.
No, Travis and AppVeyor ran and they both say the PR passed. The issue is only that the Linux build on VSTS failed and miss-islington doesn't distinguish between *required* status checks and *any* status check. It can still be merged by clicking the "Squash and merge" button.
I am not impressed.
Bring back Travis and AppVeyor. Microsoft's currently alpha-quality toy must not be a merge blocker until after it has proven itself stable for at least a month. Even after that, if it continues to have a UI that doesn't put the logs up front and center when opening a failure, I will never want to open a link to it.
I'm going to choose to assume the tone used in this email is due to a misunderstanding of not noticing that Travis and AppVeyor are in fact still running and are required along with an oversight -- that I can take blame for if that's necessary -- that miss-islington doesn't distinguish between required and optional status checks when deciding whether to automatically merge. We can temporarily flip off VSTS if automatic merging is viewed as critical right now until after the RC, but otherwise all that has happened is we have introduced more CI which isn't as stable as Travis and AppVeyor yet due to working out testing kinks which did not come up when Steve was testing this on his fork of CPython (which we had to do for both Travis and AppVeyor as well).
But in the future, please give us the benefit of the doubt. I'm sitting in MSP trying to deal with this, stressed up the wazoo from worrying that somehow someone had changed something without speaking with me or a release manager, all while about to get on a plane after frankly being yelled at by a friend. :(
Sorry. Big misunderstanding on my part. I made wrong assumptions due in part to poor UI design. :( It appears to be a combination of Github's UI failure interacting with the added Checkers. There was no longer a green merge button and the list displayed by github hides the runs i care about below a completely arbitrary invisible "fold" which led me to believe they were no longer being run. I'm happy to see that the known-good CI is still being run! But Github is no longer showing it without additional secret effort to find as it is hidden at the bottom of an invisible scrolling region. I could also blame the last decades dumb insistence on removing scrollbars from UIs for this, but it is up to github to work around the current decade bad ui paradigms. The Github UI says "show all" but does not in fact "show all". It "shows five" and doesn't tell you that it can scroll. -gps
On May 18, 2018, at 1:46 PM, Chris Angelico <rosuav@gmail.com> wrote:
On Sat, May 19, 2018 at 1:38 AM, Brett Cannon <brett@python.org> wrote:
But in the future, please give us the benefit of the doubt. I'm sitting in MSP trying to deal with this...
Sorry for the dumb question... MSP? I'm guessing that's Microsoft something?
https://en.wikipedia.org/wiki/Minneapolis%E2%80%93Saint_Paul_International_A... <https://en.wikipedia.org/wiki/Minneapolis%E2%80%93Saint_Paul_International_Airport> I assume.
On Fri, May 18, 2018 at 10:51 AM, Donald Stufft <donald@stufft.io> wrote:
On May 18, 2018, at 1:46 PM, Chris Angelico <rosuav@gmail.com> wrote:
On Sat, May 19, 2018 at 1:38 AM, Brett Cannon <brett@python.org> wrote:
But in the future, please give us the benefit of the doubt. I'm sitting in MSP trying to deal with this...
Sorry for the dumb question... MSP? I'm guessing that's Microsoft something?
https://en.wikipedia.org/wiki/Minneapolis%E2%80%93Saint_ Paul_International_Airport <https://en.wikipedia.org/wiki/Minneapolis–Saint_Paul_International_Airport> I assume.
See also: https://xkcd.com/1937/ -- --Guido van Rossum (python.org/~guido)
On Sat, May 19, 2018 at 4:15 AM, Guido van Rossum <guido@python.org> wrote:
On Fri, May 18, 2018 at 10:51 AM, Donald Stufft <donald@stufft.io> wrote:
On May 18, 2018, at 1:46 PM, Chris Angelico <rosuav@gmail.com> wrote:
On Sat, May 19, 2018 at 1:38 AM, Brett Cannon <brett@python.org> wrote:
But in the future, please give us the benefit of the doubt. I'm sitting in MSP trying to deal with this...
Sorry for the dumb question... MSP? I'm guessing that's Microsoft something?
https://en.wikipedia.org/wiki/Minneapolis%E2%80%93Saint_Paul_International_A... I assume.
See also: https://xkcd.com/1937/
Ah, thanks both. I didn't even think of airports. ChrisA now imagining an airport called "Microsoft Private" and getting the name MSP
But that isn't what happened.
Someone turned off our Travis and AppVeyor CI systems and made the unreliable VSTS system a PR merge blocker. :(
It appears not to be turned off. But is being hidden by Github's UI. No big Green "squash/merge" button shows up when VSTS builds run into infrastructure issues. So it has the _appearance_ of not being possible to merge. A white on white button apparently exists which might let me merge things anyways? In order to see what has happened on a PR, you have to ignore Github's lie about "show all checks" showing everything. It doesn't. It only shows you five. Which are ordered in an unknown manner that is currently putting VSTS at the top and putting the checks I trust at the bottom, hidden below the fold of the stupidly limited Github UI's invisible scrolling region (you can scroll it, but there is no ui indication of that). I've emailed support@github.com about this UX failure. (I expect to be ignored) As is, VSTS is getting in my way. I do want VSTS builds to run, that is the only way anyone is going to get the Microsoft supplied infrastructure into shape, but I don't want them to detract from our process: Today they appear to get in the way UI wise by hiding the stuff I care about. If we are supposed to ignore VSTS builds due to unreliability and undiagnosability, it means we're likely to revert to routinely ignoring all sorts of CI failures and merge without the "all checks passed" Green button. This moves us back towards post-commit buildbot only testing. :/ Can VSTS be setup to show all builds in a single line instead of polluting the list of checks with four lines? Travis allows this - everything in the travis test matrix is rolled up into a single check line on github, with configuration about which things on travis are allowed failures or not being configured in the .travis.yaml file. I don't like such aggregation, but so long as Github has such a useless UI, it makes sense.
Hi, Not sure where to report this, but I had a weird build failure here: https://python.visualstudio.com/cpython/_build?buildId=2049&tab=details&_a=summary Regards Antoine. On Thu, 17 May 2018 10:54:27 -0400 Brett Cannon <brett@python.org> wrote:
Since both Paul Moore and Antoine Pitrou started to ask questions about a side comment I made about VSTS, I might as well start a discussion (Steve has also *just* emailed python-committers about this topic).
Basically Steve Dower and I have been working directly with the VSTS team to improve their Python support. Along the way they have announced they are adding anonymous access to build logs which was always a major blocker for using it in open source. VSTS gave us early access to the anonymous access that they are launching along with 20 concurrent builders which is more than what we currently have on Travis and AppVeyor.
At this point Steve has added the appropriate YAML files in the .vsts directory that control the build jobs so we can test out the integration and see what the performance is like. But based on what we have been seeing there will be no queue in getting PRs tested and that applies to Windows, macOS, and Linux (IOW better coverage and no more issues about performance or having to wait too long for CI to go green ;) . There is also other benefits like making at least Windows builds easier to release as signing those builds can now be automated.
My expectation is that once we are convinced that the VSTS builds are passing consistently and everything is working as everyone wants for e.g. a week, we can plan to turn off Travis and AppVeyor for master and 3.x branches (no one wants to bother with trying to get 2.7 to work on this and we can simply continue to rely on Travis and AppVeyor as necessary for the next 18 months). I don't think we want to run multiple CIs indefinitely on the same branch as it does take time and effort to keep CI green on a provider since they all have their own quirks.
I should also mention that this free access is not limited to just CPython and any project can get free accounts and credits now, you just have to talk to Steve about getting in on the anonymous access before it launches to the general public (I have no timeline on when that's happening).
Adding Steve. On Wed, 23 May 2018 at 11:48 Antoine Pitrou <solipsis@pitrou.net> wrote:
Hi,
Not sure where to report this, but I had a weird build failure here:
https://python.visualstudio.com/cpython/_build?buildId=2049&tab=details&_a=summary
Regards
Antoine.
Since both Paul Moore and Antoine Pitrou started to ask questions about a side comment I made about VSTS, I might as well start a discussion (Steve has also *just* emailed python-committers about this topic).
Basically Steve Dower and I have been working directly with the VSTS team to improve their Python support. Along the way they have announced they are adding anonymous access to build logs which was always a major blocker for using it in open source. VSTS gave us early access to the anonymous access that they are launching along with 20 concurrent builders which is more than what we currently have on Travis and AppVeyor.
At this point Steve has added the appropriate YAML files in the .vsts directory that control the build jobs so we can test out the integration and see what the performance is like. But based on what we have been seeing there will be no queue in getting PRs tested and that applies to Windows, macOS, and Linux (IOW better coverage and no more issues about
or having to wait too long for CI to go green ;) . There is also other benefits like making at least Windows builds easier to release as signing those builds can now be automated.
My expectation is that once we are convinced that the VSTS builds are passing consistently and everything is working as everyone wants for e.g. a week, we can plan to turn off Travis and AppVeyor for master and 3.x branches (no one wants to bother with trying to get 2.7 to work on this and we can simply continue to rely on Travis and AppVeyor as necessary for
On Thu, 17 May 2018 10:54:27 -0400 Brett Cannon <brett@python.org> wrote: performance the
next 18 months). I don't think we want to run multiple CIs indefinitely on the same branch as it does take time and effort to keep CI green on a provider since they all have their own quirks.
I should also mention that this free access is not limited to just CPython and any project can get free accounts and credits now, you just have to talk to Steve about getting in on the anonymous access before it launches to the general public (I have no timeline on when that's happening).
_______________________________________________ core-workflow mailing list -- core-workflow@python.org To unsubscribe send an email to core-workflow-leave@python.org https://mail.python.org/mm3/mailman3/lists/core-workflow.python.org/ This list is governed by the PSF Code of Conduct: https://www.python.org/psf/codeofconduct
I've emailed the VSTS team about it and they're going to investigate (but since it's after hours on the US East Coast it won't be an immediate response). The first guess is we might have caught an update to the build machine at a bad time. But I'll get back when we've figured it out. Cheers, Steve On 23May2018 1312, Brett Cannon wrote:
Adding Steve.
On Wed, 23 May 2018 at 11:48 Antoine Pitrou <solipsis@pitrou.net <mailto:solipsis@pitrou.net>> wrote:
Hi,
Not sure where to report this, but I had a weird build failure here: https://python.visualstudio.com/cpython/_build?buildId=2049&tab=details&_a=summary
Regards
Antoine.
On Thu, 17 May 2018 10:54:27 -0400 Brett Cannon <brett@python.org <mailto:brett@python.org>> wrote: > Since both Paul Moore and Antoine Pitrou started to ask questions about a > side comment I made about VSTS, I might as well start a discussion (Steve > has also *just* emailed python-committers about this topic). > > Basically Steve Dower and I have been working directly with the VSTS team > to improve their Python support. Along the way they have announced they are > adding anonymous access to build logs which was always a major blocker for > using it in open source. VSTS gave us early access to the anonymous access > that they are launching along with 20 concurrent builders which is more > than what we currently have on Travis and AppVeyor. > > At this point Steve has added the appropriate YAML files in the .vsts > directory that control the build jobs so we can test out the integration > and see what the performance is like. But based on what we have been seeing > there will be no queue in getting PRs tested and that applies to Windows, > macOS, and Linux (IOW better coverage and no more issues about performance > or having to wait too long for CI to go green ;) . There is also other > benefits like making at least Windows builds easier to release as signing > those builds can now be automated. > > My expectation is that once we are convinced that the VSTS builds are > passing consistently and everything is working as everyone wants for e.g. a > week, we can plan to turn off Travis and AppVeyor for master and 3.x > branches (no one wants to bother with trying to get 2.7 to work on this and > we can simply continue to rely on Travis and AppVeyor as necessary for the > next 18 months). I don't think we want to run multiple CIs indefinitely on > the same branch as it does take time and effort to keep CI green on a > provider since they all have their own quirks. > > I should also mention that this free access is not limited to just CPython > and any project can get free accounts and credits now, you just have to > talk to Steve about getting in on the anonymous access before it launches > to the general public (I have no timeline on when that's happening). >
_______________________________________________ core-workflow mailing list -- core-workflow@python.org <mailto:core-workflow@python.org> To unsubscribe send an email to core-workflow-leave@python.org <mailto:core-workflow-leave@python.org> https://mail.python.org/mm3/mailman3/lists/core-workflow.python.org/ This list is governed by the PSF Code of Conduct: https://www.python.org/psf/codeofconduct
that miss-islington doesn't distinguish between required and optional status checks when deciding whether to automatically merge.
What's the decision surrounding this? Should I make miss-islington ignore the optional status checks? So as long as the required ones pass, then go ahead with auto-merge? Mariatta ᐧ
On Fri, 1 Jun 2018 at 13:22 Mariatta Wijaya <mariatta.wijaya@gmail.com> wrote:
that miss-islington doesn't distinguish between required and optional
status checks when deciding whether to automatically merge.
What's the decision surrounding this? Should I make miss-islington ignore the optional status checks? So as long as the required ones pass, then go ahead with auto-merge?
We didn't reach a conclusion. I'm fine with ignoring optional status checks since they are optional for a reason. :)
participants (10)
-
Antoine Pitrou -
Brett Cannon -
Carol Willing -
Chris Angelico -
Donald Stufft -
Gregory P. Smith -
Guido van Rossum -
Mariatta Wijaya -
Nick Coghlan -
Steve Dower