Travis CI for backports not working.

Travis passed https://github.com/python/cpython/pull/17366 but a half hour later twice failed https://github.com/python/cpython/pull/17370 https://github.com/python/cpython/pull/17371 saying pyenv: python3.8: command not found pyenv: version `3.7.1' not installed -- Terry Jan Reedy

On 25.11.2019 1:10, Terry Reedy wrote:
This is build logic's fault, `python3.8` is not guaranteed to be present. I believe Configure is finding pyenv's shim which is present but returns 127 when run if there's no underlying executable to invoke. This is proven by "pyenv: python3.8: command not found".
This looks like Travis' fault, 3.7.1 should be preinstalled in Xenial with `language: c` according to my findings: https://github.com/travis-ci/docs-travis-ci-com/pull/2413/files#diff-fb862f5... I reported this: https://github.com/travis-ci/docs-travis-ci-com/pull/2413#issuecomment-55794... But note that the text stating that is not yet in the official documentation. (Current text says that it should be preinstalled but not for which languages.)
-- Regards, Ivan

On 25.11.2019 9:50, Terry Reedy wrote:
Don't count on this being fixed fast: I said that this is not even an official guarantee yet. Either move to Bionic where python 3.7 is available from Apt, or use `language: python`+`python: 3.7` or `language: generic` (where 3.7.1 is hopefully more reliably preinstalled) and install Clang 7 from Apt.
-- Regards, Ivan

On 11/25/2019 12:40 PM, Ivan Pozdeev via Python-Dev wrote:
I don't understand this. All I know is that backport CI must have been working earlier yesterday and something changed so that Travis quit working and I can no longer merge backports. So my work is blocked. -- Terry Jan Reedy

I have turned Travis off as a required check on the 3.7, 3.8, and master branches until someone is able to get a fix in.

Hello, Le mar. 26 nov. 2019 à 20:40, Brett Cannon <brett@python.org> a écrit :
I have turned Travis off as a required check on the 3.7, 3.8, and master branches until someone is able to get a fix in.
That makes me sad :-( Is there an issue at bugs.python.org to track it? What's the status? Right now, I see Travis CI jobs passing on 3.7, 3.8 and master branches so I don't understand the problem. Maybe the issue has been fixed and Travis CI can be made mandatory again? Victor -- Night gathers, and now my watch begins. It shall not end until my death.

On 06Dec2019 0620, Victor Stinner wrote:
They've been passing fine for me too, I'm not quite sure what the issue was. As a related aside, I've been getting GitHub Actions support together (which I started at the sprints). My test PR is at https://github.com/zooba/cpython/pull/7 if anyone wants to check out what it could look like. Feel free to leave comments there. (One of these days I'll have to join core-workflow I guess...) Cheers, Steve

Steve Dower wrote:
As a related aside, I've been getting GitHub Actions support together (which I started at the sprints).
Would adding support for GitHub Actions make it easier/faster to temporarily disable and re-enable specific CI services when they're having external issues? IIUC, that seems to be the primary concern to address. Note that I'm not particularly well acquainted with GitHub Actions, other than briefly looking over https://github.com/features/actions. On Fri, Dec 6, 2019 at 1:01 PM Steve Dower <steve.dower@python.org> wrote:

On 06Dec2019 1023, Kyle Stanley wrote:
GitHub Actions *is* a CI service now, so my PR is actually using their machines for Windows/macOS/Ubuntu build and test. It doesn't change the ease of enabling/disabling anything - that's still very easy if you're an administrator on our repo and impossible if you're not :) However, it does save jumping to an external website to view logs, and over time we can improve the integration so that error messages are captured properly (but you can easily view each step). And you don't need a separate login to rerun checks - it's just a simple button in the GitHub UI. Cheers, Steve

Steve Dower wrote:
GitHub Actions *is* a CI service now, so my PR is actually using their machines for Windows/macOS/Ubuntu build and test.
Oh I see, that makes more sense. Thanks for the clarification. Steve Dower wrote:
Yeah I was aware that it required administrative permissions, I was just under the impression that it wasn't a simple on/off toggle, and that there wasn't an easy way to ensure failures were occurring due to external factors. I suppose that's a separate issue though. Steve Dower wrote:
That would be especially great, and much better than closing+reopening the PR. Restarting the checks is typically a last resort since intermittent regression test failures are a concern, but it's highly useful when the failure is occurring due to external factors (such as a recently merged PR or issues with the CI service itself). On Fri, Dec 6, 2019 at 1:33 PM Steve Dower <steve.dower@python.org> wrote:

Victor Stinner wrote:
Nope. I had people personally asking me to deal with it and I didn't have time to investigate so to unblock folks I just flipped it off.
What's the status?
🤷♂️
🤷♂️ -Brett
Victor Night gathers, and now my watch begins. It shall not end until my death.

This is failing again, so I had to switch off Travis from being a requirement (again). I'm not not going to flip it back on until Travis has been stable for a month as I don't like being the blocker on stuff when I can help it. And if Travis isn't stable in a month then we might need to start talking about turning it off entirely as flaky CI is never useful.

What is the issue? Can someone please open a bug report at https://bugs.python.org/ so I can try to investigate? Victor Le ven. 13 déc. 2019 à 02:05, Brett Cannon <brett@python.org> a écrit :
-- Night gathers, and now my watch begins. It shall not end until my death.

Victor Stinner wrote:
What is the issue? Can someone please open a bug report at https://bugs.python.org/ so I can try to investigate?
From my understanding, it looks to be pyenv related and not something we can fix on our end, at least based on the build logs: https://travis-ci.org/python/cpython/jobs/624160244?utm_medium=notification&utm_source=github_status. This was from a recent backport PR to 3.7 (backport also failing for 3.8 with similar issues). On Thu, Dec 12, 2019 at 8:14 PM Victor Stinner <vstinner@python.org> wrote:

Does Travis-CI serve any purpose for us still? I.e. if someone breaks some test, is Travis-CI the only thing that keeps the breakage from landing on master? On Thu, Dec 12, 2019 at 5:46 PM Kyle Stanley <aeros167@gmail.com> wrote:
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>

I.e. if someone breaks some test, is Travis-CI the only thing that keeps the breakage from landing on master?
We still have Azure Pipelines as a mandatory check for PRs before they can be merged to master, which includes a few additional platforms and hasn't had any recent issues (AFAIK). I believe Steve Dower was also recently experimenting with adding GitHub Actions as an additional CI service. On Thu, Dec 12, 2019 at 10:16 PM Guido van Rossum <guido@python.org> wrote:

I created https://bugs.python.org/issue39035 I like Travis CI. It's very close to what I have on my laptop, so it's usually trivial for me to reproduce Travis CI failures. It's also quite fast and reliable. Azure Pipelines were very unstable one year ago. It's getting better, but there are still some random bugs sometimes. They are not really blocking, so I didn't report them. On Travis CI, it's possible to only restart a single job manually when it's a CI issue (like random networking issue). On Azure Pipelines, there is no way to restart even all jobs at once. The only workaround is to close the PR and reopen it. But when you do that on a backport PR, a bot closes the PR and removes the branch. The backport PR must be recreated, it's annoying. https://pythondev.readthedocs.io/ci.html#azure-pipelines-pr For example, I saw recently apt-get failed to download packages on the Linux jobs on the Azure Pipelines. In short, having multiple CIs is a good thing :-) Victor Le ven. 13 déc. 2019 à 02:42, Kyle Stanley <aeros167@gmail.com> a écrit :
-- Night gathers, and now my watch begins. It shall not end until my death.

On 13Dec2019 0233, Victor Stinner wrote:
The only ones I'm aware of are macOS builds failing (which don't run on Travis CI, so the only thing stopping these bugs landing is Azure Pipelines) and network related issues. But I'm guessing since you said "random" bugs that they don't repro well enough to assign blame properly.
The UI is getting better here, but given GitHub Actions now has similar CI functionality I wouldn't expect Pipelines to focus as much on their integration with GitHub (in particular, being able to authorize an GitHub team to log in to our Pipelines instance - as we can with Travis - has been preventing people from rerunning individual jobs). If people are generally happy to move PR builds/checks to GitHub Actions, I'm happy to merge https://github.com/zooba/cpython/pull/7 into our active branches (with probably Brett's help) and disable Azure Pipelines? (I'd like to keep it running for post-merge builds and the manually triggered ones I use for Windows releases and dependencies.)
In short, having multiple CIs is a good thing :-)
Agreed, though it would also be nice to have a way to dismiss a failure after investigating and merge anyway. Only repo administrators can do that. Cheers, Steve

On 13Dec2019 0959, Brett Cannon wrote:
This is now running on master (and likely 3.8 and 3.7, at a guess) - you can see it on my PR at https://github.com/python/cpython/pull/17628 (adding badges and making a tweak to when the builds run) The checks are not required yet - that requires admin powers. Please just shout out, either here on at https://bugs.python.org/issue39041 if you see anything not working. (Apart from post-merge coverage checks. We know they're not working - see https://github.com/python/cpython/runs/351136928 - and if you'd like to help fix it you're more than welcome to jump in!) Cheers, Steve

On 25.11.2019 1:10, Terry Reedy wrote:
This is build logic's fault, `python3.8` is not guaranteed to be present. I believe Configure is finding pyenv's shim which is present but returns 127 when run if there's no underlying executable to invoke. This is proven by "pyenv: python3.8: command not found".
This looks like Travis' fault, 3.7.1 should be preinstalled in Xenial with `language: c` according to my findings: https://github.com/travis-ci/docs-travis-ci-com/pull/2413/files#diff-fb862f5... I reported this: https://github.com/travis-ci/docs-travis-ci-com/pull/2413#issuecomment-55794... But note that the text stating that is not yet in the official documentation. (Current text says that it should be preinstalled but not for which languages.)
-- Regards, Ivan

On 25.11.2019 9:50, Terry Reedy wrote:
Don't count on this being fixed fast: I said that this is not even an official guarantee yet. Either move to Bionic where python 3.7 is available from Apt, or use `language: python`+`python: 3.7` or `language: generic` (where 3.7.1 is hopefully more reliably preinstalled) and install Clang 7 from Apt.
-- Regards, Ivan

On 11/25/2019 12:40 PM, Ivan Pozdeev via Python-Dev wrote:
I don't understand this. All I know is that backport CI must have been working earlier yesterday and something changed so that Travis quit working and I can no longer merge backports. So my work is blocked. -- Terry Jan Reedy

I have turned Travis off as a required check on the 3.7, 3.8, and master branches until someone is able to get a fix in.

Hello, Le mar. 26 nov. 2019 à 20:40, Brett Cannon <brett@python.org> a écrit :
I have turned Travis off as a required check on the 3.7, 3.8, and master branches until someone is able to get a fix in.
That makes me sad :-( Is there an issue at bugs.python.org to track it? What's the status? Right now, I see Travis CI jobs passing on 3.7, 3.8 and master branches so I don't understand the problem. Maybe the issue has been fixed and Travis CI can be made mandatory again? Victor -- Night gathers, and now my watch begins. It shall not end until my death.

On 06Dec2019 0620, Victor Stinner wrote:
They've been passing fine for me too, I'm not quite sure what the issue was. As a related aside, I've been getting GitHub Actions support together (which I started at the sprints). My test PR is at https://github.com/zooba/cpython/pull/7 if anyone wants to check out what it could look like. Feel free to leave comments there. (One of these days I'll have to join core-workflow I guess...) Cheers, Steve

Steve Dower wrote:
As a related aside, I've been getting GitHub Actions support together (which I started at the sprints).
Would adding support for GitHub Actions make it easier/faster to temporarily disable and re-enable specific CI services when they're having external issues? IIUC, that seems to be the primary concern to address. Note that I'm not particularly well acquainted with GitHub Actions, other than briefly looking over https://github.com/features/actions. On Fri, Dec 6, 2019 at 1:01 PM Steve Dower <steve.dower@python.org> wrote:

On 06Dec2019 1023, Kyle Stanley wrote:
GitHub Actions *is* a CI service now, so my PR is actually using their machines for Windows/macOS/Ubuntu build and test. It doesn't change the ease of enabling/disabling anything - that's still very easy if you're an administrator on our repo and impossible if you're not :) However, it does save jumping to an external website to view logs, and over time we can improve the integration so that error messages are captured properly (but you can easily view each step). And you don't need a separate login to rerun checks - it's just a simple button in the GitHub UI. Cheers, Steve

Steve Dower wrote:
GitHub Actions *is* a CI service now, so my PR is actually using their machines for Windows/macOS/Ubuntu build and test.
Oh I see, that makes more sense. Thanks for the clarification. Steve Dower wrote:
Yeah I was aware that it required administrative permissions, I was just under the impression that it wasn't a simple on/off toggle, and that there wasn't an easy way to ensure failures were occurring due to external factors. I suppose that's a separate issue though. Steve Dower wrote:
That would be especially great, and much better than closing+reopening the PR. Restarting the checks is typically a last resort since intermittent regression test failures are a concern, but it's highly useful when the failure is occurring due to external factors (such as a recently merged PR or issues with the CI service itself). On Fri, Dec 6, 2019 at 1:33 PM Steve Dower <steve.dower@python.org> wrote:

Victor Stinner wrote:
Nope. I had people personally asking me to deal with it and I didn't have time to investigate so to unblock folks I just flipped it off.
What's the status?
🤷♂️
🤷♂️ -Brett
Victor Night gathers, and now my watch begins. It shall not end until my death.

This is failing again, so I had to switch off Travis from being a requirement (again). I'm not not going to flip it back on until Travis has been stable for a month as I don't like being the blocker on stuff when I can help it. And if Travis isn't stable in a month then we might need to start talking about turning it off entirely as flaky CI is never useful.

What is the issue? Can someone please open a bug report at https://bugs.python.org/ so I can try to investigate? Victor Le ven. 13 déc. 2019 à 02:05, Brett Cannon <brett@python.org> a écrit :
-- Night gathers, and now my watch begins. It shall not end until my death.

Victor Stinner wrote:
What is the issue? Can someone please open a bug report at https://bugs.python.org/ so I can try to investigate?
From my understanding, it looks to be pyenv related and not something we can fix on our end, at least based on the build logs: https://travis-ci.org/python/cpython/jobs/624160244?utm_medium=notification&utm_source=github_status. This was from a recent backport PR to 3.7 (backport also failing for 3.8 with similar issues). On Thu, Dec 12, 2019 at 8:14 PM Victor Stinner <vstinner@python.org> wrote:

Does Travis-CI serve any purpose for us still? I.e. if someone breaks some test, is Travis-CI the only thing that keeps the breakage from landing on master? On Thu, Dec 12, 2019 at 5:46 PM Kyle Stanley <aeros167@gmail.com> wrote:
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>

I.e. if someone breaks some test, is Travis-CI the only thing that keeps the breakage from landing on master?
We still have Azure Pipelines as a mandatory check for PRs before they can be merged to master, which includes a few additional platforms and hasn't had any recent issues (AFAIK). I believe Steve Dower was also recently experimenting with adding GitHub Actions as an additional CI service. On Thu, Dec 12, 2019 at 10:16 PM Guido van Rossum <guido@python.org> wrote:

I created https://bugs.python.org/issue39035 I like Travis CI. It's very close to what I have on my laptop, so it's usually trivial for me to reproduce Travis CI failures. It's also quite fast and reliable. Azure Pipelines were very unstable one year ago. It's getting better, but there are still some random bugs sometimes. They are not really blocking, so I didn't report them. On Travis CI, it's possible to only restart a single job manually when it's a CI issue (like random networking issue). On Azure Pipelines, there is no way to restart even all jobs at once. The only workaround is to close the PR and reopen it. But when you do that on a backport PR, a bot closes the PR and removes the branch. The backport PR must be recreated, it's annoying. https://pythondev.readthedocs.io/ci.html#azure-pipelines-pr For example, I saw recently apt-get failed to download packages on the Linux jobs on the Azure Pipelines. In short, having multiple CIs is a good thing :-) Victor Le ven. 13 déc. 2019 à 02:42, Kyle Stanley <aeros167@gmail.com> a écrit :
-- Night gathers, and now my watch begins. It shall not end until my death.

On 13Dec2019 0233, Victor Stinner wrote:
The only ones I'm aware of are macOS builds failing (which don't run on Travis CI, so the only thing stopping these bugs landing is Azure Pipelines) and network related issues. But I'm guessing since you said "random" bugs that they don't repro well enough to assign blame properly.
The UI is getting better here, but given GitHub Actions now has similar CI functionality I wouldn't expect Pipelines to focus as much on their integration with GitHub (in particular, being able to authorize an GitHub team to log in to our Pipelines instance - as we can with Travis - has been preventing people from rerunning individual jobs). If people are generally happy to move PR builds/checks to GitHub Actions, I'm happy to merge https://github.com/zooba/cpython/pull/7 into our active branches (with probably Brett's help) and disable Azure Pipelines? (I'd like to keep it running for post-merge builds and the manually triggered ones I use for Windows releases and dependencies.)
In short, having multiple CIs is a good thing :-)
Agreed, though it would also be nice to have a way to dismiss a failure after investigating and merge anyway. Only repo administrators can do that. Cheers, Steve

On 13Dec2019 0959, Brett Cannon wrote:
This is now running on master (and likely 3.8 and 3.7, at a guess) - you can see it on my PR at https://github.com/python/cpython/pull/17628 (adding badges and making a tweak to when the builds run) The checks are not required yet - that requires admin powers. Please just shout out, either here on at https://bugs.python.org/issue39041 if you see anything not working. (Apart from post-merge coverage checks. We know they're not working - see https://github.com/python/cpython/runs/351136928 - and if you'd like to help fix it you're more than welcome to jump in!) Cheers, Steve
participants (8)
-
Brett Cannon
-
Gregory P. Smith
-
Guido van Rossum
-
Ivan Pozdeev
-
Kyle Stanley
-
Steve Dower
-
Terry Reedy
-
Victor Stinner