[Twisted-Python] Does anyone know why trunk is failing CI on PyPy 7.3.4?
TLDR: Twisted appears broken on PyPy 7.3.4 (aka "3.7.10")? I don't have time right now to set up a PyPy-capable environment and try to reproduce this, but perhaps someone does? Long version: Trunk has been broken since the last merge a week or so ago, but I don't think the breakage is due to that merge. As an experiment I made a PR based off the last successful version of trunk, with a whitespace change, and it now fails CI as well. So I think the failure must be due to some change that isn't in Twisted or controlled-for by tox. The failure in all cases is in the pypy-3.7-alldeps-nocov-posix task. Unlike our usual CI problems it doesn't seem to be a random failure: it fails all the time in the same place. But the place doesn't make sense to me. It's in the IRC CTCP tests, and they fail in the same ways each time (an expected response is not received). The pair of CI runs closest to the change are these: run 5793: https://github.com/twisted/twisted/runs/2328450554 run 5809: https://github.com/twisted/twisted/runs/2360415474 There are a lot of differences, but sys.version went from 3.7.9 to 3.7.10 between those runs, so that seems like the most likely culprit. Last working:
sys.version : 3.7.9 (7e6e2bb30ac5, Nov 18 2020, 10:55:52) [PyPy 7.3.3-beta0 with GCC 7.3.1 20180303 (Red Hat 7.3.1-5)] sys.prefix : /opt/hostedtoolcache/PyPy/3.7.9/x64 sys.exec_prefix : /opt/hostedtoolcache/PyPy/3.7.9/x64 sys.executable : /opt/hostedtoolcache/PyPy/3.7.9/x64/bin/python
First broken:
sys.version : 3.7.10 (51efa818fd9b, Apr 04 2021, 11:22:34) [PyPy 7.3.4 with GCC 7.3.1 20180303 (Red Hat 7.3.1-5)] sys.prefix : /opt/hostedtoolcache/PyPy/3.7.10/x64 sys.exec_prefix : /opt/hostedtoolcache/PyPy/3.7.10/x64 sys.executable : /opt/hostedtoolcache/PyPy/3.7.10/x64/bin/python
PyPy's release notes for 7.3.4 don't list anything that jumps out at me: https://doc.pypy.org/en/latest/whatsnew-pypy3-7.3.4.html My guess would be some latent timing bug in Twisted that was uncovered by pypy execution time changes (I don't imagine that the CTCP code gets exercised very heavily these days) or perhaps PyPy got a bug. Wim.
Hi Wim, Thanks for the report On Tue, 27 Apr 2021 at 08:14, Wim Lewis <wiml@hhhh.org> wrote:
TLDR: Twisted appears broken on PyPy 7.3.4 (aka "3.7.10")?
I don't have time right now to set up a PyPy-capable environment and try to reproduce this, but perhaps someone does?
Long version:
Trunk has been broken since the last merge a week or so ago, but I don't think the breakage is due to that merge. As an experiment I made a PR based off the last successful version of trunk, with a whitespace change, and it now fails CI as well. So I think the failure must be due to some change that isn't in Twisted or controlled-for by tox.
The failure in all cases is in the pypy-3.7-alldeps-nocov-posix task. Unlike our usual CI problems it doesn't seem to be a random failure: it fails all the time in the same place. But the place doesn't make sense to me. It's in the IRC CTCP tests, and they fail in the same ways each time (an expected response is not received).
The pair of CI runs closest to the change are these:
run 5793: https://github.com/twisted/twisted/runs/2328450554 run 5809: https://github.com/twisted/twisted/runs/2360415474
There are a lot of differences, but sys.version went from 3.7.9 to 3.7.10 between those runs, so that seems like the most likely culprit.
Last working:
sys.version : 3.7.9 (7e6e2bb30ac5, Nov 18 2020, 10:55:52) [PyPy 7.3.3-beta0 with GCC 7.3.1 20180303 (Red Hat 7.3.1-5)] sys.prefix : /opt/hostedtoolcache/PyPy/3.7.9/x64 sys.exec_prefix : /opt/hostedtoolcache/PyPy/3.7.9/x64 sys.executable : /opt/hostedtoolcache/PyPy/3.7.9/x64/bin/python
First broken:
sys.version : 3.7.10 (51efa818fd9b, Apr 04 2021, 11:22:34) [PyPy 7.3.4 with GCC 7.3.1 20180303 (Red Hat 7.3.1-5)] sys.prefix : /opt/hostedtoolcache/PyPy/3.7.10/x64 sys.exec_prefix : /opt/hostedtoolcache/PyPy/3.7.10/x64 sys.executable : /opt/hostedtoolcache/PyPy/3.7.10/x64/bin/python
PyPy's release notes for 7.3.4 don't list anything that jumps out at me:
https://doc.pypy.org/en/latest/whatsnew-pypy3-7.3.4.html
My guess would be some latent timing bug in Twisted that was uncovered by pypy execution time changes (I don't imagine that the CTCP code gets exercised very heavily these days) or perhaps PyPy got a bug.
Wim.
I have setup a pyp3.7.4 and I can reproduce it. I see 3 options: * Option A: Skip those tests on pypy and open a separate ticket to fix the test * Option B: Pin pypy 3.7.9 for GHA and open a separate ticket to fix the test and unpin it * Option C: Just fix the tests :) I am looking at option C for one hour... if I can't find a fix will look into option A. Cheers -- Adi Roiban
Hi, A quick diff to fix it... and a ticket https://twistedmatrix.com/trac/ticket/10187 I will create a PR and we can refine the fix during the review. diff --git a/src/twisted/words/protocols/irc.py b/src/twisted/words/protocols/irc.py index 75300019b..a97abce2e 100644 --- a/src/twisted/words/protocols/irc.py +++ b/src/twisted/words/protocols/irc.py @@ -3678,10 +3678,10 @@ def ctcpExtract(message): normal_messages.append(messages.pop(0)) odd = not odd - extended_messages[:] = filter(None, extended_messages) - normal_messages[:] = filter(None, normal_messages) + extended_messages[:] = list(filter(None, extended_messages)) + normal_messages[:] = list(filter(None, normal_messages)) - extended_messages[:] = map(ctcpDequote, extended_messages) + extended_messages[:] = list(map(ctcpDequote, extended_messages)) On Wed, 28 Apr 2021 at 10:01, Adi Roiban <adi@roiban.ro> wrote:
Hi Wim,
Thanks for the report
On Tue, 27 Apr 2021 at 08:14, Wim Lewis <wiml@hhhh.org> wrote:
TLDR: Twisted appears broken on PyPy 7.3.4 (aka "3.7.10")?
I don't have time right now to set up a PyPy-capable environment and try to reproduce this, but perhaps someone does?
Long version:
Trunk has been broken since the last merge a week or so ago, but I don't think the breakage is due to that merge. As an experiment I made a PR based off the last successful version of trunk, with a whitespace change, and it now fails CI as well. So I think the failure must be due to some change that isn't in Twisted or controlled-for by tox.
The failure in all cases is in the pypy-3.7-alldeps-nocov-posix task. Unlike our usual CI problems it doesn't seem to be a random failure: it fails all the time in the same place. But the place doesn't make sense to me. It's in the IRC CTCP tests, and they fail in the same ways each time (an expected response is not received).
The pair of CI runs closest to the change are these:
run 5793: https://github.com/twisted/twisted/runs/2328450554 run 5809: https://github.com/twisted/twisted/runs/2360415474
There are a lot of differences, but sys.version went from 3.7.9 to 3.7.10 between those runs, so that seems like the most likely culprit.
Last working:
sys.version : 3.7.9 (7e6e2bb30ac5, Nov 18 2020, 10:55:52) [PyPy 7.3.3-beta0 with GCC 7.3.1 20180303 (Red Hat 7.3.1-5)] sys.prefix : /opt/hostedtoolcache/PyPy/3.7.9/x64 sys.exec_prefix : /opt/hostedtoolcache/PyPy/3.7.9/x64 sys.executable : /opt/hostedtoolcache/PyPy/3.7.9/x64/bin/python
First broken:
sys.version : 3.7.10 (51efa818fd9b, Apr 04 2021, 11:22:34) [PyPy 7.3.4 with GCC 7.3.1 20180303 (Red Hat 7.3.1-5)] sys.prefix : /opt/hostedtoolcache/PyPy/3.7.10/x64 sys.exec_prefix : /opt/hostedtoolcache/PyPy/3.7.10/x64 sys.executable : /opt/hostedtoolcache/PyPy/3.7.10/x64/bin/python
PyPy's release notes for 7.3.4 don't list anything that jumps out at me:
https://doc.pypy.org/en/latest/whatsnew-pypy3-7.3.4.html
My guess would be some latent timing bug in Twisted that was uncovered by pypy execution time changes (I don't imagine that the CTCP code gets exercised very heavily these days) or perhaps PyPy got a bug.
Wim.
I have setup a pyp3.7.4 and I can reproduce it.
I see 3 options:
* Option A: Skip those tests on pypy and open a separate ticket to fix the test * Option B: Pin pypy 3.7.9 for GHA and open a separate ticket to fix the test and unpin it * Option C: Just fix the tests :)
I am looking at option C for one hour... if I can't find a fix will look into option A.
Cheers
-- Adi Roiban
-- Adi Roiban
Hi PR at https://github.com/twisted/twisted/pull/1592 I hope we can get a quick review and make the trunk green again :) Cheeers On Wed, 28 Apr 2021 at 10:30, Adi Roiban <adi@roiban.ro> wrote:
Hi,
A quick diff to fix it... and a ticket https://twistedmatrix.com/trac/ticket/10187
I will create a PR and we can refine the fix during the review.
diff --git a/src/twisted/words/protocols/irc.py b/src/twisted/words/protocols/irc.py index 75300019b..a97abce2e 100644 --- a/src/twisted/words/protocols/irc.py +++ b/src/twisted/words/protocols/irc.py @@ -3678,10 +3678,10 @@ def ctcpExtract(message): normal_messages.append(messages.pop(0)) odd = not odd
- extended_messages[:] = filter(None, extended_messages) - normal_messages[:] = filter(None, normal_messages) + extended_messages[:] = list(filter(None, extended_messages)) + normal_messages[:] = list(filter(None, normal_messages))
- extended_messages[:] = map(ctcpDequote, extended_messages) + extended_messages[:] = list(map(ctcpDequote, extended_messages))
On Wed, 28 Apr 2021 at 10:01, Adi Roiban <adi@roiban.ro> wrote:
Hi Wim,
Thanks for the report
On Tue, 27 Apr 2021 at 08:14, Wim Lewis <wiml@hhhh.org> wrote:
TLDR: Twisted appears broken on PyPy 7.3.4 (aka "3.7.10")?
I don't have time right now to set up a PyPy-capable environment and try to reproduce this, but perhaps someone does?
Long version:
Trunk has been broken since the last merge a week or so ago, but I don't think the breakage is due to that merge. As an experiment I made a PR based off the last successful version of trunk, with a whitespace change, and it now fails CI as well. So I think the failure must be due to some change that isn't in Twisted or controlled-for by tox.
The failure in all cases is in the pypy-3.7-alldeps-nocov-posix task. Unlike our usual CI problems it doesn't seem to be a random failure: it fails all the time in the same place. But the place doesn't make sense to me. It's in the IRC CTCP tests, and they fail in the same ways each time (an expected response is not received).
The pair of CI runs closest to the change are these:
run 5793: https://github.com/twisted/twisted/runs/2328450554 run 5809: https://github.com/twisted/twisted/runs/2360415474
There are a lot of differences, but sys.version went from 3.7.9 to 3.7.10 between those runs, so that seems like the most likely culprit.
Last working:
sys.version : 3.7.9 (7e6e2bb30ac5, Nov 18 2020, 10:55:52) [PyPy 7.3.3-beta0 with GCC 7.3.1 20180303 (Red Hat 7.3.1-5)] sys.prefix : /opt/hostedtoolcache/PyPy/3.7.9/x64 sys.exec_prefix : /opt/hostedtoolcache/PyPy/3.7.9/x64 sys.executable : /opt/hostedtoolcache/PyPy/3.7.9/x64/bin/python
First broken:
sys.version : 3.7.10 (51efa818fd9b, Apr 04 2021, 11:22:34) [PyPy 7.3.4 with GCC 7.3.1 20180303 (Red Hat 7.3.1-5)] sys.prefix : /opt/hostedtoolcache/PyPy/3.7.10/x64 sys.exec_prefix : /opt/hostedtoolcache/PyPy/3.7.10/x64 sys.executable : /opt/hostedtoolcache/PyPy/3.7.10/x64/bin/python
PyPy's release notes for 7.3.4 don't list anything that jumps out at me:
https://doc.pypy.org/en/latest/whatsnew-pypy3-7.3.4.html
My guess would be some latent timing bug in Twisted that was uncovered by pypy execution time changes (I don't imagine that the CTCP code gets exercised very heavily these days) or perhaps PyPy got a bug.
Wim.
I have setup a pyp3.7.4 and I can reproduce it.
I see 3 options:
* Option A: Skip those tests on pypy and open a separate ticket to fix the test * Option B: Pin pypy 3.7.9 for GHA and open a separate ticket to fix the test and unpin it * Option C: Just fix the tests :)
I am looking at option C for one hour... if I can't find a fix will look into option A.
Cheers
-- Adi Roiban
-- Adi Roiban
-- Adi Roiban
It's merged. On Wed, Apr 28, 2021, at 3:07 AM, Adi Roiban wrote:
Hi
PR at https://github.com/twisted/twisted/pull/1592
I hope we can get a quick review and make the trunk green again :)
Cheeers
On Wed, 28 Apr 2021 at 10:30, Adi Roiban <adi@roiban.ro> wrote:
Hi,
A quick diff to fix it... and a ticket https://twistedmatrix.com/trac/ticket/10187
I will create a PR and we can refine the fix during the review.
diff --git a/src/twisted/words/protocols/irc.py b/src/twisted/words/protocols/irc.py index 75300019b..a97abce2e 100644 --- a/src/twisted/words/protocols/irc.py +++ b/src/twisted/words/protocols/irc.py @@ -3678,10 +3678,10 @@ def ctcpExtract(message): normal_messages.append(messages.pop(0)) odd = not odd
- extended_messages[:] = filter(None, extended_messages) - normal_messages[:] = filter(None, normal_messages) + extended_messages[:] = list(filter(None, extended_messages)) + normal_messages[:] = list(filter(None, normal_messages))
- extended_messages[:] = map(ctcpDequote, extended_messages) + extended_messages[:] = list(map(ctcpDequote, extended_messages))
On Wed, 28 Apr 2021 at 10:01, Adi Roiban <adi@roiban.ro> wrote:
Hi Wim,
Thanks for the report
On Tue, 27 Apr 2021 at 08:14, Wim Lewis <wiml@hhhh.org> wrote:
TLDR: Twisted appears broken on PyPy 7.3.4 (aka "3.7.10")?
I don't have time right now to set up a PyPy-capable environment and try to reproduce this, but perhaps someone does?
Long version:
Trunk has been broken since the last merge a week or so ago, but I don't think the breakage is due to that merge. As an experiment I made a PR based off the last successful version of trunk, with a whitespace change, and it now fails CI as well. So I think the failure must be due to some change that isn't in Twisted or controlled-for by tox.
The failure in all cases is in the pypy-3.7-alldeps-nocov-posix task. Unlike our usual CI problems it doesn't seem to be a random failure: it fails all the time in the same place. But the place doesn't make sense to me. It's in the IRC CTCP tests, and they fail in the same ways each time (an expected response is not received).
The pair of CI runs closest to the change are these:
run 5793: https://github.com/twisted/twisted/runs/2328450554 run 5809: https://github.com/twisted/twisted/runs/2360415474
There are a lot of differences, but sys.version went from 3.7.9 to 3.7.10 between those runs, so that seems like the most likely culprit.
Last working:
sys.version : 3.7.9 (7e6e2bb30ac5, Nov 18 2020, 10:55:52) [PyPy 7.3.3-beta0 with GCC 7.3.1 20180303 (Red Hat 7.3.1-5)] sys.prefix : /opt/hostedtoolcache/PyPy/3.7.9/x64 sys.exec_prefix : /opt/hostedtoolcache/PyPy/3.7.9/x64 sys.executable : /opt/hostedtoolcache/PyPy/3.7.9/x64/bin/python
First broken:
sys.version : 3.7.10 (51efa818fd9b, Apr 04 2021, 11:22:34) [PyPy 7.3.4 with GCC 7.3.1 20180303 (Red Hat 7.3.1-5)] sys.prefix : /opt/hostedtoolcache/PyPy/3.7.10/x64 sys.exec_prefix : /opt/hostedtoolcache/PyPy/3.7.10/x64 sys.executable : /opt/hostedtoolcache/PyPy/3.7.10/x64/bin/python
PyPy's release notes for 7.3.4 don't list anything that jumps out at me:
https://doc.pypy.org/en/latest/whatsnew-pypy3-7.3.4.html
My guess would be some latent timing bug in Twisted that was uncovered by pypy execution time changes (I don't imagine that the CTCP code gets exercised very heavily these days) or perhaps PyPy got a bug.
Wim.
I have setup a pyp3.7.4 and I can reproduce it.
I see 3 options:
* Option A: Skip those tests on pypy and open a separate ticket to fix the test * Option B: Pin pypy 3.7.9 for GHA and open a separate ticket to fix the test and unpin it * Option C: Just fix the tests :)
I am looking at option C for one hour... if I can't find a fix will look into option A.
Cheers
-- Adi Roiban
-- Adi Roiban
-- Adi Roiban _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com <mailto:Twisted-Python%40twistedmatrix.com> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On Thu, 29 Apr 2021 at 07:23, Glyph <glyph@twistedmatrix.com> wrote:
It's merged.
Thanks for the merge. I will try to help Twisted by keeping the trunk green. If there is any other issue with the trunk branch getting red out of the blue, ping me and I will try to help. ------- We now have a fix for Twisted, but I think that this is a bug in PYPY 7.3.4 FYI I have reported this bug upstream. https://foss.heptapod.net/pypy/pypy/-/issues/3451 Cheers
On Wed, Apr 28, 2021, at 3:07 AM, Adi Roiban wrote:
Hi
PR at https://github.com/twisted/twisted/pull/1592
I hope we can get a quick review and make the trunk green again :)
Cheeers
On Wed, 28 Apr 2021 at 10:30, Adi Roiban <adi@roiban.ro> wrote:
Hi,
A quick diff to fix it... and a ticket https://twistedmatrix.com/trac/ticket/10187
I will create a PR and we can refine the fix during the review.
diff --git a/src/twisted/words/protocols/irc.py b/src/twisted/words/protocols/irc.py index 75300019b..a97abce2e 100644 --- a/src/twisted/words/protocols/irc.py +++ b/src/twisted/words/protocols/irc.py @@ -3678,10 +3678,10 @@ def ctcpExtract(message): normal_messages.append(messages.pop(0)) odd = not odd
- extended_messages[:] = filter(None, extended_messages) - normal_messages[:] = filter(None, normal_messages) + extended_messages[:] = list(filter(None, extended_messages)) + normal_messages[:] = list(filter(None, normal_messages))
- extended_messages[:] = map(ctcpDequote, extended_messages) + extended_messages[:] = list(map(ctcpDequote, extended_messages))
On Wed, 28 Apr 2021 at 10:01, Adi Roiban <adi@roiban.ro> wrote:
Hi Wim,
Thanks for the report
On Tue, 27 Apr 2021 at 08:14, Wim Lewis <wiml@hhhh.org> wrote:
TLDR: Twisted appears broken on PyPy 7.3.4 (aka "3.7.10")?
I don't have time right now to set up a PyPy-capable environment and try to reproduce this, but perhaps someone does?
Long version:
Trunk has been broken since the last merge a week or so ago, but I don't think the breakage is due to that merge. As an experiment I made a PR based off the last successful version of trunk, with a whitespace change, and it now fails CI as well. So I think the failure must be due to some change that isn't in Twisted or controlled-for by tox.
The failure in all cases is in the pypy-3.7-alldeps-nocov-posix task. Unlike our usual CI problems it doesn't seem to be a random failure: it fails all the time in the same place. But the place doesn't make sense to me. It's in the IRC CTCP tests, and they fail in the same ways each time (an expected response is not received).
The pair of CI runs closest to the change are these:
run 5793: https://github.com/twisted/twisted/runs/2328450554 run 5809: https://github.com/twisted/twisted/runs/2360415474
There are a lot of differences, but sys.version went from 3.7.9 to 3.7.10 between those runs, so that seems like the most likely culprit.
Last working:
sys.version : 3.7.9 (7e6e2bb30ac5, Nov 18 2020, 10:55:52) [PyPy 7.3.3-beta0 with GCC 7.3.1 20180303 (Red Hat 7.3.1-5)] sys.prefix : /opt/hostedtoolcache/PyPy/3.7.9/x64 sys.exec_prefix : /opt/hostedtoolcache/PyPy/3.7.9/x64 sys.executable : /opt/hostedtoolcache/PyPy/3.7.9/x64/bin/python
First broken:
sys.version : 3.7.10 (51efa818fd9b, Apr 04 2021, 11:22:34) [PyPy 7.3.4 with GCC 7.3.1 20180303 (Red Hat 7.3.1-5)] sys.prefix : /opt/hostedtoolcache/PyPy/3.7.10/x64 sys.exec_prefix : /opt/hostedtoolcache/PyPy/3.7.10/x64 sys.executable : /opt/hostedtoolcache/PyPy/3.7.10/x64/bin/python
PyPy's release notes for 7.3.4 don't list anything that jumps out at me:
https://doc.pypy.org/en/latest/whatsnew-pypy3-7.3.4.html
My guess would be some latent timing bug in Twisted that was uncovered by pypy execution time changes (I don't imagine that the CTCP code gets exercised very heavily these days) or perhaps PyPy got a bug.
Wim.
I have setup a pyp3.7.4 and I can reproduce it.
I see 3 options:
* Option A: Skip those tests on pypy and open a separate ticket to fix the test * Option B: Pin pypy 3.7.9 for GHA and open a separate ticket to fix the test and unpin it * Option C: Just fix the tests :)
I am looking at option C for one hour... if I can't find a fix will look into option A.
Cheers
-- Adi Roiban
-- Adi Roiban
-- Adi Roiban _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
-- Adi Roiban
On Saturday, May 1, 2021 1:58:45 AM PDT, Adi Roiban wrote:
We now have a fix for Twisted, but I think that this is a bug in PYPY 7.3.4
FYI I have reported this bug upstream.
Looks like we weren't the only project to run into this problem! (I think what happens is that, although the right-hand-side is always evaluated before the list assignment starts, it's undefined whether the generator returned by the RHS is completely consumed before the list begins to be altered — so, even though pypy7.3.5 will be reverting to the old behavior, it's good that twisted no longer depends on it.) Thanks for all your work keeping CI happy, Adi!
participants (3)
-
Adi Roiban -
Glyph -
Wim Lewis