[Twisted-Python] Does anyone know why trunk is failing CI on PyPy 7.3.4?
![](https://secure.gravatar.com/avatar/82bbf56e965eb9c44f5f8a40c6958d77.jpg?s=120&d=mm&r=g)
TLDR: Twisted appears broken on PyPy 7.3.4 (aka "3.7.10")? I don't have time right now to set up a PyPy-capable environment and try to reproduce this, but perhaps someone does? Long version: Trunk has been broken since the last merge a week or so ago, but I don't think the breakage is due to that merge. As an experiment I made a PR based off the last successful version of trunk, with a whitespace change, and it now fails CI as well. So I think the failure must be due to some change that isn't in Twisted or controlled-for by tox. The failure in all cases is in the pypy-3.7-alldeps-nocov-posix task. Unlike our usual CI problems it doesn't seem to be a random failure: it fails all the time in the same place. But the place doesn't make sense to me. It's in the IRC CTCP tests, and they fail in the same ways each time (an expected response is not received). The pair of CI runs closest to the change are these: run 5793: https://github.com/twisted/twisted/runs/2328450554 run 5809: https://github.com/twisted/twisted/runs/2360415474 There are a lot of differences, but sys.version went from 3.7.9 to 3.7.10 between those runs, so that seems like the most likely culprit. Last working:
First broken:
PyPy's release notes for 7.3.4 don't list anything that jumps out at me: https://doc.pypy.org/en/latest/whatsnew-pypy3-7.3.4.html My guess would be some latent timing bug in Twisted that was uncovered by pypy execution time changes (I don't imagine that the CTCP code gets exercised very heavily these days) or perhaps PyPy got a bug. Wim.
![](https://secure.gravatar.com/avatar/c194a4d2f2f8269aa052942e87985198.jpg?s=120&d=mm&r=g)
Hi Wim, Thanks for the report On Tue, 27 Apr 2021 at 08:14, Wim Lewis <wiml@hhhh.org> wrote:
I have setup a pyp3.7.4 and I can reproduce it. I see 3 options: * Option A: Skip those tests on pypy and open a separate ticket to fix the test * Option B: Pin pypy 3.7.9 for GHA and open a separate ticket to fix the test and unpin it * Option C: Just fix the tests :) I am looking at option C for one hour... if I can't find a fix will look into option A. Cheers -- Adi Roiban
![](https://secure.gravatar.com/avatar/c194a4d2f2f8269aa052942e87985198.jpg?s=120&d=mm&r=g)
Hi, A quick diff to fix it... and a ticket https://twistedmatrix.com/trac/ticket/10187 I will create a PR and we can refine the fix during the review. diff --git a/src/twisted/words/protocols/irc.py b/src/twisted/words/protocols/irc.py index 75300019b..a97abce2e 100644 --- a/src/twisted/words/protocols/irc.py +++ b/src/twisted/words/protocols/irc.py @@ -3678,10 +3678,10 @@ def ctcpExtract(message): normal_messages.append(messages.pop(0)) odd = not odd - extended_messages[:] = filter(None, extended_messages) - normal_messages[:] = filter(None, normal_messages) + extended_messages[:] = list(filter(None, extended_messages)) + normal_messages[:] = list(filter(None, normal_messages)) - extended_messages[:] = map(ctcpDequote, extended_messages) + extended_messages[:] = list(map(ctcpDequote, extended_messages)) On Wed, 28 Apr 2021 at 10:01, Adi Roiban <adi@roiban.ro> wrote:
-- Adi Roiban
![](https://secure.gravatar.com/avatar/c194a4d2f2f8269aa052942e87985198.jpg?s=120&d=mm&r=g)
Hi PR at https://github.com/twisted/twisted/pull/1592 I hope we can get a quick review and make the trunk green again :) Cheeers On Wed, 28 Apr 2021 at 10:30, Adi Roiban <adi@roiban.ro> wrote:
-- Adi Roiban
![](https://secure.gravatar.com/avatar/c194a4d2f2f8269aa052942e87985198.jpg?s=120&d=mm&r=g)
On Thu, 29 Apr 2021 at 07:23, Glyph <glyph@twistedmatrix.com> wrote:
It's merged.
Thanks for the merge. I will try to help Twisted by keeping the trunk green. If there is any other issue with the trunk branch getting red out of the blue, ping me and I will try to help. ------- We now have a fix for Twisted, but I think that this is a bug in PYPY 7.3.4 FYI I have reported this bug upstream. https://foss.heptapod.net/pypy/pypy/-/issues/3451 Cheers
-- Adi Roiban
![](https://secure.gravatar.com/avatar/82bbf56e965eb9c44f5f8a40c6958d77.jpg?s=120&d=mm&r=g)
On Saturday, May 1, 2021 1:58:45 AM PDT, Adi Roiban wrote:
Looks like we weren't the only project to run into this problem! (I think what happens is that, although the right-hand-side is always evaluated before the list assignment starts, it's undefined whether the generator returned by the RHS is completely consumed before the list begins to be altered — so, even though pypy7.3.5 will be reverting to the old behavior, it's good that twisted no longer depends on it.) Thanks for all your work keeping CI happy, Adi!
![](https://secure.gravatar.com/avatar/c194a4d2f2f8269aa052942e87985198.jpg?s=120&d=mm&r=g)
Hi Wim, Thanks for the report On Tue, 27 Apr 2021 at 08:14, Wim Lewis <wiml@hhhh.org> wrote:
I have setup a pyp3.7.4 and I can reproduce it. I see 3 options: * Option A: Skip those tests on pypy and open a separate ticket to fix the test * Option B: Pin pypy 3.7.9 for GHA and open a separate ticket to fix the test and unpin it * Option C: Just fix the tests :) I am looking at option C for one hour... if I can't find a fix will look into option A. Cheers -- Adi Roiban
![](https://secure.gravatar.com/avatar/c194a4d2f2f8269aa052942e87985198.jpg?s=120&d=mm&r=g)
Hi, A quick diff to fix it... and a ticket https://twistedmatrix.com/trac/ticket/10187 I will create a PR and we can refine the fix during the review. diff --git a/src/twisted/words/protocols/irc.py b/src/twisted/words/protocols/irc.py index 75300019b..a97abce2e 100644 --- a/src/twisted/words/protocols/irc.py +++ b/src/twisted/words/protocols/irc.py @@ -3678,10 +3678,10 @@ def ctcpExtract(message): normal_messages.append(messages.pop(0)) odd = not odd - extended_messages[:] = filter(None, extended_messages) - normal_messages[:] = filter(None, normal_messages) + extended_messages[:] = list(filter(None, extended_messages)) + normal_messages[:] = list(filter(None, normal_messages)) - extended_messages[:] = map(ctcpDequote, extended_messages) + extended_messages[:] = list(map(ctcpDequote, extended_messages)) On Wed, 28 Apr 2021 at 10:01, Adi Roiban <adi@roiban.ro> wrote:
-- Adi Roiban
![](https://secure.gravatar.com/avatar/c194a4d2f2f8269aa052942e87985198.jpg?s=120&d=mm&r=g)
Hi PR at https://github.com/twisted/twisted/pull/1592 I hope we can get a quick review and make the trunk green again :) Cheeers On Wed, 28 Apr 2021 at 10:30, Adi Roiban <adi@roiban.ro> wrote:
-- Adi Roiban
![](https://secure.gravatar.com/avatar/c194a4d2f2f8269aa052942e87985198.jpg?s=120&d=mm&r=g)
On Thu, 29 Apr 2021 at 07:23, Glyph <glyph@twistedmatrix.com> wrote:
It's merged.
Thanks for the merge. I will try to help Twisted by keeping the trunk green. If there is any other issue with the trunk branch getting red out of the blue, ping me and I will try to help. ------- We now have a fix for Twisted, but I think that this is a bug in PYPY 7.3.4 FYI I have reported this bug upstream. https://foss.heptapod.net/pypy/pypy/-/issues/3451 Cheers
-- Adi Roiban
![](https://secure.gravatar.com/avatar/82bbf56e965eb9c44f5f8a40c6958d77.jpg?s=120&d=mm&r=g)
On Saturday, May 1, 2021 1:58:45 AM PDT, Adi Roiban wrote:
Looks like we weren't the only project to run into this problem! (I think what happens is that, although the right-hand-side is always evaluated before the list assignment starts, it's undefined whether the generator returned by the RHS is completely consumed before the list begins to be altered — so, even though pypy7.3.5 will be reverting to the old behavior, it's good that twisted no longer depends on it.) Thanks for all your work keeping CI happy, Adi!
participants (3)
-
Adi Roiban
-
Glyph
-
Wim Lewis