INCOMPATIBLE CHANGE: Towards the adoption of coroutines
![](https://secure.gravatar.com/avatar/607cfd4a5b41fe6c886c978128b9c03e.jpg?s=120&d=mm&r=g)
Hello, I've just written up an idea for an approach to getting many parts of Twisted and Twisted-based applications onto coroutines (ie `async def`). Details are given on https://twistedmatrix.com/trac/ticket/10327 The change involves the potential for some incompatibilities - though as I described on the ticket, I think these are likely to be very rare in practice and worth imposing for the resulting payoff. This thread represents the start of the process for making incompatible changes to Twisted as described at https://twistedmatrix.com/documents/current/core/development/policy/compatib... There is not yet a branch but there is a patch in a comment on the issue (the change is extremely small) for anyone to try out or comment on. I will follow up when there is a version closer to what would be merged into trunk. Meanwhile, feedback is welcome. Jean-Paul
![](https://secure.gravatar.com/avatar/3d37232726396a1d3c7412dd915095ea.jpg?s=120&d=mm&r=g)
On 7/4/22 11:21, Jean-Paul Calderone wrote:
[quietly, from the house across the street] woo! yeah! right on! ... In terms of returning a coroutine, I believe that Python (especially under ASYNCIODEBUG=1) gets annoyed and spews warnings if a coroutine is not awaited (I believe when it is GC'd). Because returning a coroutine without scheduling it is possibly able to run into this case (e.g. a coroutine that performs a 'send clean goodbye' that is not scheduled until the connection is started to be shut down, but is never called on an unclean shutdown, just as a strawman), I think that it is completely valid to ignore it. Especially since returning a function-that-returns-a-couroutine that you then call when you need it is 1000x better of an API. That is to say, from my increasingly rusty memory of how all of this works, I believe this is a good way to go. - Amber
![](https://secure.gravatar.com/avatar/eba6eb871de2549c7447a8701352cd35.jpg?s=120&d=mm&r=g)
Thanks Jean-Paul for the email On Thu, 7 Apr 2022 at 02:22, Jean-Paul Calderone <exarkun@twistedmatrix.com> wrote:
There is not yet a branch but there is a patch in a comment on the issue (the change is extremely small) for anyone to try out or comment on. I will follow up when there is a version closer to what would be merged into trunk. Meanwhile, feedback is welcome.
Jean-Paul
I am not using coroutines. And I try to avoid using maybeDeferred But I think it is best to add support for coroutines in maybeDeferred. We have limited dev resources so it's OK to go with this backward incompatible change. -- Adi Roiban
![](https://secure.gravatar.com/avatar/64dae5cd646e90a4d41a0ff402bbd54f.jpg?s=120&d=mm&r=g)
On Wed, Apr 6, 2022 at 9:21 PM Jean-Paul Calderone < exarkun@twistedmatrix.com> wrote:
Thanks to everyone who supplied feedback. Since it seemed positive, I've taken the next step in the process - https://github.com/twisted/twisted/pull/1718 is a PR that implements the proposed change in behavior. According to the compatibility policy, this should be accepted if three committers approve and there are no objections for one week. I'm not in any huge rush so I don't imagine trying to merge this in exactly 7 days from this email (in fact, I'm on vacation all next week...) Please have a look and let me know if you spot any problems. Thanks, Jean-Paul
![](https://secure.gravatar.com/avatar/607cfd4a5b41fe6c886c978128b9c03e.jpg?s=120&d=mm&r=g)
On Mon, Apr 11, 2022 at 6:31 PM Glyph <glyph@twistedmatrix.com> wrote:
Thank you to the three reviewers so far - glyph, twm, and adiroiban. I believe that the PR is now in its final form and has been approved by three Twisted committers. So at this point, the change will land. As I mentioned previously, I probably won't land it Monday but I may towards the end of the week - unless there is further feedback before then. Jean-Paul
![](https://secure.gravatar.com/avatar/3d37232726396a1d3c7412dd915095ea.jpg?s=120&d=mm&r=g)
On 7/4/22 11:21, Jean-Paul Calderone wrote:
[quietly, from the house across the street] woo! yeah! right on! ... In terms of returning a coroutine, I believe that Python (especially under ASYNCIODEBUG=1) gets annoyed and spews warnings if a coroutine is not awaited (I believe when it is GC'd). Because returning a coroutine without scheduling it is possibly able to run into this case (e.g. a coroutine that performs a 'send clean goodbye' that is not scheduled until the connection is started to be shut down, but is never called on an unclean shutdown, just as a strawman), I think that it is completely valid to ignore it. Especially since returning a function-that-returns-a-couroutine that you then call when you need it is 1000x better of an API. That is to say, from my increasingly rusty memory of how all of this works, I believe this is a good way to go. - Amber
![](https://secure.gravatar.com/avatar/eba6eb871de2549c7447a8701352cd35.jpg?s=120&d=mm&r=g)
Thanks Jean-Paul for the email On Thu, 7 Apr 2022 at 02:22, Jean-Paul Calderone <exarkun@twistedmatrix.com> wrote:
There is not yet a branch but there is a patch in a comment on the issue (the change is extremely small) for anyone to try out or comment on. I will follow up when there is a version closer to what would be merged into trunk. Meanwhile, feedback is welcome.
Jean-Paul
I am not using coroutines. And I try to avoid using maybeDeferred But I think it is best to add support for coroutines in maybeDeferred. We have limited dev resources so it's OK to go with this backward incompatible change. -- Adi Roiban
![](https://secure.gravatar.com/avatar/64dae5cd646e90a4d41a0ff402bbd54f.jpg?s=120&d=mm&r=g)
On Wed, Apr 6, 2022 at 9:21 PM Jean-Paul Calderone < exarkun@twistedmatrix.com> wrote:
Thanks to everyone who supplied feedback. Since it seemed positive, I've taken the next step in the process - https://github.com/twisted/twisted/pull/1718 is a PR that implements the proposed change in behavior. According to the compatibility policy, this should be accepted if three committers approve and there are no objections for one week. I'm not in any huge rush so I don't imagine trying to merge this in exactly 7 days from this email (in fact, I'm on vacation all next week...) Please have a look and let me know if you spot any problems. Thanks, Jean-Paul
![](https://secure.gravatar.com/avatar/607cfd4a5b41fe6c886c978128b9c03e.jpg?s=120&d=mm&r=g)
On Mon, Apr 11, 2022 at 6:31 PM Glyph <glyph@twistedmatrix.com> wrote:
Thank you to the three reviewers so far - glyph, twm, and adiroiban. I believe that the PR is now in its final form and has been approved by three Twisted committers. So at this point, the change will land. As I mentioned previously, I probably won't land it Monday but I may towards the end of the week - unless there is further feedback before then. Jean-Paul
participants (6)
-
Adi Roiban
-
Amber Brown (hawkowl)
-
Chris Withers
-
Glyph
-
Jean-Paul Calderone
-
Jean-Paul Calderone