fun new directions for Twisted for engaging new developers
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
Hello Twisted list, We had a brief Twisted birds-of-a-feather session at PyCon 2024. One issue that came up was that while Twisted works fine for the things that it does, we don't have a great onboarding process to motivate new developers to get involved and maintain it or build new things. In particular, Twisted's main value these days is no longer in the reactor, as the stdlib's core asyncio functionality is quite capable, but in our suite of protocol implementations and integrations. However, "it's got a ton of custom wire protocols" is less interesting in an era where there are fewer mainstream things that use custom wire protocols. So what are interesting things that we already do with wire protocols that, with a fresh coat of paint, could appeal to an audience of new developers to revitalize the project in general? To that end, we discussed a few projects: We should resurrect the effort to build a first-party websockets implementation: <https://github.com/twisted/twisted/issues/4173>. It's been nice to have Autobahn available, but websockets are by far the quickest and easiest way to provide a native demonstration of Twisted's capabilities. Luckily, we do not have to build an implementation ourselves, as a maintained sans-io kernel implementation exists in the <https://github.com/python-hyper/wsproto> project. So we just need to do a direct integration with the Resource model in twisted.web. I will probably do some work on this project myself. Twitch's chat is IRC <https://dev.twitch.tv/docs/irc/>, but Twisted's IRC implementation is missing some implementation details. An IRC bot tutorial could thus actually be a very relevant introduction to a large audience of developers. Joel McGrady has volunteered for this effort and has written up some of the IRCv3 issues that will need addressing as part of it: https://github.com/twisted/twisted/issues/12180 We could probably do something interesting and fun with email, if we updated twisted.mail to make sure it worked with recent mail clients and did a little tutorial? Email is still relentlessly popular despite decades of progress. Nobody has volunteered for this yet. We already have a ton of work on SMB that just needs to be un-stuck, both in terms of reviews and fixes: <https://github.com/twisted/twisted/pull/1274>. I really appreciate Ian contributing this to Twisted and I'd like it to see it eventually make it into a release, so I'm highlighting it here. If anyone wants to pick up responding to review feedback, you can make a fork of his fork, grab the branch and open a new PR to start addressing things. Everyone then mumbled "HTTP/3"? But nobody seemed to actually care about that. All of these efforts should be done documentation-first, to try to evaluate how to build a successor to the aging "finger" tutorial series, and to make sure that we have something to point new developers at so that they can find their way around Twisted. -g
![](https://secure.gravatar.com/avatar/eba6eb871de2549c7447a8701352cd35.jpg?s=120&d=mm&r=g)
Hi Glyph, Thanks for sharing this info from PyCon. Much appreciated. On Fri, 31 May 2024 at 00:07, Glyph <glyph@twistedmatrix.com> wrote:
However, "it's got a ton of custom wire protocols" is less interesting in
To that end, we discussed a few projects:
1. For Twisted web, before looking into websocket I think that it would help to add support for streaming. The main thing is to allow dispatching / traversing the URL as soon as the headers are received, rather than as soon as the whole request is received. I am doing a few "custom" HTTP things and since the beginning I had to use a fork of twisted.web But yes. Websocket would be nice. 3. SMTP is still popular. I am using it in production with simple authentication methods. I don't think that POP3 is popular. And I don't know if IMAP4 is still relevant. For SMTP and IMAP4 what we can do is provide "out of the box" helpers for implementing the whole XOAUTH2 dance for various OAuth2 providers. Recently I had to interact with Exchange Online. I have implemented XOAUTH2 with Entra ID, but the Exchange Online security setup was complicated. I ended up just using the MS Graph API for the job. 4. For SMB , I don't know if we need yet another Python SMB library. I am using https://github.com/jborean93/smbprotocol .... it uses thread, but I know that the maintainer would like to have support for async. In general, I would prefer to see Python developers working on sans-io libraries and then you can have them backed by asyncio, twisted, trio, greenlet. 5. Before implementing HTTP3, I think it would help to get support for streaming in HTTP 1.1 and web socket. --------- The main reason why I use Twisted is the commitment for backward compatibility, twisted.conch.ssh and ldaptor I have not yet tried AsyncSSH I don't know if there is any other LDAP server for Python, with MIT or BSD licence. All of these efforts should be done documentation-first, to try to evaluate
Writing documentation is hard... and in the end , new developers will always go with the project that has the best documentation... even if the project might not have a clean code or a good suite of automated tests. I think that one of the big issues that a Twisted developer is facing is the lack of documentation. I feel that with the goal of creating secure and complete examples, the documentation and examples are hard to understand. A new developer might be overwhelmed by all the complexity and then try some other library. I remember that I was trying to create a simple SSH server example. And the code ended up quite complex to avoid users reusing a testing SSH key or supporting both user + password and ssh key... and using the Twisted Cred infrastructure. Recently I checked the IMAP4 client. There is an example, but I found it hard to navigate. The TrivialPrompter adds extra complexity, and the whole code would be much easier to read with inlineCallbacks. Another example is setting up an SSH server on Windows or SFTP server or client on any OS... this is hard Regards -g
-- Adi Roiban
![](https://secure.gravatar.com/avatar/d3dd3ba5cbd197a5927366c6c9a5003d.jpg?s=120&d=mm&r=g)
On Thu, May 30, 2024, at 19:05, Glyph wrote:
One issue that came up was that while Twisted works fine for the things that it does, we don't have a great onboarding process to motivate new developers to get involved and maintain it or build new things.
Perhaps it would be worth revisiting anyio support? It's been a few years and I am now working in asyncio (because that's what the company started with), but here are a few relevant links if anyone wanted to try to get this moving again. https://github.com/twisted/twisted/issues/9815 https://github.com/twisted/twisted/pull/1263 https://github.com/python-trio/sniffio/pull/13 https://github.com/python-trio/sniffio/pull/10 I understand that the goal here is high value with minimal resources in an effort to create some interest. I think that structured concurrency may be a bit buzzy but is also really useful for getting async code 'correct'. It also isn't a major rework to get support. Cheers, -kyle
![](https://secure.gravatar.com/avatar/a93db92c60e9d5435d204407264457f2.jpg?s=120&d=mm&r=g)
On Thu, 30 May 2024, at 17:05, Glyph wrote:
We should resurrect the effort to build a first-party websockets implementation: <https://github.com/twisted/twisted/issues/4173>. It's been nice to have Autobahn available, but websockets are by far the quickest and easiest way to provide a *native* demonstration of Twisted's capabilities. Luckily, we do not have to build an implementation ourselves, as a maintained sans-io kernel implementation exists in the <https://github.com/python-hyper/wsproto> project. So we just need to do a direct integration with the Resource model in twisted.web. I will probably do some work on this project myself.
Hello, I don't want to tell excited developers to _not_ do a thing, but to me this seems a little weird since Autobahn already does all of this (including "a direct integration with the Resource model in twisted.web"). And has a done a _ton_ of work to ensure strict RFC compliance and high performance. I'm 98% sure that PRs to improve any aspects of the Autobahn implementation would be accepted (by that project). Although IIRC it wasn't written with exact "Sans-IO" guidance in mind [1], the implementation is already 'fairly' separated since it supports both asyncio and Twisted (your choice) out of the box. It has been a few years while since I've directly worked on Autobahn / Crossbar but I'd be happy to pair with anyone excited about this to pass on what I do know and where things are located. I guess what I'm saying is that maybe modifying the proposed project slightly to become "integrate Hyper's state machine into Autobahn" makes a lot more sense. Assuming they handle that vast array of edge cases and weirdness that is WebSockets ;) ...or perhaps I misunderstand what "demonstrate Twisted's native capabilities" actually means in this context. [1] I don't think e.g. a "threaded" / blocking backend would work well in the current implementation, but adding an AnyIO or curio one should be (relatively) straightforward. -- meejah
![](https://secure.gravatar.com/avatar/7cedcd5fe799f836e4ea4f9560e873d0.jpg?s=120&d=mm&r=g)
Am 18.06.24 um 18:36 schrieb meejah:
how is "native" defined? as in, part of and installed as Twisted package? why is that desirable? why wouldn't I care about "other stuff" that is _also_ install apart from Twisted? like eg a Posix kernel, or a Python run-time?
fwiw, Autobahn was built "sans-IO" pretty much from the beginning, simply as a necessity of targeting _both_ Twisted and asyncio. but Autobahn _is_ not zero-sans, but _contains_ a zero-sans websocket implementation this can be seen from the absence of both Twisted and asyncio imports, only txaio imports here https://github.com/crossbario/autobahn-python/blob/f38f16ba28fa253dee951068c... the reason for txaio is: Autobahn not _only_ provides a zero-sans WebSocket implementation, but also respective network layer bindings (to Twisted and asyncio), and to aggregate/factorize that code, there is txaio Autobahn is also a complete (eg it has facilities for WebSocket standard extensions, including deflate) and compliant WebSocket implementation, both extensively tested using Autobahn Testsuite, which was the first and still is the most complete WebSocket implementation testsuite Autobahn also is proven real-world, and using both Twisted and asyncio, and over various transports, including stuff like Unix domain sockets or stdio or serial .. Finally, Autobahn _also_ includes not only WebSocket, but WAMP as well - integrated with WebSocket as one transport ..
Yes, exactly, a _ton_ of work .. like writing a testsuite in the first place and fixing/nagging upstream bugs .. from low-level (UTF8) to high level ... testing in various real-world settings
I'm 98% sure that PRs to improve any aspects of the Autobahn implementation would be accepted (by that project). Although IIRC it
"improve", yes, sure! "any": that depends. what is wrong with it anyways? rgd WebSocket standard compliance, I would say it is as good as it gets.
wasn't written with exact "Sans-IO" guidance in mind [1], the
I would say we did it without knowing there would be a name for it later on;)
implementation is already 'fairly' separated since it supports both
it is perfectly separated (see above), including proper internal APIs for zero-sans https://github.com/crossbario/autobahn-python/blob/f38f16ba28fa253dee951068c...
I don't see a need to integrate another state machine, can you point me to a bug? otherwise, what's the upside?
same here, what is "native" supposed to mean in this context? code shipped as part of Twisted? and if so, why? why ship the world as part of Twisted?
fwiw, rgd Autobahn, the one transport which is missing and I am interested in is: VirtIO VirtIO would allow zero-copy, not only single-copy - which is already available in Posix via UDS. "AnyIO or curio": these are just wrappers around classic Posix sockets, right? I mean, Autobahn on Linux should and is using epoll already, so what exactly is wrong with that?
![](https://secure.gravatar.com/avatar/7cedcd5fe799f836e4ea4f9560e873d0.jpg?s=120&d=mm&r=g)
to make that a bit more concrete, how could we implement an Autobahn transport for WebSocket - that is an implementation of autobahn.websocket.interfaces.IWebSocketChannel running over VirtIO and using virtqueues, and using epoll for event integration. does Python have proper wrappers for Linux VirtIO / virtqueues already? integrated with epoll in Python? implemented in CPython and in PyPy? if so, let's do it;) I could have my Rust code talk to Python with zero-copy (at least on the Rust side) - and using WebSocket as proper/standard message framing the one aspect I am not sure about: WebSocket provides framing .. as it runs over a byte level transport .. and for WAMP, we wouldn't need that as VirtIO already is message based. if we would go down that road, it'll means providing an implementation of autobahn.wamp.interfaces.ITransport https://github.com/crossbario/autobahn-python/blob/f38f16ba28fa253dee951068c... for VirtIO / virtqueues. anyways, I don't really think it matters much performance wise. I can shift out bytes into memory up to maximum main memory bandwidth limited by hardware in Rust easily, and "almost" in PyPy, but still ..
![](https://secure.gravatar.com/avatar/eba6eb871de2549c7447a8701352cd35.jpg?s=120&d=mm&r=g)
Hi Glyph, Thanks for sharing this info from PyCon. Much appreciated. On Fri, 31 May 2024 at 00:07, Glyph <glyph@twistedmatrix.com> wrote:
However, "it's got a ton of custom wire protocols" is less interesting in
To that end, we discussed a few projects:
1. For Twisted web, before looking into websocket I think that it would help to add support for streaming. The main thing is to allow dispatching / traversing the URL as soon as the headers are received, rather than as soon as the whole request is received. I am doing a few "custom" HTTP things and since the beginning I had to use a fork of twisted.web But yes. Websocket would be nice. 3. SMTP is still popular. I am using it in production with simple authentication methods. I don't think that POP3 is popular. And I don't know if IMAP4 is still relevant. For SMTP and IMAP4 what we can do is provide "out of the box" helpers for implementing the whole XOAUTH2 dance for various OAuth2 providers. Recently I had to interact with Exchange Online. I have implemented XOAUTH2 with Entra ID, but the Exchange Online security setup was complicated. I ended up just using the MS Graph API for the job. 4. For SMB , I don't know if we need yet another Python SMB library. I am using https://github.com/jborean93/smbprotocol .... it uses thread, but I know that the maintainer would like to have support for async. In general, I would prefer to see Python developers working on sans-io libraries and then you can have them backed by asyncio, twisted, trio, greenlet. 5. Before implementing HTTP3, I think it would help to get support for streaming in HTTP 1.1 and web socket. --------- The main reason why I use Twisted is the commitment for backward compatibility, twisted.conch.ssh and ldaptor I have not yet tried AsyncSSH I don't know if there is any other LDAP server for Python, with MIT or BSD licence. All of these efforts should be done documentation-first, to try to evaluate
Writing documentation is hard... and in the end , new developers will always go with the project that has the best documentation... even if the project might not have a clean code or a good suite of automated tests. I think that one of the big issues that a Twisted developer is facing is the lack of documentation. I feel that with the goal of creating secure and complete examples, the documentation and examples are hard to understand. A new developer might be overwhelmed by all the complexity and then try some other library. I remember that I was trying to create a simple SSH server example. And the code ended up quite complex to avoid users reusing a testing SSH key or supporting both user + password and ssh key... and using the Twisted Cred infrastructure. Recently I checked the IMAP4 client. There is an example, but I found it hard to navigate. The TrivialPrompter adds extra complexity, and the whole code would be much easier to read with inlineCallbacks. Another example is setting up an SSH server on Windows or SFTP server or client on any OS... this is hard Regards -g
-- Adi Roiban
![](https://secure.gravatar.com/avatar/d3dd3ba5cbd197a5927366c6c9a5003d.jpg?s=120&d=mm&r=g)
On Thu, May 30, 2024, at 19:05, Glyph wrote:
One issue that came up was that while Twisted works fine for the things that it does, we don't have a great onboarding process to motivate new developers to get involved and maintain it or build new things.
Perhaps it would be worth revisiting anyio support? It's been a few years and I am now working in asyncio (because that's what the company started with), but here are a few relevant links if anyone wanted to try to get this moving again. https://github.com/twisted/twisted/issues/9815 https://github.com/twisted/twisted/pull/1263 https://github.com/python-trio/sniffio/pull/13 https://github.com/python-trio/sniffio/pull/10 I understand that the goal here is high value with minimal resources in an effort to create some interest. I think that structured concurrency may be a bit buzzy but is also really useful for getting async code 'correct'. It also isn't a major rework to get support. Cheers, -kyle
![](https://secure.gravatar.com/avatar/a93db92c60e9d5435d204407264457f2.jpg?s=120&d=mm&r=g)
On Thu, 30 May 2024, at 17:05, Glyph wrote:
We should resurrect the effort to build a first-party websockets implementation: <https://github.com/twisted/twisted/issues/4173>. It's been nice to have Autobahn available, but websockets are by far the quickest and easiest way to provide a *native* demonstration of Twisted's capabilities. Luckily, we do not have to build an implementation ourselves, as a maintained sans-io kernel implementation exists in the <https://github.com/python-hyper/wsproto> project. So we just need to do a direct integration with the Resource model in twisted.web. I will probably do some work on this project myself.
Hello, I don't want to tell excited developers to _not_ do a thing, but to me this seems a little weird since Autobahn already does all of this (including "a direct integration with the Resource model in twisted.web"). And has a done a _ton_ of work to ensure strict RFC compliance and high performance. I'm 98% sure that PRs to improve any aspects of the Autobahn implementation would be accepted (by that project). Although IIRC it wasn't written with exact "Sans-IO" guidance in mind [1], the implementation is already 'fairly' separated since it supports both asyncio and Twisted (your choice) out of the box. It has been a few years while since I've directly worked on Autobahn / Crossbar but I'd be happy to pair with anyone excited about this to pass on what I do know and where things are located. I guess what I'm saying is that maybe modifying the proposed project slightly to become "integrate Hyper's state machine into Autobahn" makes a lot more sense. Assuming they handle that vast array of edge cases and weirdness that is WebSockets ;) ...or perhaps I misunderstand what "demonstrate Twisted's native capabilities" actually means in this context. [1] I don't think e.g. a "threaded" / blocking backend would work well in the current implementation, but adding an AnyIO or curio one should be (relatively) straightforward. -- meejah
![](https://secure.gravatar.com/avatar/7cedcd5fe799f836e4ea4f9560e873d0.jpg?s=120&d=mm&r=g)
Am 18.06.24 um 18:36 schrieb meejah:
how is "native" defined? as in, part of and installed as Twisted package? why is that desirable? why wouldn't I care about "other stuff" that is _also_ install apart from Twisted? like eg a Posix kernel, or a Python run-time?
fwiw, Autobahn was built "sans-IO" pretty much from the beginning, simply as a necessity of targeting _both_ Twisted and asyncio. but Autobahn _is_ not zero-sans, but _contains_ a zero-sans websocket implementation this can be seen from the absence of both Twisted and asyncio imports, only txaio imports here https://github.com/crossbario/autobahn-python/blob/f38f16ba28fa253dee951068c... the reason for txaio is: Autobahn not _only_ provides a zero-sans WebSocket implementation, but also respective network layer bindings (to Twisted and asyncio), and to aggregate/factorize that code, there is txaio Autobahn is also a complete (eg it has facilities for WebSocket standard extensions, including deflate) and compliant WebSocket implementation, both extensively tested using Autobahn Testsuite, which was the first and still is the most complete WebSocket implementation testsuite Autobahn also is proven real-world, and using both Twisted and asyncio, and over various transports, including stuff like Unix domain sockets or stdio or serial .. Finally, Autobahn _also_ includes not only WebSocket, but WAMP as well - integrated with WebSocket as one transport ..
Yes, exactly, a _ton_ of work .. like writing a testsuite in the first place and fixing/nagging upstream bugs .. from low-level (UTF8) to high level ... testing in various real-world settings
I'm 98% sure that PRs to improve any aspects of the Autobahn implementation would be accepted (by that project). Although IIRC it
"improve", yes, sure! "any": that depends. what is wrong with it anyways? rgd WebSocket standard compliance, I would say it is as good as it gets.
wasn't written with exact "Sans-IO" guidance in mind [1], the
I would say we did it without knowing there would be a name for it later on;)
implementation is already 'fairly' separated since it supports both
it is perfectly separated (see above), including proper internal APIs for zero-sans https://github.com/crossbario/autobahn-python/blob/f38f16ba28fa253dee951068c...
I don't see a need to integrate another state machine, can you point me to a bug? otherwise, what's the upside?
same here, what is "native" supposed to mean in this context? code shipped as part of Twisted? and if so, why? why ship the world as part of Twisted?
fwiw, rgd Autobahn, the one transport which is missing and I am interested in is: VirtIO VirtIO would allow zero-copy, not only single-copy - which is already available in Posix via UDS. "AnyIO or curio": these are just wrappers around classic Posix sockets, right? I mean, Autobahn on Linux should and is using epoll already, so what exactly is wrong with that?
![](https://secure.gravatar.com/avatar/7cedcd5fe799f836e4ea4f9560e873d0.jpg?s=120&d=mm&r=g)
to make that a bit more concrete, how could we implement an Autobahn transport for WebSocket - that is an implementation of autobahn.websocket.interfaces.IWebSocketChannel running over VirtIO and using virtqueues, and using epoll for event integration. does Python have proper wrappers for Linux VirtIO / virtqueues already? integrated with epoll in Python? implemented in CPython and in PyPy? if so, let's do it;) I could have my Rust code talk to Python with zero-copy (at least on the Rust side) - and using WebSocket as proper/standard message framing the one aspect I am not sure about: WebSocket provides framing .. as it runs over a byte level transport .. and for WAMP, we wouldn't need that as VirtIO already is message based. if we would go down that road, it'll means providing an implementation of autobahn.wamp.interfaces.ITransport https://github.com/crossbario/autobahn-python/blob/f38f16ba28fa253dee951068c... for VirtIO / virtqueues. anyways, I don't really think it matters much performance wise. I can shift out bytes into memory up to maximum main memory bandwidth limited by hardware in Rust easily, and "almost" in PyPy, but still ..
participants (5)
-
Adi Roiban
-
Glyph
-
Kyle Altendorf
-
meejah
-
Tobias Oberstein