(tiny) twisted success story
Hello, world! I'm very happy to be using Twisted again -- after a break of about 20 years! In ol' days I did mailing lists in Twisted, implemented Dovecot AUTH protocol in Twisted and a lot more. Mostly it was mail related. A lot have been forgotten and have to be re-discovered. A couple of words about use-case: I was curious about what could be achieved using the filtering facility in the OpenSMTPD MTA. The filters are programs which are executed by the daemon, and some data is exchanged on stdin/stdout using non-blocking fashion. More at https://man.openbsd.org/smtpd-filters I tried getting Python asyncio to work at first, even using aioconsole package, but I kept hitting something which made the thing break after handshake exchange. This evening I did a rewrite in Twisted, it took me a couple of hours, but the was running on the first try! Thanks for all the fish! -- Kirill
On Dec 13, 2024, at 4:51 PM, Kirill Miazine <km@krot.org> wrote:
Hello, world!
Hi Kirill!
I'm very happy to be using Twisted again -- after a break of about 20 years! In ol' days I did mailing lists in Twisted, implemented Dovecot AUTH protocol in Twisted and a lot more. Mostly it was mail related. A lot have been forgotten and have to be re-discovered.
Thank you so much for taking the time to write this up!
A couple of words about use-case: I was curious about what could be achieved using the filtering facility in the OpenSMTPD MTA. The filters are programs which are executed by the daemon, and some data is exchanged on stdin/stdout using non-blocking fashion. More at https://man.openbsd.org/smtpd-filters
I tried getting Python asyncio to work at first, even using aioconsole package, but I kept hitting something which made the thing break after handshake exchange. This evening I did a rewrite in Twisted, it took me a couple of hours, but the was running on the first try!
Thanks for all the fish!
Nice. Spending so much time, as we do, on bug reports and feature requests, it is a rare treat to just read an uncomplicated description of things working. Even reading this, I am immediately thinking of all the stuff I need to fix in StandardIO. But for the moment I will try not to. Thanks again! -g
• Glyph [2024-12-14 02:01]:
On Dec 13, 2024, at 4:51 PM, Kirill Miazine <km@krot.org> wrote:
Hello, world!
Hi Kirill!
I'm very happy to be using Twisted again -- after a break of about 20 years! In ol' days I did mailing lists in Twisted, implemented Dovecot AUTH protocol in Twisted and a lot more. Mostly it was mail related. A lot have been forgotten and have to be re-discovered.
Thank you so much for taking the time to write this up!
It makes me very happy to pick up and old tool and see that it's still at least as valuable now as it was 20 years ago, and that it survived the python3 transition. Thanks for all the work on it!
A couple of words about use-case: I was curious about what could be achieved using the filtering facility in the OpenSMTPD MTA. The filters are programs which are executed by the daemon, and some data is exchanged on stdin/stdout using non-blocking fashion. More at https://man.openbsd.org/smtpd-filters
I tried getting Python asyncio to work at first, even using aioconsole package, but I kept hitting something which made the thing break after handshake exchange. This evening I did a rewrite in Twisted, it took me a couple of hours, but the was running on the first try!
Thanks for all the fish!
Nice. Spending so much time, as we do, on bug reports and feature requests, it is a rare treat to just read an uncomplicated description of things working. Even reading this, I am immediately thinking of all the stuff I need to fix in StandardIO. But for the moment I will try not to.
Thanks again!
In November last year I challenged myself with a task to make a service monitoring solution to replace NodePing for me, and I was considering asyncio vs Twisted. In the end, I went with the asyncio route and managed to get away using just standard Python libs (to run checks, and to send alerts, I use external commands such as ping/ping6, drill, ldns-dane, nc, curl -- especially the latter is able do do *a lot*). Now I'm tempted to make it work under Twisted, too. The checks themselves are not themselves dependent on asyncio, so I'd have to orchestrating facilities, including some queue.
• km@krot.org [2024-12-14 18:18]: [...]
In November last year I challenged myself with a task to make a service monitoring solution to replace NodePing for me, and I was considering asyncio vs Twisted. In the end, I went with the asyncio route and managed to get away using just standard Python libs (to run checks, and to send alerts, I use external commands such as ping/ping6, drill, ldns-dane, nc, curl -- especially the latter is able do do *a lot*). Now I'm tempted to make it work under Twisted, too. The checks themselves are not themselves dependent on asyncio, so I'd have to orchestrating facilities, including some queue.
so now I've got a prototype, a simple test run with 100K checks on my 3 y.o. thinkpad gives me 1425 check/second with twisted and 1130 check/second with asyncio and standard loop and something very horrible with asyncio with uvloop. the checks were simple calls to /usr/bin/true, which always succeeds. that's more that enough for my needs (I'm doing about 2-3 check/second), so I'll see if I can complete the twisted version of the mointor and have it run along the current one. in any case, I have a possible other usage in sight: custom auth <-> ldap proxy via ldaptor (to see if it would work with stalwart mail server). this is fun!
On Dec 16, 2024, at 5:04 PM, Kirill Miazine <km@krot.org> wrote:
• km@krot.org [2024-12-14 18:18]: [...]
In November last year I challenged myself with a task to make a service monitoring solution to replace NodePing for me, and I was considering asyncio vs Twisted. In the end, I went with the asyncio route and managed to get away using just standard Python libs (to run checks, and to send alerts, I use external commands such as ping/ping6, drill, ldns-dane, nc, curl -- especially the latter is able do do *a lot*). Now I'm tempted to make it work under Twisted, too. The checks themselves are not themselves dependent on asyncio, so I'd have to orchestrating facilities, including some queue.
so now I've got a prototype, a simple test run with 100K checks on my 3 y.o. thinkpad gives me 1425 check/second with twisted and 1130 check/second with asyncio and standard loop and something very horrible with asyncio with uvloop. the checks were simple calls to /usr/bin/true, which always succeeds.
that's more that enough for my needs (I'm doing about 2-3 check/second), so I'll see if I can complete the twisted version of the mointor and have it run along the current one.
in any case, I have a possible other usage in sight: custom auth <-> ldap proxy via ldaptor (to see if it would work with stalwart mail server).
this is fun!
This is a very cool anecdote and I really appreciate you sharing it :) Let us know how it goes! -g
• Glyph [2025-01-06 23:24]:
On Dec 16, 2024, at 5:04 PM, Kirill Miazine <km@krot.org> wrote:
• km@krot.org [2024-12-14 18:18]: [...]
In November last year I challenged myself with a task to make a service monitoring solution to replace NodePing for me, and I was considering asyncio vs Twisted. In the end, I went with the asyncio route and managed to get away using just standard Python libs (to run checks, and to send alerts, I use external commands such as ping/ping6, drill, ldns-dane, nc, curl -- especially the latter is able do do *a lot*). Now I'm tempted to make it work under Twisted, too. The checks themselves are not themselves dependent on asyncio, so I'd have to orchestrating facilities, including some queue.
so now I've got a prototype, a simple test run with 100K checks on my 3 y.o. thinkpad gives me 1425 check/second with twisted and 1130 check/second with asyncio and standard loop and something very horrible with asyncio with uvloop. the checks were simple calls to /usr/bin/true, which always succeeds.
that's more that enough for my needs (I'm doing about 2-3 check/second), so I'll see if I can complete the twisted version of the mointor and have it run along the current one.
in any case, I have a possible other usage in sight: custom auth <-> ldap proxy via ldaptor (to see if it would work with stalwart mail server).
this is fun!
This is a very cool anecdote and I really appreciate you sharing it :)
Let us know how it goes!
Well, I didn't proceed with the ldaptor _yet_, but I successfully migrated my "Pyng" monitoring (details about it in Norwegian https://www.kode24.no/artikkel/ville-spare-penger-erstatta-nodeping-med-1350...) to Twisted, and also finally managed to add a facility which not would do active monitoring (such as e.g. ping etc) but do passive monitoring (push/heartbeat-like, such as https://healthchecks.io/ or https://deadmanssnitch.com/). I had been thinking about how to do the latter in an elegant way without adding some external storage for about a year, and with Twisted I had all the necessary pieces right there. I also added some web stuff with JSON reporting. Another "success" story is migration of uptime.is API: https://get.uptime.is/api (more about it at https://uptime.is/about#api) is also Twisted now. In a way I feel like I'm going backwards in time, but I do appreciate boring tech which just works. Sometimes it works differently to what the docs say, though ;) Greetings Kirill
-g _______________________________________________ Twisted mailing list -- twisted@python.org To unsubscribe send an email to twisted-leave@python.org https://mail.python.org/mailman3/lists/twisted.python.org/ Message archived at https://mail.python.org/archives/list/twisted@python.org/message/LNFM75DGHHZ... Code of Conduct: https://twisted.org/conduct
On Jan 7, 2025, at 12:05 AM, km@krot.org wrote:
• Glyph [2025-01-06 23:24]:
On Dec 16, 2024, at 5:04 PM, Kirill Miazine <km@krot.org> wrote:
• km@krot.org [2024-12-14 18:18]: [...]
In November last year I challenged myself with a task to make a service monitoring solution to replace NodePing for me, and I was considering asyncio vs Twisted. In the end, I went with the asyncio route and managed to get away using just standard Python libs (to run checks, and to send alerts, I use external commands such as ping/ping6, drill, ldns-dane, nc, curl -- especially the latter is able do do *a lot*). Now I'm tempted to make it work under Twisted, too. The checks themselves are not themselves dependent on asyncio, so I'd have to orchestrating facilities, including some queue.
so now I've got a prototype, a simple test run with 100K checks on my 3 y.o. thinkpad gives me 1425 check/second with twisted and 1130 check/second with asyncio and standard loop and something very horrible with asyncio with uvloop. the checks were simple calls to /usr/bin/true, which always succeeds.
that's more that enough for my needs (I'm doing about 2-3 check/second), so I'll see if I can complete the twisted version of the mointor and have it run along the current one.
in any case, I have a possible other usage in sight: custom auth <-> ldap proxy via ldaptor (to see if it would work with stalwart mail server).
this is fun! This is a very cool anecdote and I really appreciate you sharing it :) Let us know how it goes!
Well, I didn't proceed with the ldaptor _yet_, but I successfully migrated my "Pyng" monitoring (details about it in Norwegian https://www.kode24.no/artikkel/ville-spare-penger-erstatta-nodeping-med-1350...) to Twisted, and also finally managed to add a facility which not would do active monitoring (such as e.g. ping etc) but do passive monitoring (push/heartbeat-like, such as https://healthchecks.io/ or https://deadmanssnitch.com/). I had been thinking about how to do the latter in an elegant way without adding some external storage for about a year, and with Twisted I had all the necessary pieces right there. I also added some web stuff with JSON reporting.
Another "success" story is migration of uptime.is API: https://get.uptime.is/api (more about it at https://uptime.is/about#api) is also Twisted now.
Thanks for these updates! This is really cool.
In a way I feel like I'm going backwards in time, but I do appreciate boring tech which just works. Sometimes it works differently to what the docs say, though ;)
We'll get that fixed eventually :). Maybe… very eventually. But eventually.
Greetings Kirill
-g _______________________________________________ Twisted mailing list -- twisted@python.org To unsubscribe send an email to twisted-leave@python.org https://mail.python.org/mailman3/lists/twisted.python.org/ Message archived at https://mail.python.org/archives/list/twisted@python.org/message/LNFM75DGHHZ... Code of Conduct: https://twisted.org/conduct
_______________________________________________ Twisted mailing list -- twisted@python.org To unsubscribe send an email to twisted-leave@python.org https://mail.python.org/mailman3/lists/twisted.python.org/ Message archived at https://mail.python.org/archives/list/twisted@python.org/message/J5VERCKGSBN... Code of Conduct: https://twisted.org/conduct
participants (3)
-
Glyph -
Kirill Miazine -
km＠krot.org