[Twisted-Python] upcoming changes to twistedmatrix.com mail infrastructure
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
Over the last few months, twistedmatrix.com's mailman installation has been used increasingly frequently to execute denial-of-service attacks against people's mailboxes. This is accomplished by sending huge numbers of subscription requests to our website, which in turn sends huge numbers of confirmation emails to their inbox. Based on some information that some targeted users have sent me, I now believe that this is to cause those users' mail quotas to be exceeded so that password reset or login notification emails won't reach them. This has been going on for some time, but the frequency and severity of the attacks seem to be increasing; I only recently realized that this was considerably worse than an annoyance for those affected. I now have at least 1 confirmed report of this attack being a part of a (partially successful) identity theft. This isn't the only problem we have with email: We're running our own infrastructure which puts load on our already beyond-overloaded volunteer system administration team. Despite running our own infrastructure, we are not dogfooding Twisted at all in the process, so we're not even learning anything useful from the pain; "exim is bad" is a lesson we've already learned many times, we do not need to keep learning it. Given how hard it is for us to upgrade Mailman in our current system, we aren't even dogfooding our fellow community project terribly well. Our infrastructure runs on the same host as the website and the buildmaster, overloading a very creaky system. In addition to mailing lists, we run a mail forwarder. Our server's sender reputation is ... not great. We don't have SPF records, we don't do DKIM, and we don't provide authenticated SMTP for users, so emails just come from "wherever" when they are sent from, e.g. 'glyph@twistedmatrix.com' :-). In order to address this, as soon as I can reasonably manage to do so, I will be moving Twisted's email infrastructure to mailgun.com <http://mailgun.com/>, a product that I've been successfully using for a range of personal domains (in particular, the divmod.com <http://divmod.com/> email forwarder - yes, I still operate that, when the Twisted community promises you an email address for life you get it ;-)). Additionally, Mailgun uses a bunch of Twisted within their infrastructure, so (although we won't be operating it) we will actually be dogfooding considerably more. (Mailgun is a product of my employer, Rackspace, but they've given us a generous open source discount so there's no conflict of interest; the Twisted project won't be spending money on this.) There will be a couple of inconveniences immediately after the transition: At first, there will be no self-service subscription to mailing lists any more. If you want to subscribe, you'll have to send a message to twisted-python-owner@twistedmatrix.com <mailto:twisted-python-owner@twistedmatrix.com> and the list administrator (right now, probably just me) will manually add your address. (Self-service unsubscription will still be possible.) I'm not sure if I'll be able to keep the list archives at https://twistedmatrix.com/pipermail/ <https://twistedmatrix.com/pipermail/> updated, at least at first. I would encourage everyone to use http://news.gmane.org/gmane.comp.python.twisted <http://news.gmane.org/gmane.comp.python.twisted> and http://news.gmane.org/gmane.comp.python.twisted.web <http://news.gmane.org/gmane.comp.python.twisted.web> in the meanwhile. Speaking of the contents of that sad URL, many disused mailing lists will be deleted. I doubt anyone will notice since there haven't been any posts to most of them in many years. If you presently send email from a twistedmatrix.com <http://twistedmatrix.com/> address, you will probably want to start using the mailgun forwarder so that your messages will have nice shiny DKIM/SPF headers; I suspect you may start having more deliverability problems than you already do once other mail servers notice that we have said records if you're not using them. I'll distribute SMTP credentials via GPG-encrypted email to everyone I'm aware of who uses such an address. There will be considerable benefits though: For those of you with @twistedmatrix.com <http://twistedmatrix.com/> addresses, Mailgun operates a pretty conservative low-pass spam filter, but in looking at the analytics from my own personal domains, it really helps a lot and it is definitely more effective than the setup we've got right now. Deliverability and mail-sending performance should be much improved; messages should arrive faster because they will be quarantined or deferred-bounced by major senders like GMail et. al. far less often, because we'll be forwarding less spam and legitimate messages will have appropriate anti-spam headers. Trac will get faster at certain times because email DoSes should stop hitting the server. Administrative overhead will decrease; we can just stop maintaining email ourselves. Last but certainly not least, we'll stop being a collective unwilling accessory to cybercrime. Probably these changes will all be pretty subtle, and most folks won't notice, but I wanted it to be clear in advance that they were intentional, in case there is some disruption associated with them :-). If anyone wants to give me a hand with parts of this (for example, setting up a smarthost configuration so that trac can still send email) please let me know. -glyph
![](https://secure.gravatar.com/avatar/426d6dbf6554a9b3fca1fd04e6b75f38.jpg?s=120&d=mm&r=g)
On 16/03/16 18:52, Glyph wrote:
Over the last few months, twistedmatrix.com <http://twistedmatrix.com>'s mailman installation has been used increasingly frequently to execute denial-of-service attacks against people's mailboxes. This is
My sympathies; this exact problem was the reason we CAPTCHA-ised our install of mailman and have to keep a very close eye on it. It's really a shame there's so little open-source competition in the email sector these days; it all appears to have been hoovered up by Gmail, Office 365 and various spam (sorry - bulk email) providers.
There will be a couple of inconveniences immediately after the transition:
Couple of random thoughts: Does mailgun actually contain a mailman-alike product or are you effectively building one on top of it? Will the mailman-style List-X headers remain? Will the behaviour of the list w.r.t. things like routing of To:/Cc:'ed people change. Good luck with the migration.
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
On Mar 16, 2016, at 12:06 PM, Phil Mayers <p.mayers@imperial.ac.uk> wrote:
On 16/03/16 18:52, Glyph wrote:
Over the last few months, twistedmatrix.com <http://twistedmatrix.com>'s mailman installation has been used increasingly frequently to execute denial-of-service attacks against people's mailboxes. This is
My sympathies; this exact problem was the reason we CAPTCHA-ised our install of mailman and have to keep a very close eye on it.
Yeah. If this were the only problem we'd probably be going that route, but given issues with the rest of our mail infrastructure, getting rid of it is a lot more satisfying :). When I do self-service subscription I do very definitely plan to integrate a CAPTCHA.
It's really a shame there's so little open-source competition in the email sector these days; it all appears to have been hoovered up by Gmail, Office 365 and various spam (sorry - bulk email) providers.
There will be a couple of inconveniences immediately after the transition:
Couple of random thoughts:
Does mailgun actually contain a mailman-alike product or are you effectively building one on top of it?
Mailgun does have mailing lists: https://documentation.mailgun.com/api-mailinglists.html This is not really a "mailman-alike"; its feature-set is extremely minimal (and, as mailgun will readily tell you, using it for members-only mailing lists is a bit of a weird case for their product; their primary target is transactional application emails, like notifications of activity in a web app, invoices, alerts, that sort of thing). There are some things we will miss (particularly archives; I'm hoping we can just pipe the messages into pipermail somehow); but huge amounts of Mailman's customizability are just useless fluff. Some are actively bad, like mailing you all your passwords in plain text every month. We don't use most of its features, and we have to explicitly disable a lot of them. Many of these things are better in more recent releases, but for us, upgrading to a more recent release is quite a bit more work than abandoning it entirely. However, despite peer-to-peer lists being a little outside Mailgun's core demographic, they're totally supported, and I've had a pretty good experience (better than mailman administration, certainly) administering a medium-sized mailing list using their web UI. I do plan to build a few small tools, like a self-service subscription tool, using the API, but even that will be good; it'll make a nice little demo Klein app.
Will the mailman-style List-X headers remain?
Yes, although for unfortunate technical reasons the values of those headers may change (the way lists vs. personal addresses are name-spaced on twistedmatrix.com is unfortunate for reasons having nothing to do with mailgun, but it will probably matter now whereas it didn't before).
Will the behaviour of the list w.r.t. things like routing of To:/Cc:'ed people change.
For members-post mailing lists, mailgun unconditionally sets the reply-to header, which is exactly the way we have mailman configured right now, so: no.
Good luck with the migration.
Thanks! And thanks for your questions, I was worried I put a ton of work into that email only for it to land in the void :). -glyph
![](https://secure.gravatar.com/avatar/182974f8b2562287a54415119be4535c.jpg?s=120&d=mm&r=g)
On 03/16/2016 03:53 PM, Glyph wrote:
On Mar 16, 2016, at 12:06 PM, Phil Mayers <p.mayers@imperial.ac.uk <mailto:p.mayers@imperial.ac.uk>> wrote:
Good luck with the migration.
Thanks! And thanks for your questions, I was worried I put a ton of work into that email only for it to land in the void :).
FWIW, Glyph, I read *all* your emails from beginning to end -- both because they are always educational (for me) and because it helps me in my never-ending quest to avoid doing real work. :D (So, another part of the void heard from ... ;) But seriously: thanks for all your work on this and Twisted in general, and to you and all the Twisted minions for the continuing and sustained high quality of Twisted! Cheers, Steve
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
On Mar 16, 2016, at 11:52 AM, Glyph <glyph@twistedmatrix.com> wrote:
Over the last few months, twistedmatrix.com <http://twistedmatrix.com/>'s mailman installation has been used increasingly frequently to execute denial-of-service attacks against people's mailboxes. This is accomplished by sending huge numbers of subscription requests to our website, which in turn sends huge numbers of confirmation emails to their inbox. Based on some information that some targeted users have sent me, I now believe that this is to cause those users' mail quotas to be exceeded so that password reset or login notification emails won't reach them.
I've taken the first few steps to migrating us over to Mailgun, but just as a minor status update: web-based subscription is now disabled. If any generous souls would like to help out and update whatever wiki pages link to the mailman listinfo pages to say 'please send subscription requests to twisted-python-owner@ or twisted-web-owner@ instead' that would be great :-). Those email addresses (and hopefully all the others, too) will keep working post-migration. -glyph
![](https://secure.gravatar.com/avatar/9d7e611f31c52f4d62bbe279d4f334de.jpg?s=120&d=mm&r=g)
I prefer web-based access more than a list, so a thing like http://try.discourse.org/ is more appropriate for me in terms of usability. Especially those shiny "with" login buttons. On Fri, Apr 1, 2016 at 5:04 AM, Glyph <glyph@twistedmatrix.com> wrote:
On Mar 16, 2016, at 11:52 AM, Glyph <glyph@twistedmatrix.com> wrote:
Over the last few months, twistedmatrix.com's mailman installation has been used increasingly frequently to execute denial-of-service attacks against people's mailboxes. This is accomplished by sending huge numbers of subscription requests to our website, which in turn sends huge numbers of confirmation emails to their inbox. Based on some information that some targeted users have sent me, I now believe that this is to cause those users' mail quotas to be exceeded so that password reset or login notification emails won't reach them.
I've taken the first few steps to migrating us over to Mailgun, but just as a minor status update: web-based subscription is now disabled. If any generous souls would like to help out and update whatever wiki pages link to the mailman listinfo pages to say 'please send subscription requests to twisted-python-owner@ or twisted-web-owner@ instead' that would be great :-). Those email addresses (and hopefully all the others, too) will keep working post-migration.
-glyph
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
-- anatoly t.
participants (4)
-
anatoly techtonik
-
Glyph
-
Phil Mayers
-
Steve Waterbury