From gtaylor at tnetconsulting.net Mon Apr 2 15:14:16 2018 From: gtaylor at tnetconsulting.net (Grant Taylor) Date: Mon, 2 Apr 2018 13:14:16 -0600 Subject: [Mailman-Users] (relatively) new DMARC issues - and Gmail In-Reply-To: <1522521095.68575.108.camel@fmp.com> References: <1522521095.68575.108.camel@fmp.com> Message-ID: Have you considered sending your message to the Mailop mailing list? I know that there are a couple of Gmail admins / coworkers that are subscribed to Mailop and will respond to issues like this. Plus, it might also be a better forum and get more engagement / suggestions / gratitude by others learning from your toils. On 03/31/2018 12:31 PM, Lindsay Haisley wrote: > At some point Amazon (amazon.com) started publishing a DMARC > "p=quarantine" policy, which means that any email which gets redirected > and hits my dmarc_shield piece is going to have its From address re- > written to "postmaster at fmp.com" (fmp.com has a proper SPF record). I'm sure that Amazon is just one of /many/ companies that are working with DMARC. - Seeing as how some ~> more governments are (going to be) requiring DMARC, I expect that we will see more of this. > I don't know what Gmail's policy is with regard to "p=quarantine" > - whether it rejects such email outright or relegates it to the > recipient's spam folder. I know that if the sending site publishes > "p=reject", redirected email is refused by Gmail at the front door. > I'll have to test the "p=quarantine" behavior. I'm confident that Mailop subscribers can respond to this. > Here's the really annoying thing. My dmarc_shield processor rewrites the > From header as per SOP for Mailman with the proper switch turned on. The > From header address becomes "postmaster at fmp.com" with the original From > address in the address comment (from xxx at yyz.com). If the email didn't > already have a Reply-To address, the original From address is inserted > as the Reply-To address. If a Gmail user replies to such an email, the > reply goes to the Reply-To address, but Gmail **whitelists** the From > address! Thereafter, any email which comes in with a munged From address > is accepted, bypassing Gmail's otherwise pretty good spam filtering. I'm > noticing a lot of spam email going out with From addresses for which > a DMARC "p=reject" policy is published, which means that any such spam > redirected to the Gmail user via FMP is also whitelisted. Bah! It's a > fucking war zone out there! I'm confident that Mailop subscribers can respond to this too. Probably including reasons as to why something is done. I speculate that it's to prevent abuse of meaningless addresses being used in the From: address and causing replies to go somewhere other than back to the (purported) sender. > The only possible solution here would be to randomize the username portion > of the rewritten From address, which makes the email look more like spam, > and the Gmail user would end up with a whole lot of useless whitelisted > address which would need to be deleted. Not to mention the fact that > FMP's mail server might be blocked from sending ANY email to Gmail. I initially thought about something like an MD5 hash of the (purported) From address. Though that still suffers from the multiple addresses being white listed. Despite that, I'd consider forwarding from a "forwarding" (sub)domain. Something to hopefully help articulate to the human looking at the complaints that the message is forwarded. Plus this I would expect this to help differentiate email reputation for fmp.com from the (sub)domain used for forwarding. (I don't know if a sub-domain would suffice or if it should be a different parallel / sibling domain, fmp-forwarding.com.) -- Grant. . . . unix || die From mark at msapiro.net Mon Apr 2 19:47:04 2018 From: mark at msapiro.net (Mark Sapiro) Date: Mon, 2 Apr 2018 16:47:04 -0700 Subject: [Mailman-Users] (relatively) new DMARC issues - and Gmail In-Reply-To: <1522521095.68575.108.camel@fmp.com> References: <1522521095.68575.108.camel@fmp.com> Message-ID: On 03/31/2018 11:31 AM, Lindsay Haisley wrote: > > At some point Amazon (amazon.com) started publishing a DMARC > "p=quarantine" policy, which means that any email which gets redirected > and hits my dmarc_shield piece is going to have its From address re- > written to "postmaster at fmp.com" (fmp.com has a proper SPF record). Why do you feel this is necessary? I suppose it is possible that amazon publishises a DMARC policy and does NOT DKIM sign it's outgoing email but relies solely on SPF domain alignment to pass DMARC, but I think this would be a rare exception. If the mail from Amazon is DKIM signed with an aligned domain and you make no transformations that would break that sig, i.e. you are a simple .forward or alias type forwarder, the DKIM sig will still validate at the receiver and you don't need to munge the From: > Here's the really annoying thing. My dmarc_shield processor rewrites > the From header as per SOP for Mailman with the proper switch turned > on. The From header address becomes "postmaster at fmp.com" with the > original From address in the address comment (from xxx at yyz.com). If > the email didn't already have a Reply-To address, the original From > address is inserted as the Reply-To address. If a Gmail user replies to > such an email, the reply goes to the Reply-To address, but Gmail > **whitelists** the From address! Thereafter, any email which comes in > with a munged From address is accepted, bypassing Gmail's otherwise > pretty good spam filtering. I'm noticing a lot of spam email going out > with From addresses for which a DMARC "p=reject" policy is published, > which means that any such spam redirected to the Gmail user via FMP is > also whitelisted. Bah! It's a fucking war zone out there! The first question is why would the ultimate gmail recipient reply to the spam in the first place. The next question is assuming it is spam, does it originate from an amazon server. If not, it should fail DMARC when you receive it and you should consider honoring the amazon DMARC police and not forward the mail. And if it does originate from an amazon server with a valid DKIM sig are you making transformations that invalidate the DKIM sig? Again, if not, if you are a simple forwarder, you shouldn't need to mung the From:. I understand part of the intent is a heads-up to people like me, and in my case, I am not a simple forwarder, but I'm hopeful that eventually at least, ARC can help, but I still don't understand why this is an issue for you. It seems your case is simple. If it fails DMARC when it reaches you, honor the p=quarantine and don't forward the mail. If it passes DMARC based on DKIM, forward it without munging the From:. That leaves only the case where it passes DMARC solely on SPF, and my guess is that this is an empty or almost empty set. -- Mark Sapiro The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan From kontakt at michaelbakonyi.de Sun Apr 8 08:08:49 2018 From: kontakt at michaelbakonyi.de (Michael Bakonyi) Date: Sun, 8 Apr 2018 14:08:49 +0200 Subject: [Mailman-Users] Blank emails due to convert_html_to_plaintext Message-ID: Dear Mailman-enthusiasts, I administer a list, where emails by some senders (e.g. the german mass-mailprovider GMX) arrive with a stripped message (but existing Mailman-footer) due to convert_html_to_plaintext enabled. Following you find other settings: filter_content: yes pass_mime_types: multipart/mixed multipart/alternative text/rtf text/plain text/comma-separated-values text/tab-separated-values image text/html message/rfc822 I don't see the option "collapse multipart/alternative" anywhere (which is mentioned in other threads regarding empty mails) so I guess it's an older version of Mailman. I don't have access to any configuration-files as it's a Mailman-instance of a shared hoster. Here's a test-email where the message would be stripped (X-Provags-ID and X-UI-Out-Filterresults shortened for better readability). Delivery-date: Sun, 08 Apr 2018 13:56:12 +0200 Received: from [80.67.18.6] (helo=mx06.ispgateway.de) by mailcluster2-1.ispgateway.de with esmtp (Exim 4.90_1) (envelope-from ) id 1f58vc-0007ME-Ov; Sun, 08 Apr 2018 13:56:12 +0200 Return-path: X-Envelope-to: bla at blub.de Received: from [212.227.17.20] (helo=mout.gmx.net) by mx06.ispgateway.de with esmtps (TLSv1.2:ECDHE-RSA-AES128-GCM-SHA256:128) (Exim 4.90_1) (envelope-from ) id 1f58vi-00032J-W2 for bla at blub.de; Sun, 08 Apr 2018 13:56:19 +0200 Received: from [91.63.233.19] ([91.63.233.19]) by 3c-app-gmx-bs65.server.lan (via HTTP); Sun, 8 Apr 2018 13:56:12 +0200 MIME-Version: 1.0 Message-ID: From: "Michael Bakonyi" To: bla at blub.de Subject: test Content-Type: text/html; charset=UTF-8 Date: Sun, 8 Apr 2018 13:56:12 +0200 Importance: normal Sensitivity: Normal X-Priority: 3 X-Provags-ID: shortened X-UI-Out-Filterresults: notjunk:1;shortened X-Received-SPF: pass ( mx06.ispgateway.de: domain of gmx.net designates 212.227.17.20 as permitted sender ) X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on spamfilter16.ispgateway.de X-Spam-Level: X-Spam-Status: No, hits=0.0 required=9999.0 tests=BAYES_50 autolearn=disabled version=3.4.0 X-Spam-CMAETAG: v=2.2 cv=TruWvHfh c=1 sm=1 tr=0 a=EDFhQv9wJ3PvMGwfvWgztQ==:17 a=IkcTkHD0fZMA:10 a=x7bEGLp0ZPQA:10 a=z6JOVgwrxLwA:10 a=Kd1tUaAdevIA:10 a=7lxF1IeVJ3SWydATLDQA:9 a=_W_S_7VecoQA:10 a=QEXdDO2ut3YA:10 X-Spam-CMAECATEGORY: X-Spam-CMAESUBCATEGORY: X-Spam-CMAESCORE:
Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet.
If i turn off convert_html_to_plaintext the mails of the user passes. Is there anything I can do other to turning conversion of HTML off? Actually I don't want to allow HTML-mails ... Maybe a charset-related problem? Cheers, Michael -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: Message signed with OpenPGP URL: From mark at msapiro.net Mon Apr 9 09:21:03 2018 From: mark at msapiro.net (Mark Sapiro) Date: Mon, 09 Apr 2018 06:21:03 -0700 Subject: [Mailman-Users] Blank emails due to convert_html_to_plaintext In-Reply-To: References: Message-ID: <11BEC330-83E3-4875-A838-A922B43DBD74@msapiro.net> On April 8, 2018 5:08:49 AM PDT, Michael Bakonyi wrote: > >I administer a list, where emails by some senders (e.g. the german >mass-mailprovider GMX) arrive with a stripped message (but existing >Mailman-footer) due to convert_html_to_plaintext enabled. ... >I don't see the option "collapse multipart/alternative" anywhere (which >is mentioned in other threads regarding empty mails) so I guess it's an >older version of Mailman. It should be on the content filtering page just above the "convert_html_to_plaintext" setting. > I don't have access to any >configuration-files as it's a Mailman-instance of a shared hoster. You will have to contact the host. HTML to plaintext conversion is done by the configured HTML_TO_PLAIN_TEXT_COMMAND (default lynx). There will be errors in Mailman's error log. Whatever the configured command is, it isn't working. -- Mark Sapiro Sent from my Not_an_iThing with standards compliant, open source software. From kontakt at michaelbakonyi.de Tue Apr 10 02:14:24 2018 From: kontakt at michaelbakonyi.de (Michael Bakonyi) Date: Tue, 10 Apr 2018 08:14:24 +0200 Subject: [Mailman-Users] Blank emails due to convert_html_to_plaintext In-Reply-To: <11BEC330-83E3-4875-A838-A922B43DBD74@msapiro.net> References: <11BEC330-83E3-4875-A838-A922B43DBD74@msapiro.net> Message-ID: Hi Mark, thanks a lot for responding so quickly! > It should be on the content filtering page just above the "convert_html_to_plaintext" setting. Nope, in my case there's pass_mime_types. >> I don't have access to any >> configuration-files as it's a Mailman-instance of a shared hoster. > > You will have to contact the host. HTML to plaintext conversion is done by the configured HTML_TO_PLAIN_TEXT_COMMAND (default lynx). There will be errors in Mailman's error log. Whatever the configured command is, it isn't working. Well the conversion works in general ? only with some mails of some senders there seems to be a problem. Alright, I'll contact the hoster, thanks for your feedback! Cheers, Michael -- Michael Bakonyi c/o Anke Seyfried Endemannstr. 8 69115 Heidelberg Tel: 0170 40 232 62 > Am 09.04.2018/ 15 um 15:21 schrieb Mark Sapiro : > > On April 8, 2018 5:08:49 AM PDT, Michael Bakonyi wrote: >> >> I administer a list, where emails by some senders (e.g. the german >> mass-mailprovider GMX) arrive with a stripped message (but existing >> Mailman-footer) due to convert_html_to_plaintext enabled. > ... >> I don't see the option "collapse multipart/alternative" anywhere (which >> is mentioned in other threads regarding empty mails) so I guess it's an >> older version of Mailman. > > > It should be on the content filtering page just above the "convert_html_to_plaintext" setting. > > >> I don't have access to any >> configuration-files as it's a Mailman-instance of a shared hoster. > > > You will have to contact the host. HTML to plaintext conversion is done by the configured HTML_TO_PLAIN_TEXT_COMMAND (default lynx). There will be errors in Mailman's error log. Whatever the configured command is, it isn't working. > > > > > -- > Mark Sapiro > Sent from my Not_an_iThing with standards compliant, open source software. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: Message signed with OpenPGP URL: From mark at msapiro.net Tue Apr 10 12:35:34 2018 From: mark at msapiro.net (Mark Sapiro) Date: Tue, 10 Apr 2018 09:35:34 -0700 Subject: [Mailman-Users] Blank emails due to convert_html_to_plaintext In-Reply-To: References: <11BEC330-83E3-4875-A838-A922B43DBD74@msapiro.net> Message-ID: <213ddbbc-6525-5000-1776-fb7eebc12779@msapiro.net> On 04/09/2018 11:14 PM, Michael Bakonyi wrote: > Hi Mark, > > thanks a lot for responding so quickly! > >> It should be on the content filtering page just above the "convert_html_to_plaintext" setting. > > Nope, in my case there's pass_mime_types. If you don't have filter_filename_extensions and pass_filename_extensions between pass_mime_types and convert_html_to_plaintext then this is either some modified Mailman or older than 2.1.6 (May, 2005). Since you don't have collapse_alternatives which has been in 2.1 from the beginning, I suspect mods particularly because of what I say below. > Well the conversion works in general ? only with some mails of some senders there seems to be a problem. Alright, I'll contact the hoster, thanks for your feedback! I suspect the conversion never works. I suspect that when you think it works, it is because the original message is multipart/alternative and you are seeing the text/plain alternative from the original, not converted HTML. Either collapse_alternatives is being done in spite of your not seeing it as an option, or it is not being done and the result is multipart/alternative with the original text/plain alternative followed by an empty text/plain alternative converted from the original text/html alternative. The message you posted in your OP was just a text/html message, and I suspect these are the ones that wind up blank. -- Mark Sapiro The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 181 bytes Desc: OpenPGP digital signature URL: From kontakt at michaelbakonyi.de Tue Apr 10 13:32:12 2018 From: kontakt at michaelbakonyi.de (Michael Bakonyi) Date: Tue, 10 Apr 2018 19:32:12 +0200 Subject: [Mailman-Users] Blank emails due to convert_html_to_plaintext In-Reply-To: <213ddbbc-6525-5000-1776-fb7eebc12779@msapiro.net> References: <11BEC330-83E3-4875-A838-A922B43DBD74@msapiro.net> <213ddbbc-6525-5000-1776-fb7eebc12779@msapiro.net> Message-ID: Hi Mark, >> Well the conversion works in general ? only with some mails of some senders there seems to be a problem. Alright, I'll contact the hoster, thanks for your feedback! > > > I suspect the conversion never works. I suspect that when you think it > works, it is because the original message is multipart/alternative and > you are seeing the text/plain alternative from the original, not > converted HTML. Aaah you are right! I've just sent a text/html-only mail via Swiftmailer-script and the mail which arrives is empty aswell! I never would have thought about this as the hoster is pretty big and of course I assumed the conversion to be working. I'll contact them and hopefully they will be able to fix it soon. Thanks a lot for your help! Cheers, Michael -- Michael Bakonyi c/o Anke Seyfried Endemannstr. 8 69115 Heidelberg Tel: 0170 40 232 62 > Am 10.04.2018/ 15 um 18:35 schrieb Mark Sapiro : > > On 04/09/2018 11:14 PM, Michael Bakonyi wrote: >> Hi Mark, >> >> thanks a lot for responding so quickly! >> >>> It should be on the content filtering page just above the "convert_html_to_plaintext" setting. >> >> Nope, in my case there's pass_mime_types. > > > If you don't have filter_filename_extensions and > pass_filename_extensions between pass_mime_types and > convert_html_to_plaintext then this is either some modified Mailman or > older than 2.1.6 (May, 2005). Since you don't have collapse_alternatives > which has been in 2.1 from the beginning, I suspect mods particularly > because of what I say below. > > >> Well the conversion works in general ? only with some mails of some senders there seems to be a problem. Alright, I'll contact the hoster, thanks for your feedback! > > > I suspect the conversion never works. I suspect that when you think it > works, it is because the original message is multipart/alternative and > you are seeing the text/plain alternative from the original, not > converted HTML. Either collapse_alternatives is being done in spite of > your not seeing it as an option, or it is not being done and the result > is multipart/alternative with the original text/plain alternative > followed by an empty text/plain alternative converted from the original > text/html alternative. > > The message you posted in your OP was just a text/html message, and I > suspect these are the ones that wind up blank. > > -- > Mark Sapiro The highway is for gamblers, > San Francisco Bay Area, California better use your sense - B. Dylan > > ------------------------------------------------------ > Mailman-Users mailing list Mailman-Users at python.org > https://mail.python.org/mailman/listinfo/mailman-users > Mailman FAQ: http://wiki.list.org/x/AgA3 > Security Policy: http://wiki.list.org/x/QIA9 > Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ > Unsubscribe: https://mail.python.org/mailman/options/mailman-users/kontakt%40michaelbakonyi.de -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: Message signed with OpenPGP URL: From mark at msapiro.net Tue Apr 10 14:03:20 2018 From: mark at msapiro.net (Mark Sapiro) Date: Tue, 10 Apr 2018 11:03:20 -0700 Subject: [Mailman-Users] Blank emails due to convert_html_to_plaintext In-Reply-To: References: Message-ID: <12eab525-9a41-99ca-29d1-5270a35867d1@msapiro.net> Michael Bakonyi wrote: > Aaah you are right! I've just sent a text/html-only mail via Swiftmailer-script and the mail which arrives is empty aswell! I never would have thought about this as the hoster is pretty big and of course I assumed the conversion to be working. I'll contact them and hopefully they will be able to fix it soon. Also, it may not be relevant, but it is possible that the issue is the one described at (see the last paragraph). -- Mark Sapiro The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan From kontakt at michaelbakonyi.de Tue Apr 10 14:07:56 2018 From: kontakt at michaelbakonyi.de (Michael Bakonyi) Date: Tue, 10 Apr 2018 20:07:56 +0200 Subject: [Mailman-Users] Blank emails due to convert_html_to_plaintext In-Reply-To: <12eab525-9a41-99ca-29d1-5270a35867d1@msapiro.net> References: <12eab525-9a41-99ca-29d1-5270a35867d1@msapiro.net> Message-ID: <7A1221CE-2402-4ECE-A4EE-C36179D67DFF@michaelbakonyi.de> > Also, it may not be relevant, but it is possible that the issue is the > one described at (see the last paragraph). I had seen the wiki-entry but not the last paragraph, thx! I forwarded our conversation to my host ? hopefully they will read through it. Cheers, Michael > Am 10.04.2018/ 15 um 20:03 schrieb Mark Sapiro : > > Michael Bakonyi wrote: > >> Aaah you are right! I've just sent a text/html-only mail via Swiftmailer-script and the mail which arrives is empty aswell! I never would have thought about this as the hoster is pretty big and of course I assumed the conversion to be working. I'll contact them and hopefully they will be able to fix it soon. > > > Also, it may not be relevant, but it is possible that the issue is the > one described at (see the last paragraph). > > -- > Mark Sapiro The highway is for gamblers, > San Francisco Bay Area, California better use your sense - B. Dylan > ------------------------------------------------------ > Mailman-Users mailing list Mailman-Users at python.org > https://mail.python.org/mailman/listinfo/mailman-users > Mailman FAQ: http://wiki.list.org/x/AgA3 > Security Policy: http://wiki.list.org/x/QIA9 > Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ > Unsubscribe: https://mail.python.org/mailman/options/mailman-users/kontakt%40michaelbakonyi.de -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: Message signed with OpenPGP URL: From mpn at icabs.co.zw Fri Apr 13 07:25:34 2018 From: mpn at icabs.co.zw (MPN Kazishe) Date: Fri, 13 Apr 2018 13:25:34 +0200 Subject: [Mailman-Users] Bug in Mailman version 2.1.25 Message-ID: <000001d3d31a$3ca2f7c0$b5e8e740$@icabs.co.zw> Dear All, I am running mailman version 2.1.25 on FreeBSD 11.1-RELEASE. I am getting this error after a power outage on one of the lists. The web page give this error here: Bug in Mailman version 2.1.25 We're sorry, we hit a bug! Please inform the webmaster for this site of this problem. Printing of traceback and other system information has been explicitly inhibited, but the webmaster can find this information in the Mailman error logs. >From the shell whilst doing a db check as follows: root at snappers:~ # /usr/local/mailman/bin/check_db -v myproblemlist Traceback (most recent call last): File "/usr/local/mailman/bin/check_db", line 153, in main() File "/usr/local/mailman/bin/check_db", line 121, in main mlist = MailList(listname, lock=0) File "/usr/local/mailman/Mailman/MailList.py", line 131, in __init__ self.Load() File "/usr/local/mailman/Mailman/MailList.py", line 684, in Load raise Errors.MMCorruptListDatabaseError, e Mailman.Errors.MMCorruptListDatabaseError: [Errno 2] No such file or directory: '/usr/local/mailman/lists/myproblemlist/config.db.last' In the error logs I am getting the following: Apr 13 13:04:49 2018 (58769) couldn't load config file /usr/local/mailman/lists/myproblemlist/config.db.last [Errno 2] No such file or directory: '/usr/local/mailman/lists/myproblemlist/config.db.last' Apr 13 13:04:49 2018 (58769) All myproblemlist fallbacks were corrupt, giving up Can someone please let me know, where I am getting it wrong, and what to look at. Many thanks MPN/ From mark at msapiro.net Fri Apr 13 10:45:44 2018 From: mark at msapiro.net (Mark Sapiro) Date: Fri, 13 Apr 2018 07:45:44 -0700 Subject: [Mailman-Users] Bug in Mailman version 2.1.25 In-Reply-To: <000001d3d31a$3ca2f7c0$b5e8e740$@icabs.co.zw> References: <000001d3d31a$3ca2f7c0$b5e8e740$@icabs.co.zw> Message-ID: <46f78857-74c7-c523-7992-0d38e7105a49@msapiro.net> On 04/13/2018 04:25 AM, MPN Kazishe wrote: > > Apr 13 13:04:49 2018 (58769) couldn't load config file > /usr/local/mailman/lists/myproblemlist/config.db.last > > [Errno 2] No such file or directory: > '/usr/local/mailman/lists/myproblemlist/config.db.last' > > Apr 13 13:04:49 2018 (58769) All myproblemlist fallbacks were corrupt, > giving up The /usr/local/mailman/lists/myproblemlist/config.pck file containing the list configuration, membership, etc. is corrupt. When this happens, Mailman tries in order /usr/local/mailman/lists/myproblemlist/config.pck.last /usr/local/mailman/lists/myproblemlist/config.db /usr/local/mailman/lists/myproblemlist/config.db.last the config.pck.last file is the immediately prior version and it too is apparently corrupt. The config.db and config.db.last files are artifacts from Mailman 2.0 and should not exist so the [Errno 2] No such file or directory: '/usr/local/mailman/lists/myproblemlist/config.db.last' is expected and normal in this situation. Hopefully, you have a recent backup from before the outage with a good /usr/local/mailman/lists/myproblemlist/config.pck file that you can restore. -- Mark Sapiro The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan From mpn at icabs.co.zw Sat Apr 14 08:33:55 2018 From: mpn at icabs.co.zw (MPN Kazishe) Date: Sat, 14 Apr 2018 14:33:55 +0200 Subject: [Mailman-Users] Bug in Mailman version 2.1.25 In-Reply-To: <46f78857-74c7-c523-7992-0d38e7105a49@msapiro.net> References: <000001d3d31a$3ca2f7c0$b5e8e740$@icabs.co.zw> <46f78857-74c7-c523-7992-0d38e7105a49@msapiro.net> Message-ID: <001201d3d3ec$df669160$9e33b420$@icabs.co.zw> > -----Original Message----- > From: Mailman-Users [mailto:mailman-users-bounces+mpn=icabs.co.zw at python.org] On Behalf Of Mark Sapiro > Sent: Friday, 13 April 2018 4:46 PM > To: mailman-users at python.org > Subject: Re: [Mailman-Users] Bug in Mailman version 2.1.25 > > Hopefully, you have a recent backup from before the outage with a good /usr/local/mailman/lists/myproblemlist/config.pck file that you can restore. Unfortunately, I have no backups! How can I forcefully remove the list. My only hope is in re-adding it. Many thanks. From odhiambo at gmail.com Sat Apr 14 09:12:44 2018 From: odhiambo at gmail.com (Odhiambo Washington) Date: Sat, 14 Apr 2018 16:12:44 +0300 Subject: [Mailman-Users] Bug in Mailman version 2.1.25 In-Reply-To: <001201d3d3ec$df669160$9e33b420$@icabs.co.zw> References: <000001d3d31a$3ca2f7c0$b5e8e740$@icabs.co.zw> <46f78857-74c7-c523-7992-0d38e7105a49@msapiro.net> <001201d3d3ec$df669160$9e33b420$@icabs.co.zw> Message-ID: On 14 April 2018 at 15:33, MPN Kazishe wrote: > > -----Original Message----- > > From: Mailman-Users > [mailto:mailman-users-bounces+mpn=icabs.co.zw at python.org] On Behalf Of > Mark > Sapiro > > Sent: Friday, 13 April 2018 4:46 PM > > To: mailman-users at python.org > > Subject: Re: [Mailman-Users] Bug in Mailman version 2.1.25 > > > > Hopefully, you have a recent backup from before the outage with a good > /usr/local/mailman/lists/myproblemlist/config.pck file that you can > restore. > > > Unfortunately, I have no backups! How can I forcefully remove the list. My > only hope is in re-adding it. > > Many thanks. > Hi Kazishe, If your list has archives, you can re-use them, I believe. The fact that the config file is corrupt should not lead you to blow away the list and create it afresh. You can use another list as a template to create your corrupted list, I believe. You could lose the members, yes, but they will re-subscribe. Here is what I would do: 1. Export a working list configuration (a) /usr/local/mailman/bin/config_list -o working_listname > /tmp/listname.bak (b) vi /tmp/listname.bak and change the settings to match the name of the corrupted list. This is a text file, with comments explaining what is what :-) 2. Make a backup of the archives of the corrupted list for restore later (a) cd /usr/local/mailman/archives/private tar czf /tmp/corrupted_listname_private.tgz listname listname.mbox 3. Remove the corrupt list, recreate it, import configuration, restore the archives (a) /usr/local/mailman/bin/rmlist -a corrupt_listname # Blow it away (b) /usr/local/mailman/bin/newlist corrupt_listname # Recreate it (c) /usr/local/mailman/bin/config_list -i /tmp/listname.bak # Import the configuration (d) cd /usr/local/mailman/archives/private && tar yxvf /tmp/corrupted_listname_private.tgz # Restore the archives 4. Fix the permissions /usr/local/mailman/bin/check_perms -f Repeat this again just to be sure the permissions are fixed I hope I haven't forgotten a step, but that should give you a working list in a short time. Fix any subsequent errors. ASSUMPTION: Mailman on FreeBSD is installed in /usr/local/mailman/ -- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254 7 3200 0004/+254 7 2274 3223 "Oh, the cruft." From mark at msapiro.net Sat Apr 14 11:06:00 2018 From: mark at msapiro.net (Mark Sapiro) Date: Sat, 14 Apr 2018 08:06:00 -0700 Subject: [Mailman-Users] Bug in Mailman version 2.1.25 In-Reply-To: References: <000001d3d31a$3ca2f7c0$b5e8e740$@icabs.co.zw> <46f78857-74c7-c523-7992-0d38e7105a49@msapiro.net> <001201d3d3ec$df669160$9e33b420$@icabs.co.zw> Message-ID: On 04/14/2018 06:12 AM, Odhiambo Washington wrote: > > Here is what I would do: ... The suggestions here are way too complicated. Do the following: Either run Mailman's bin/rmlist myproblemlist or just rm -r /usr/local/mailman/lists/myproblemlist/ These will do the same thing which is remove the list but not it's archives. Then recreate the list either via the web UI or Mailman's bin/newlist. Configure the list as desired and add the members. -- Mark Sapiro The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan From mpn at icabs.co.zw Sat Apr 14 15:17:17 2018 From: mpn at icabs.co.zw (MPN Kazishe) Date: Sat, 14 Apr 2018 21:17:17 +0200 Subject: [Mailman-Users] Bug in Mailman version 2.1.25 In-Reply-To: References: <000001d3d31a$3ca2f7c0$b5e8e740$@icabs.co.zw> <46f78857-74c7-c523-7992-0d38e7105a49@msapiro.net> <001201d3d3ec$df669160$9e33b420$@icabs.co.zw> Message-ID: <000001d3d425$39c1ff90$ad45feb0$@icabs.co.zw> Many thanks. I have restored my list and all is well. Lesson learnt backups now in progress! > -----Original Message----- > From: Mailman-Users [mailto:mailman-users- > bounces+mpn=icabs.co.zw at python.org] On Behalf Of Mark Sapiro > Sent: Saturday, 14 April 2018 5:06 PM > To: mailman-users at python.org > Subject: Re: [Mailman-Users] Bug in Mailman version 2.1.25 > > On 04/14/2018 06:12 AM, Odhiambo Washington wrote: > > > > Here is what I would do: > ... > > The suggestions here are way too complicated. > > Do the following: > > Either run Mailman's > > bin/rmlist myproblemlist > > or just > > rm -r /usr/local/mailman/lists/myproblemlist/ > > These will do the same thing which is remove the list but not it's archives. > > Then recreate the list either via the web UI or Mailman's bin/newlist. > > Configure the list as desired and add the members. > > > > -- > Mark Sapiro The highway is for gamblers, > San Francisco Bay Area, California better use your sense - B. Dylan > ------------------------------------------------------ > Mailman-Users mailing list Mailman-Users at python.org > https://mail.python.org/mailman/listinfo/mailman-users > Mailman FAQ: http://wiki.list.org/x/AgA3 Security Policy: > http://wiki.list.org/x/QIA9 Searchable Archives: http://www.mail- > archive.com/mailman-users%40python.org/ > Unsubscribe: https://mail.python.org/mailman/options/mailman- > users/mpn%40icabs.co.zw From steven.jones at vuw.ac.nz Sun Apr 15 18:53:09 2018 From: steven.jones at vuw.ac.nz (Steven Jones) Date: Sun, 15 Apr 2018 22:53:09 +0000 Subject: [Mailman-Users] Brute force attacks on mailman web ui Message-ID: Hi, We are currently under brute force attack on our mailman server's web ui. Is there anything / feature that Mailman has that can be used to watch/monitor it? Sadly I think we'll have to remove it off the Internet..... regards Steven From mailman-admin at uni-konstanz.de Mon Apr 16 03:08:43 2018 From: mailman-admin at uni-konstanz.de (mailman-admin) Date: Mon, 16 Apr 2018 09:08:43 +0200 Subject: [Mailman-Users] Brute force attacks on mailman web ui In-Reply-To: References: Message-ID: <00cfd5c3-c318-6714-214e-25c6b539a6b6@uni-konstanz.de> Hi Am 16.04.2018 um 00:53 schrieb Steven Jones: > Hi, > > We are currently under brute force attack on our mailman server's web ui. > Is there anything / feature that Mailman has that can be used to watch/monitor it? > Sadly I think we'll have to remove it off the Internet..... > > This is not a mailman specific problem. Brute Force attempts can only be mitigated by e.g. fail2ban. It monitors your log files and will block access for IPs that try to login too often with invalid credentials for a short time. This will only catch IPs which try multiple times. A DDoS with constantly changing IPs will still hurt you. Kind regards, Christian Mack From rsk at gsp.org Mon Apr 16 07:38:40 2018 From: rsk at gsp.org (Rich Kulawiec) Date: Mon, 16 Apr 2018 07:38:40 -0400 Subject: [Mailman-Users] Brute force attacks on mailman web ui In-Reply-To: <00cfd5c3-c318-6714-214e-25c6b539a6b6@uni-konstanz.de> References: <00cfd5c3-c318-6714-214e-25c6b539a6b6@uni-konstanz.de> Message-ID: <20180416113840.GA10061@gsp.org> On Mon, Apr 16, 2018 at 09:08:43AM +0200, mailman-admin wrote: > Brute Force attempts can only be mitigated by e.g. fail2ban. Nope. There are other ways. Brute force attacks can be pre-emptively blocked by nearly everyone operating a Mailman instance. (I say "nearly" for specific reasons that will become clear below.) 1. You'll need a firewall, either in front of your Mailman instance or onboard the same system, that can handle a significant number of IP ranges in CIDR format. I highly recommend "pf", which is part of OpenBSD (among others) and is easily the best firewall available. However, you can do this with other firewalls such as iptables just as readily. 2. Get the Spamhaus DROP (Don't Route Or Peer) list, along with the EDROP list: http://www.spamhaus.org/drop/drop.txt http://www.spamhaus.org/drop/edrop.txt They're small. Take a look at them. The DROP/EDROP lists are well-curated and consist of blocks that are known to be hijacked and known to belong to malicious actors. You should *bidirectionally* block *all* traffic to/from the networks listed on them: HTTP, SMTP, DNS, everything. Update them by fetching new copies once a month. 3. The next step depends on the intended audience for your mailing lists. If you're operating one for a bowling league in Akron, Ohio, then you probably don't need to accept subscription attempts from Peru, Pakistan, or Portugal. If on the other hand you're operating one with global reach then you'll need a different approach. In either case, you'll want ipdeny.com's list of all network blocks by country: http://ipdeny.com/ipblocks/data/countries/all-zones.tar.gz and you may want the Okean lists of blocks for China and Korea: http://www.okean.com/chinacidr.txt http://www.okean.com/koreacidr.txt (In theory, the list of CN and KR blocks in ipdeny's compilation should exactly match Okean's. In practice, there are slight differences that tend to resolve over time. I think if you're only configuring for CN and KR, use the latter; if you need more, use the former. But we'll get to that.) By the way, if you use these, you should update them once a month just like the DROP/EDROP lists. 3a. If your list is localized to one country or two or six, then use the ipdeny data to configure your firewall to block *all* HTTP traffic to your mailman instance and then only allow traffic from the countries that you need. This usually takes a huge bite out of incoming abuse. 3b. If your list is global or nearly so in scope, then block as many countries as you can. Look at your logs, see where the attacks are coming from, see where real subscriptions are coming from, and figure out the disjoint set. (The utility "grepcidr" is often very helpful.) If you have, let's say, zero subscriptions from FR over the past nine years but do have a parade of attacks: firewall it out. I recommend, in doing this analysis, that you start by looking at CN and KR. Why? Because two decades of careful logging and analysis of data from quite a few sites indicates that these are #1 and #2 on the hit parade of attackers. There's no point in trying to file complaints in the hope that some responsible professional will take remedial action and make it stop: just firewall and forget. (Next on the list are RU and IN. Same problems.) Comment on 3a and 3b: Yes, this is draconian. That's a good thing. Let me quote for you what I think is arguably the best thing ever said on NANOG, and given the tens of thousands of years of accumulated network experience represented there, that's saying a lot: If you give people the means to hurt you, and they do it, and you take no action except to continue giving them the means to hurt you, and they take no action except to keep hurting you, then one of the ways you can describe the situation is "it isn't scaling well". --- Paul Vixie You can either get to work with a firewall or you can continue to be a victim. Choose. If you want to continue to allow SMTP traffic from these countries so that users can subscribe/unsubscribe/etc. via mail that's your call. I've tailored my configuration to suit the lists that I run and in some cases, that includes configuring the MTA to deny traffic from the same ranges as the web server; in others, it includes denying traffic from the TLD; in others, both. The key to all of this is understanding (a) what you need to allow in order for your lists to function as intended and (b) what your own logs are telling you about what attacks/abuse are coming from. 4. Supplement this by blocking operations that are unlikely to originate any valid subscriptions but are likely to originate abuse. For example, Amazon's cloud -- due to the negligence and incompetence of the people running it -- is a notorious source of nonstop brute-force attacks. They not only refuse to do anything about it, they refuse to even accept complaints. So you might want to firewall AWS outright. (By the way, you most certainly want to block it from any ssh services you run.) Digital Ocean is similar. And there are others. I have compiled full/partial network blocks for these if you want them. Yes, this is also draconian. But I'm done playing around with ignorant newbie lusers -- like the ones at Amazon -- who can't or won't run their networks properly. And I'm not interested in pathetic excuses like "we're too big to watch our own network" or even more feeble ones like "we get too many complaints". Can't handle Operations 101? Shut down the systems, turn off the power, lock the doors, and go home. You're. Not. Good. Enough. 5. Supplement all of this with individual blocks as the need arises. Watch your own logs and look for patterns. If you want to drop individual IP addresses into the firewall, sure, but since attacks generally originate from multiple addresses on a network, it's usually faster, easier, and more efficient to just plonk entire networks. I have some of these compiled as well, although the ones attacking my resources may not be the same as the ones attacking yours. "whois -h whois.arin.net" is your friend. Hint: if you watch your logs long enough and pay attention to what's in them, you'll probably notice that many attack patterns are localized. One of the quick-and-dirty approaches that is often quite effective is to assume that each attacking IP address is part of a /24 (even though is might really be a /28 or a /16) and block the entire putative /24. Yes, this assumption is often wrong, BUT it usually doesn't matter if it's wrong, because the goal here is to stop the attacks from getting through. (Why a /24? Because in some implementations, network blocks that size or larger are handled more efficiently than smaller ones. For example, I never block anything smaller than a /24 in the sendmail access file.) Folks who've never done this before and haven't looked at their logs often think this is too much. It's not. If you spend enough time looking at the data, it becomes apparent in most cases that if you're seeing attacks from one IP address on a network, it's only a matter of time until the second and third and forty-second show up. Might as well block them now and avoid the annoyance. 6. Yes, all of this can be bypassed with proxies and VPNs and Tor and botnets and and and. It's not a panacea. But it does take the edge off, and that in turn makes the remaining problem more tractable. If you do this diligently, you'll find that you'll steadily have less to do. And pre-emptive blocking like this is vastly more effective that reactive blocking like fail2ban, if for no other reason than it makes your logs smaller and easier to deal with. I wish none of this was necessary. It *shouldn't* be necessary. But a lot of operations are run by negligent, incompetent people, and others are run by people who are deliberately allowing attackers and abusers to use their networks. We are well past the time when asking either of these nicely if just maybe they'd like to show some professional responsibility and run their networks properly: there is no point in it. Firewall and forget. ---rsk From david at midrange.com Mon Apr 16 09:39:44 2018 From: david at midrange.com (David Gibbs) Date: Mon, 16 Apr 2018 08:39:44 -0500 Subject: [Mailman-Users] Brute force attacks on mailman web ui In-Reply-To: References: Message-ID: On 4/15/2018 5:53 PM, Steven Jones wrote: > We are currently under brute force attack on our mailman server's web > ui. > > Is there anything / feature that Mailman has that can be used to > watch/monitor it? Can you elaborate on how they are attacking? If it's a detectable pattern, I suggest you investigate fail2ban (as Christian suggested). david -- IBM i on Power Systems: For when you can't afford to be out of business! I'm riding 615 miles (Yes, you read that right) in the American Diabetes Association's Tour de Cure to raise money for diabetes research, education, advocacy, and awareness. You can make a tax deductible donation to my ride by visiting https://gmane.diabetessucks.net. You can see where my donations come from by visiting my interactive donation map ... https://gmane.diabetessucks.net/map (it's a geeky thing). I may have diabetes, but diabetes doesn't have me! From fmouse at fmp.com Mon Apr 16 10:46:21 2018 From: fmouse at fmp.com (Lindsay Haisley) Date: Mon, 16 Apr 2018 09:46:21 -0500 Subject: [Mailman-Users] Brute force attacks on mailman web ui In-Reply-To: References: Message-ID: <1523889981.85783.44.camel@fmp.com> On Sun, 2018-04-15 at 22:53 +0000, Steven Jones wrote: > We are currently under brute force attack on our mailman server's web > ui. > > > Is there anything / feature that Mailman has that can be used to > watch/monitor it? A related question would be whether there's any way to correlate failed web UI login attempts with IP addresses. It doesn't appear that at present Mailman 2 logs failed web UI attempts at all, although I may be missing something. If this were possible, a system-level utility such as fail2ban could be used to monitor logs and establish kernel filter rules to block these IPs. -- Lindsay Haisley | "The first casualty when FMP Computer Services | war comes is truth." 512-259-1190 | http://www.fmp.com | -- Hiram W Johnson From mark at msapiro.net Mon Apr 16 12:29:16 2018 From: mark at msapiro.net (Mark Sapiro) Date: Mon, 16 Apr 2018 09:29:16 -0700 Subject: [Mailman-Users] Brute force attacks on mailman web ui In-Reply-To: <1523889981.85783.44.camel@fmp.com> References: <1523889981.85783.44.camel@fmp.com> Message-ID: <15fd540d-a115-0cca-962b-ac1ef89f41f2@msapiro.net> On 04/16/2018 07:46 AM, Lindsay Haisley wrote: > > A related question would be whether there's any way to correlate failed > web UI login attempts with IP addresses. It doesn't appear that at > present Mailman 2 logs failed web UI attempts at all, although I may be > missing something. Mailman responds to invalid login attempts from the admin, admindb, options and private CGIs with a 401 Unauthorized status. These are (or should be) logged by the web server along with the IP address and other info. In addition, if a list's membership is private, i.e. available only to members or the admin, failed attempts to log in to the options page or obtain a password reminder are logged by Mailman in the mischief log, but only login failures have the IP address. -- Mark Sapiro The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan From heller at deepsoft.com Mon Apr 16 13:26:09 2018 From: heller at deepsoft.com (Robert Heller) Date: Mon, 16 Apr 2018 13:26:09 -0400 (EDT) Subject: [Mailman-Users] Brute force attacks on mailman web ui In-Reply-To: <1523889981.85783.44.camel@fmp.com> References: <1523889981.85783.44.camel@fmp.com> Message-ID: <20180416172610.EE21A26C27B6@sharky3.deepsoft.com> At Mon, 16 Apr 2018 09:46:21 -0500 fmouse at fmp.com wrote: > > On Sun, 2018-04-15 at 22:53 +0000, Steven Jones wrote: > > We are currently under brute force attack on our mailman server's web > > ui. > > > > > > Is there anything / feature that Mailman has that can be used to > > watch/monitor it? > > A related question would be whether there's any way to correlate failed > web UI login attempts with IP addresses. It doesn't appear that at > present Mailman 2 logs failed web UI attempts at all, although I may be > missing something. They might be in Apache's logs. > > If this were possible, a system-level utility such as fail2ban could be > used to monitor logs and establish kernel filter rules to block these > IPs. > -- Robert Heller -- 978-544-6933 Deepwoods Software -- Custom Software Services http://www.deepsoft.com/ -- Linux Administration Services heller at deepsoft.com -- Webhosting Services From fmouse at fmp.com Mon Apr 16 13:45:29 2018 From: fmouse at fmp.com (Lindsay Haisley) Date: Mon, 16 Apr 2018 12:45:29 -0500 Subject: [Mailman-Users] Brute force attacks on mailman web ui In-Reply-To: <20180416172610.EE21A26C27B6@sharky3.deepsoft.com> References: <1523889981.85783.44.camel@fmp.com> <20180416172610.EE21A26C27B6@sharky3.deepsoft.com> Message-ID: <1523900729.107035.11.camel@fmp.com> On Mon, 2018-04-16 at 13:26 -0400, Robert Heller wrote: > > > Is there anything / feature that Mailman has that can be used to > > > watch/monitor it? > >? > > A related question would be whether there's any way to correlate failed > > web UI login attempts with IP addresses. It doesn't appear that at > > present Mailman 2 logs failed web UI attempts at all, although I may be > > missing something. > > They might be in Apache's logs. Apache will log the access, with IP addresse, but to the best of my knowledge it won't log a Web UI login failure since this is an internal matter for Mailman. The connecting IP address is available in the environment to any web application and it shouldn't be difficult to set up logging for login failures. -- Lindsay Haisley | "The first casualty when FMP Computer Services | war comes is truth." 512-259-1190 | http://www.fmp.com | -- Hiram W Johnson From tlhackque at yahoo.com Mon Apr 16 14:05:35 2018 From: tlhackque at yahoo.com (tlhackque) Date: Mon, 16 Apr 2018 14:05:35 -0400 Subject: [Mailman-Users] Brute force attacks on mailman web ui In-Reply-To: <20180416113840.GA10061@gsp.org> References: <00cfd5c3-c318-6714-214e-25c6b539a6b6@uni-konstanz.de> <20180416113840.GA10061@gsp.org> Message-ID: <104b4fe8-ffa5-5b47-95f7-10de69a6e5ec@yahoo.com> On 16-Apr-18 07:38, Rich Kulawiec wrote: > On Mon, Apr 16, 2018 at 09:08:43AM +0200, mailman-admin wrote: >> Brute Force attempts can only be mitigated by e.g. fail2ban. > Nope. There are other ways. > > Brute force attacks can be pre-emptively blocked by nearly everyone > operating a Mailman instance. (I say "nearly" for specific reasons > that will become clear below.) > > 1. You'll need a firewall, either in front of your Mailman instance > or onboard the same system, that can handle a significant number of > IP ranges in CIDR format. I highly recommend "pf", which is part > of OpenBSD (among others) and is easily the best firewall available. > However, you can do this with other firewalls such as iptables just as readily. > > 2. Get the Spamhaus DROP (Don't Route Or Peer) list, along with the EDROP list: > > http://www.spamhaus.org/drop/drop.txt > http://www.spamhaus.org/drop/edrop.txt > > They're small. Take a look at them. > > The DROP/EDROP lists are well-curated and consist of blocks that are > known to be hijacked and known to belong to malicious actors. You > should *bidirectionally* block *all* traffic to/from the networks > listed on them: HTTP, SMTP, DNS, everything. > > Update them by fetching new copies once a month. Good advice.??? But use httpS: (and make sure the UA validates the server certificate). Unless you fancy experimenting with DOS attacks. > 3. The next step depends on the intended audience for your mailing > lists. If you're operating one for a bowling league in Akron, Ohio, > then you probably don't need to accept subscription attempts from > Peru, Pakistan, or Portugal. If on the other hand you're operating > one with global reach then you'll need a different approach. > > In either case, you'll want ipdeny.com's list of all network blocks > by country: > > http://ipdeny.com/ipblocks/data/countries/all-zones.tar.gz > > and you may want the Okean lists of blocks for China and Korea: > > http://www.okean.com/chinacidr.txt > http://www.okean.com/koreacidr.txt > > (In theory, the list of CN and KR blocks in ipdeny's compilation > should exactly match Okean's. In practice, there are slight differences > that tend to resolve over time. I think if you're only configuring > for CN and KR, use the latter; if you need more, use the former. > But we'll get to that.) > > By the way, if you use these, you should update them once a month > just like the DROP/EDROP lists. > > 3a. If your list is localized to one country or two or six, then > use the ipdeny data to configure your firewall to block *all* HTTP traffic > to your mailman instance and then only allow traffic from the countries > that you need. This usually takes a huge bite out of incoming abuse. > > 3b. If your list is global or nearly so in scope, then block as many > countries as you can. Look at your logs, see where the attacks are coming > from, see where real subscriptions are coming from, and figure out the > disjoint set. (The utility "grepcidr" is often very helpful.) > If you have, let's say, zero subscriptions from FR over the past > nine years but do have a parade of attacks: firewall it out. > > I recommend, in doing this analysis, that you start by looking at CN > and KR. Why? Because two decades of careful logging and analysis > of data from quite a few sites indicates that these are #1 and #2 on > the hit parade of attackers. There's no point in trying to file complaints > in the hope that some responsible professional will take remedial action > and make it stop: just firewall and forget. (Next on the list are > RU and IN. Same problems.) Yup.??? And iran and afghanistan & the other former soviet countries. But the biggest source of attacks, by far, is the US.??? Unfortunately, while some people run business that don't interact with the US, in most cases a non-country based approach is necessary for that :-) > Comment on 3a and 3b: > > Yes, this is draconian. That's a good thing. Let me quote for you what > I think is arguably the best thing ever said on NANOG, and given the tens > of thousands of years of accumulated network experience represented > there, that's saying a lot: > > If you give people the means to hurt you, and they do it, and > you take no action except to continue giving them the means to > hurt you, and they take no action except to keep hurting you, > then one of the ways you can describe the situation is "it isn't > scaling well". > --- Paul Vixie > > You can either get to work with a firewall or you can continue to be > a victim. Choose. > > If you want to continue to allow SMTP traffic from these countries > so that users can subscribe/unsubscribe/etc. via mail that's your > call. I've tailored my configuration to suit the lists that I run > and in some cases, that includes configuring the MTA to deny > traffic from the same ranges as the web server; in others, > it includes denying traffic from the TLD; in others, both. The key > to all of this is understanding (a) what you need to allow in order > for your lists to function as intended and (b) what your own logs are > telling you about what attacks/abuse are coming from. If you have iptables, you also might want to look at https://github.com/tlhackque/BlockCountries A new release that provides better management is overdue -- but hopefully soon. BlockCountries manages the by-country blocking, and the next release will support fetching the spamhaus (and other) non-country lists.??? It handles both the "block these" and "permit only these" paradigms, and does a fair amount of optimization of the rulesets.??? It also handles IPv6 & has a bunch of options that let you avoid most of the hassle of working directly with iptables for exceptions. You do need to be careful to balance reducing your attack surface with your business' need for access.??? Also, be aware that the "by-country" blocking lists of ipdeny come from the regional registries.??? They are not exact.??? They don't necessarily reflect the server's actual current physical location for a number of reasons.??? But they are a reasonable approximation.??? Just be sure that you have an easy way to introduce exceptions when one of your customers is blocked. I also recommend https://zeustracker.abuse.ch/blocklist.php (add?download=badips for the data). > > 4. Supplement this by blocking operations that are unlikely to originate > any valid subscriptions but are likely to originate abuse. For example, > Amazon's cloud -- due to the negligence and incompetence of the people > running it -- is a notorious source of nonstop brute-force attacks. > They not only refuse to do anything about it, they refuse to even > accept complaints. So you might want to firewall AWS outright. > (By the way, you most certainly want to block it from any ssh services > you run.) Digital Ocean is similar. And there are others. I have > compiled full/partial network blocks for these if you want them. The best defense for ssh is to configure it for certificate authentication only. The script kiddies will make their 10,000 login attempts - while they're busy annoying you, they're not attacking someone else.??? Or so one hopes.?????? fail2ban is also useful if you're not so civic minded... [I'm not kidding; I do see lists of 10K+ attempts from "adam adam", "adam password" thru "zeke password" "zeke zeke"...] If you keep up your lists of cloud services' network blocks & have them on a publicly accessible website, I'll add them to my list of optional block lists.??? (Hopefully you use a standard format - e.g. ipaddress[/netmask or length] with # or ; comments...) > Yes, this is also draconian. But I'm done playing around with > ignorant newbie lusers -- like the ones at Amazon -- who can't > or won't run their networks properly. And I'm not interested > in pathetic excuses like "we're too big to watch our own network" > or even more feeble ones like "we get too many complaints". Can't > handle Operations 101? Shut down the systems, turn off the power, > lock the doors, and go home. You're. Not. Good. Enough. > > 5. Supplement all of this with individual blocks as the need arises. > Watch your own logs and look for patterns. If you want to drop > individual IP addresses into the firewall, sure, but since attacks > generally originate from multiple addresses on a network, it's usually > faster, easier, and more efficient to just plonk entire networks. > I have some of these compiled as well, although the ones attacking > my resources may not be the same as the ones attacking yours. > "whois -h whois.arin.net" is your friend. > > Hint: if you watch your logs long enough and pay attention to what's > in them, you'll probably notice that many attack patterns are localized. > One of the quick-and-dirty approaches that is often quite effective > is to assume that each attacking IP address is part of a /24 > (even though is might really be a /28 or a /16) and block the > entire putative /24. Yes, this assumption is often wrong, BUT > it usually doesn't matter if it's wrong, because the goal here is > to stop the attacks from getting through. (Why a /24? Because in > some implementations, network blocks that size or larger are handled > more efficiently than smaller ones. For example, I never block > anything smaller than a /24 in the sendmail access file.) > > Folks who've never done this before and haven't looked at their > logs often think this is too much. It's not. If you spend enough > time looking at the data, it becomes apparent in most cases that > if you're seeing attacks from one IP address on a network, it's only > a matter of time until the second and third and forty-second show > up. Might as well block them now and avoid the annoyance. Whatever lists you use, it's really important to stay up to date.??? (I generally recommend weekly or monthly.)??? The attackers are mobile and adapt.??? That cuts both ways: if you block a specific attack today, tomorrow they may well have moved to another host.??? Then you have to block them again.??? And, the block that they vacate may be assigned to a legitimate use - that you don't want to block. Be very careful not to block your customers or users; it is easy to be overly aggressive, which can be career limiting. > 6. Yes, all of this can be bypassed with proxies and VPNs and Tor > and botnets and and and. It's not a panacea. But it does take the > edge off, and that in turn makes the remaining problem more tractable. > If you do this diligently, you'll find that you'll steadily have > less to do. And pre-emptive blocking like this is vastly more effective > that reactive blocking like fail2ban, if for no other reason than > it makes your logs smaller and easier to deal with. > > I wish none of this was necessary. It *shouldn't* be necessary. > But a lot of operations are run by negligent, incompetent people, > and others are run by people who are deliberately allowing attackers > and abusers to use their networks. We are well past the time when > asking either of these nicely if just maybe they'd like to show some > professional responsibility and run their networks properly: there is > no point in it. Firewall and forget. > > ---rsk > Sadly, this is true.??? However, e-mail to abuse@ can be effective if you provide details... -- This communication may not represent my employer's views, if any, on the matters discussed. From mark at msapiro.net Mon Apr 16 14:06:20 2018 From: mark at msapiro.net (Mark Sapiro) Date: Mon, 16 Apr 2018 11:06:20 -0700 Subject: [Mailman-Users] Brute force attacks on mailman web ui In-Reply-To: <1523900729.107035.11.camel@fmp.com> References: <1523889981.85783.44.camel@fmp.com> <20180416172610.EE21A26C27B6@sharky3.deepsoft.com> <1523900729.107035.11.camel@fmp.com> Message-ID: <5788eda1-fc99-d5b1-fe62-ad7d5e869e42@msapiro.net> On 04/16/2018 10:45 AM, Lindsay Haisley wrote: > > Apache will log the access, with IP addresse, but to the best of my > knowledge it won't log a Web UI login failure since this is an internal > matter for Mailman. As I said in my prior reply, all Mailman login failures return a 401 status. Just look in the Apache logs for Mailman URLs with a 401 status. -- Mark Sapiro The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan From fmouse at fmp.com Mon Apr 16 14:29:55 2018 From: fmouse at fmp.com (Lindsay Haisley) Date: Mon, 16 Apr 2018 13:29:55 -0500 Subject: [Mailman-Users] Brute force attacks on mailman web ui In-Reply-To: <5788eda1-fc99-d5b1-fe62-ad7d5e869e42@msapiro.net> References: <1523889981.85783.44.camel@fmp.com> <20180416172610.EE21A26C27B6@sharky3.deepsoft.com> <1523900729.107035.11.camel@fmp.com> <5788eda1-fc99-d5b1-fe62-ad7d5e869e42@msapiro.net> Message-ID: <1523903395.107035.36.camel@fmp.com> On Mon, 2018-04-16 at 11:06 -0700, Mark Sapiro wrote: > On 04/16/2018 10:45 AM, Lindsay Haisley wrote: > >? > > Apache will log the access, with IP addresse, but to the best of my > > knowledge it won't log a Web UI login failure since this is an internal > > matter for Mailman. > > > As I said in my prior reply, Sorry, Mark. I've been running about and missed your reply. > all Mailman login failures return a 401 > status. Just look in the Apache logs for Mailman URLs with a 401 status. In which case, fail2ban should be able to parse these from log files quite reliably. It's a very effective security tool which parses log files of your choice, looking for strings (by regular expression) of your choice, and writes rules to the system firewall (via iptables in the case of Linux). Your challenge with fail2ban is writing the search rules. fail2ban allows very flexible criteria for determining what constitutes an attack, how long a blocking rules should last, etc. I use it for many kinds of attacks and probes such as ssh brute force attacks, Apache attempts to access non-existent pages, WordPress login failures (via the "Fail2ban Redux" plugin), FTP login failures, and a couple of others. As along as there's a log file which provides a basic unique failure string, and an IP address source for the failure, fail2ban will manage blocking.? -- Lindsay Haisley | "The first casualty when FMP Computer Services | war comes is truth." 512-259-1190 | http://www.fmp.com | -- Hiram W Johnson From rsk at gsp.org Tue Apr 17 10:20:13 2018 From: rsk at gsp.org (Rich Kulawiec) Date: Tue, 17 Apr 2018 10:20:13 -0400 Subject: [Mailman-Users] Brute force attacks on mailman web ui In-Reply-To: <104b4fe8-ffa5-5b47-95f7-10de69a6e5ec@yahoo.com> References: <00cfd5c3-c318-6714-214e-25c6b539a6b6@uni-konstanz.de> <20180416113840.GA10061@gsp.org> <104b4fe8-ffa5-5b47-95f7-10de69a6e5ec@yahoo.com> Message-ID: <20180417142013.GA19660@gsp.org> On Mon, Apr 16, 2018 at 02:05:35PM -0400, tlhackque via Mailman-Users wrote: > Good advice.??? But use httpS: (and make sure the UA validates the server > certificate). > Unless you fancy experimenting with DOS attacks. Yep. You're exactly right. > But the biggest source of attacks, by far, is the US.??? Unfortunately, > while some people run business that don't interact with the US, in most > cases a non-country based approach is necessary for that :-) Yes. There's no question that the US is a huge source of attacks, and if I were running a mailing list for birdwatchers in Australia, I'd seriously consider blocking it. But you're right, that bumps into all kinds of hosting/infrastructure issues and so blocking the whole country will likely have unpleasant side effects. > https://github.com/tlhackque/BlockCountries > A new release that provides better management is overdue -- but > hopefully soon. That...is cool. Thanks for the pointer. > The best defense for ssh is to configure it for certificate > authentication only. >The script kiddies will make their 10,000 login attempts [...] True, but I find the clutter in logs annoying. ;) So in situations where I know a priori that a valid login attempt will never originate from an operation, I just firewall it and let them eat dropped packets. > [I'm not kidding; I do see lists of 10K+ attempts from "adam adam", > "adam password" thru "zeke password" "zeke zeke"...] I stood up a new server last fall with *no* valid ssh access and logged about 750,000 attempts in a month. Similar patterns. > If you keep up your lists of cloud services' network blocks & have them > on a publicly accessible > website, I'll add them to my list of optional block lists.??? (Hopefully > you use a standard format - e.g. > ipaddress[/netmask or length] with # or ; comments...) I keep them in CIDRnetwork-name but honestly I'm not diligent enough about maintaining them. As a result, they're always under-inclusive (very rarely over-inclusive). That works for what I use them for, but I'm hesitant to inflict my laziness on others. Let me see if I can locate someone who's doing a better job than I am. ---rsk From rsk at gsp.org Tue Apr 17 10:28:20 2018 From: rsk at gsp.org (Rich Kulawiec) Date: Tue, 17 Apr 2018 10:28:20 -0400 Subject: [Mailman-Users] GSOC idea: mail server/DNS server/mailing list healthcheck Message-ID: <20180417142617.GA20774@gsp.org> The idea for this comes from some of the web sites that perform this; unfortunately most of them are "upgrading" from simple, fast, easy checks to bloated ones that use a ton of Javascript, can't be scripted, and are increasingly behind signups/paywalls/etc. The concept is simple: given a domain, check its DNS, mail, etc. setup for correctness and consistency. For example: - does it have an A record? - is that valid hostname? - does it have an AAAA record? - is that valid hostname? - does it have MX records? - are the MX records *not* CNAMEs? - are they valid hostnames? - do those hostnames resolve to public IP addresses? - are any of those MX records duplicates? - are all the MX responding? - are the MX weights valid? - do all MX's pass FCrDNS check? - does it have NS records? - are they valid hostnames? - do they have A, AAAA records? - are they in public IP space? - are the NS records *not* CNAMEs? - do all NS pass FCrDNS check? - are any of those NS records duplicates? - does the list of NS match the list of authoritative NS? - do all the NS return the same list of NS? - do all the NS return the same list of MX? - do the NS *not* allow recursion? - are any of the NS lame? - are any of the NS missing? - are all the NS responding? - is there a working postmaster address? - is there a working abuse address? - is there a working hostmaster address? - if not is there a working address that matches the one in the SOA? - is the domain's SOA sane? (e.g. plausible serial number, refresh, retry, etc.) - do all the NS return the same SOA with the same serial number? - is there ASN diversity among the NS? - and so on Those are the sort of checks that pertain to the operation of any domain and its nameservers and mail exchangers. But in addition, if run on a Mailman 2 or 3 host: - what mailing lists are public? - what mailing lists are private? - does every list have a working -request address? - does every list have a working -owner address? - does every list have a working -bounces address? - if any list supports the optional -subscribe address, does it have a -unsubscribe address? - if any list supports the optional -join address, does it have a -leave address? - and so on Note: command-line tool. It has to be, because then it can be scripted and run via cron and so on. Besides, anyone running name servers, mail servers, etc., had better be able to work at the command line. Note: should work on systems that aren't running Mailman: just can't analyze Mailman then, of course. This leaves open the door for people using other MLMs to write checks that work with those. And maybe that'd be a nice thing to do. Note: should have varying levels of verbosity, including one that explains why something is wrong by referencing RFCs/BCPs/manual by chapter and verse. Note: the second set of checks (above) are outside the scope of what Mailman checks inside itself. That is, they require correlating what Mailman thinks should be in place versus what's actually in place in the MTA. ---rsk From rsk at gsp.org Tue Apr 17 10:56:38 2018 From: rsk at gsp.org (Rich Kulawiec) Date: Tue, 17 Apr 2018 10:56:38 -0400 Subject: [Mailman-Users] GSOC idea: The central scrutinizer ;) Message-ID: <20180417145638.GA22067@gsp.org> I have a partially-completed spec for a module that will examine messages for various issues but my Python-fu is likely not sufficient to realize it and I'm busy writing anyway. This is probably a GSOC-size and GSOC-scope project, so if anybody is game, below is a poorly-written and large incomplete description of what I have in mind. 1. As I've said for years, anti-abuse controls should be layered and should start at the network perimeter (with things like the Spamhaus DROP list in border routers). Additional layers may be in firewalls, in the MTA, in the MLM (such as Mailman). No one layer can catch everything because it doesn't have contextual knowledge, e.g., the firewall doesn't know who the members of a mailing list are or even that a particular SMTP connection it's allowing contains traffic for a mailing list. Unfortunately, email accounts have been hijacked by the billions (and that's just counting Yahoo). The new owners of those accounts likely enjoy the same SMTP reachability that the old owners did and can thus drop messages onto mailing lists. (I'm going to omit the long explanation about why context-sensitive measures in the MTA are not only inadequate to stop this, but are also highly undesirable.) To put it another way: we don't have many defenses against our friends. But we need them. 2. We all have our own views on proper netiquette for mail/mailing lists, and of course, mine are the only correct ones. ;) But regardless of those, it'd be useful to have a mechanism to scrutinize messages for things like "800 lines of quoted digest with one top-posted line above it". Of course some of this is easy to see and hard to code, but I think a modest attempt in this direction would be helpful. (Besides, if the action taken on detection of these is to merely hold the message, then little harm is done by false positives.) 3. 1 and 2 are related. The same kinds of criteria that are useful in detecting and putting a hold on spam are useful in detecting undesirable content in messages, like URL shorteners and third-party tracking links. So it makes sense to have a highly configurable module that comes with a minimal/loose default configuration that can be salted to taste. So what I have in mind is a module that scrutinizes messages based on a set of (enabled/disabled) criteria, each one of which is configurable. I know that's not very clear. Let me make up an example and see if that helps. Let's suppose we call this module the Central Scrutinizer (CS) because oblique tributes to Frank Zappa are always in order. The CS might have a list of dozens of checks like this: - URL shorteners (1) - tracking links (2) - full digest quoting (3) Check (1) would consult a list of known URL shortening domains and do pattern matching of any URLs in the message against them. Check (2) would attempt to detect tracking links/"web bugs". Check (3) would check messages to see if it includes an entire quoted digest. Each one of these would be associated with an action -- and I strongly think that "hold" would be best, because these tests are going to make lots of mistakes for a while. The results would be presented to list owners (in email notifications and in the browser interface) with something like: Central scrutinizer report: - URL shorteners - fail, example.net detected - tracking links - pass, none detected - full digest quoting - pass, none detected with appropriate presentation so that it's easy to read and so that when a test fails, it reports *why* it failed. Let me note that the MTA is wrong place to do this stuff for a whole bunch of reasons. Among other things, lots and lots of people want different policies applied to mailing list traffic (which will result in outbound mail traffic) than they due to local traffic (which won't). This is a serious issue for anybody concerned with their mail system's reputation, which is based on what it emits, not what it accepts. Of course not everyone will agree with every check, and not every check is appropriate on every mailing list. (For example, a one-way announce-only list is unlikely to need most of these.) But if this is modular, and if individual checks can be switched on/off readily, then it can ship with everything off and folks can enable whatever subset they find palatable. Let me also note that the motivation for this comes not just from things I've bumped into while running lists, but things I've seen on other lists. (And I'm on rather a lot of them.) I've been thinking about this for years, but have become convinced in the last six months or so that the problem has now reached a point where a solution is worth constructing. ---rsk From tlhackque at yahoo.com Tue Apr 17 12:28:21 2018 From: tlhackque at yahoo.com (tlhackque) Date: Tue, 17 Apr 2018 12:28:21 -0400 Subject: [Mailman-Users] GSOC idea: mail server/DNS server/mailing list healthcheck In-Reply-To: <20180417142617.GA20774@gsp.org> References: <20180417142617.GA20774@gsp.org> Message-ID: <3f1d7512-a7b0-aaad-458d-b2bcee95cbc9@yahoo.com> On 17-Apr-18 10:28, Rich Kulawiec wrote: > The idea for this comes from some of the web sites that perform this; > unfortunately most of them are "upgrading" from simple, fast, easy > checks to bloated ones that use a ton of Javascript, can't be scripted, > and are increasingly behind signups/paywalls/etc. > > The concept is simple: given a domain, check its DNS, mail, etc. > setup for correctness and consistency. For example: > > - does it have an A record? > - is that valid hostname? > - does it have an AAAA record? > - is that valid hostname? > - does it have MX records? > - are the MX records *not* CNAMEs? > - are they valid hostnames? > - do those hostnames resolve to public IP addresses? > - are any of those MX records duplicates? > - are all the MX responding? > - are the MX weights valid? > - do all MX's pass FCrDNS check? > - does it have NS records? > - are they valid hostnames? > - do they have A, AAAA records? > - are they in public IP space? > - are the NS records *not* CNAMEs? > - do all NS pass FCrDNS check? > - are any of those NS records duplicates? > - does the list of NS match the list of authoritative NS? > - do all the NS return the same list of NS? > - do all the NS return the same list of MX? > - do the NS *not* allow recursion? > - are any of the NS lame? > - are any of the NS missing? > - are all the NS responding? > - is there a working postmaster address? > - is there a working abuse address? > - is there a working hostmaster address? > - if not is there a working address that matches the one in the SOA? > - is the domain's SOA sane? (e.g. plausible serial number, > refresh, retry, etc.) > - do all the NS return the same SOA with the same serial number? > - is there ASN diversity among the NS? > - and so on There are plenty of tools for this.? Don't re-invent the wheel. Here are a few (free) web-based tools (in no particular order); each has its niche: http://www.digwebinterface.com/ https://www.ssllabs.com/ssltest/ https://www.huque.com/bin/danecheck https://dane.sys4.de/ https://mxtoolbox.com/ # many non-mx tests too https://www.dnsstuff.com/ http://dnscheck.iis.se/ https://www.zonemaster.net/ http://dnsviz.net https://www.ultratools.com/tools/asnInfo # and many other tools. http://www.kitterman.com/spf/validate.html https://www.dmarcanalyzer.com/ # Tools.? All but DMARC Analyzer are free; it's freemium Yes, there is a trend toward heavy UIs, but these are not frequent tests.? If you set things up properly, it's not set and forget - but set & validate changes need not be frequent enough to care. If you maintain a few domains, it's not worth scripting.? If you maintain a lot, the key is picking reputable providers for whatever you don't do yourself.? If you have decent providers, there's not much for you to monitor.? If you don't, you have bigger problems. curl will let you script access to most, or you can use libraries such as Perl's LWP, WWW::Mechanize::Firefox, WWW::Scripter, WWW::Selenium - the last few support Javascript by running a browser. It's better to maintain an interface to tests that someone else maintains than to try to create & maintain all the tests - many seem easy, but you'll find it becomes a serious time sink.? In the long term, there is no such thing as a "small" P.roject. > Those are the sort of checks that pertain to the operation of any domain > and its nameservers and mail exchangers. But in addition, if run on a > Mailman 2 or 3 host: > > - what mailing lists are public? > - what mailing lists are private? > - does every list have a working -request address? > - does every list have a working -owner address? > - does every list have a working -bounces address? > - if any list supports the optional -subscribe address, > does it have a -unsubscribe address? > - if any list supports the optional -join address, > does it have a -leave address? > - and so on These are more interesting for mailman-users.? V3 has an API - but it can be quite painful to navigate; I opened quite a few issues before concluding that V3 isn't ready for (my) primetime. V2 has some CLI tools in /usr/lib/mailman/bin - parsing output can get you a lot of information. In either case, it's not particularly interesting what Mailman THINKS is configured; more things go wrong with bitrot in mailserver configuration than in Mailman.? So you need to script something that actually tests each action, receives list mail.? And while you (and I) may use the CLIs, most users will use a GUI.? You have to test what the customer uses.? So again, curl and company are your friends. > Note: command-line tool. It has to be, because then it can be scripted > and run via cron and so on. Besides, anyone running name servers, > mail servers, etc., had better be able to work at the command line. > > Note: should work on systems that aren't running Mailman: just can't > analyze Mailman then, of course. This leaves open the door for people > using other MLMs to write checks that work with those. And maybe that'd > be a nice thing to do. > > Note: should have varying levels of verbosity, including one that > explains why something is wrong by referencing RFCs/BCPs/manual by > chapter and verse. > > Note: the second set of checks (above) are outside the scope of what > Mailman checks inside itself. That is, they require correlating > what Mailman thinks should be in place versus what's actually in > place in the MTA. > > ---rsk > There's a lot of work to turn this wish list into reality and get it right. Have fun.? And remember: "A small matter of code usually isn't." P.S. In your spare time, do start that bird-watching site in Australia.? I'll be interested to hear how long blocking the US lasts. -- This communication may not represent my employer's views, if any, on the matters discussed. From cpz at tuunq.com Tue Apr 17 23:27:42 2018 From: cpz at tuunq.com (Carl Zwanzig) Date: Tue, 17 Apr 2018 20:27:42 -0700 Subject: [Mailman-Users] Brute force attacks on mailman web ui In-Reply-To: <20180417142013.GA19660@gsp.org> References: <00cfd5c3-c318-6714-214e-25c6b539a6b6@uni-konstanz.de> <20180416113840.GA10061@gsp.org> <104b4fe8-ffa5-5b47-95f7-10de69a6e5ec@yahoo.com> <20180417142013.GA19660@gsp.org> Message-ID: On 4/17/2018 7:20 AM, Rich Kulawiec wrote: > I stood up a new server last fall with *no* valid ssh access and logged > about 750,000 attempts in a month. Similar patterns. There's a reason I don't put sshd on port 22; moving it elsewhere and blackhole-ing 22 cut the auth log tremendously. (Nothing to do with mailman, though.) z! From stephen at xemacs.org Thu Apr 19 02:46:35 2018 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Thu, 19 Apr 2018 15:46:35 +0900 Subject: [Mailman-Users] Brute force attacks on mailman web ui In-Reply-To: <20180416113840.GA10061@gsp.org> References: <00cfd5c3-c318-6714-214e-25c6b539a6b6@uni-konstanz.de> <20180416113840.GA10061@gsp.org> Message-ID: <23256.15179.454906.332210@turnbull.sk.tsukuba.ac.jp> Rich Kulawiec writes: > Brute force attacks can be pre-emptively blocked by nearly everyone > operating a Mailman instance. (I say "nearly" for specific reasons > that will become clear below.) Nice summary! > 3. The next step depends on the intended audience for your mailing > lists. So here's my problem. A lot of my constituency resides in CN, occasionally including people at frequently problematic domains like 163.com. Do you know any resources (or keywords to start googling even!) at subnational levels? KR and CN breakdowns would be most useful to me; breakdowns for RU and former USSR would be appreciated by many of my colleagues. > Hint: if you watch your logs long enough and pay attention to what's > in them, you'll probably notice that many attack patterns are localized. This is helpful regardless of whether there are subnational breakdowns. I got the point the first time! :-) Regards, Steve From jon.clements at umass.edu Thu Apr 19 07:16:16 2018 From: jon.clements at umass.edu (Jon Clements) Date: Thu, 19 Apr 2018 07:16:16 -0400 Subject: [Mailman-Users] Mailman not sending mail??? Message-ID: Hi, my Mailman seems to have stopped sending mail. Posts to a list(s) are not going anywhere, nor for example, subscribe notifications, etc. are not going out. It seems to be receiving mail as pending moderator requests are there. Webadmin seems to be working normally (except for not sending mail when requested). Where do I start looking? Sorry, not very technically proficient, it's a wonder I got it (Mailman v. 2.1.20) up and running a couple years ago on Mac OS 10.10.5 (using Server v. 5.0.15) but it has been running great since then (until now). I will say there was quite a bit of spam coming in before the problem started, and I stopped Mail service (on Server) for about a day to hopefully make it go away, and then the Mailman sending problem cropped up after I re-started Mail service (I think). Local/normal Mail accounts seem to be working fine. Appreciate any help you can give me... Jon -- Jon Clements aka 'Mr Honeycrisp' University of Massachusetts Amherst Extension UMass Cold Spring Orchard 393 Sabin St. Belchertown, MA 01007 413-478-7219 Verizon 413-378-3068 Project Fi umassfruit.com From mailman-admin at uni-konstanz.de Thu Apr 19 09:03:43 2018 From: mailman-admin at uni-konstanz.de (mailman-admin) Date: Thu, 19 Apr 2018 15:03:43 +0200 Subject: [Mailman-Users] Mailman not sending mail??? In-Reply-To: References: Message-ID: Am 19.04.2018 um 13:16 schrieb Jon Clements: > Hi, my Mailman seems to have stopped sending mail. Posts to a list(s) are > not going anywhere, nor for example, subscribe notifications, etc. are not > going out. It seems to be receiving mail as pending moderator requests are > there. Webadmin seems to be working normally (except for not sending mail > when requested). Where do I start looking? Sorry, not very technically > proficient, it's a wonder I got it (Mailman v. 2.1.20) up and running a > couple years ago on Mac OS 10.10.5 (using Server v. 5.0.15) but it has been > running great since then (until now). I will say there was quite a bit of > spam coming in before the problem started, and I stopped Mail service (on > Server) for about a day to hopefully make it go away, and then the Mailman > sending problem cropped up after I re-started Mail service (I think). > Local/normal Mail accounts seem to be working fine. > > Appreciate any help you can give me... > I had a died OutgoingRunner a couple of times. Check if all mailman processes are still running. If not restart mailman. Kind regards Christian Mack From weif at weif.net Thu Apr 19 09:57:29 2018 From: weif at weif.net (Keith Seyffarth) Date: Thu, 19 Apr 2018 07:57:29 -0600 Subject: [Mailman-Users] Mailman not sending mail??? In-Reply-To: (message from mailman-admin on Thu, 19 Apr 2018 15:03:43 +0200) Message-ID: <84in8npad2.fsf@maxwell.cjones.org> mailman-admin writes: > I had a died OutgoingRunner a couple of times. > Check if all mailman processes are still running. > If not restart mailman. I run into this frequently on my CentOS machine. If mailman isn't running and it won't restart, check the log files and wherever the PID and lock files are being stored to make sure Mailman can write its files. Some system upgrades appear to "fix" the ownership and permissions on these directories preventing some processes from being able to write their logs or PIDs. This will prevent mailman from sending. Keith -- ---- from my mac to yours... Keith Seyffarth mailto:weif at weif.net http://www.weif.net/ - Home of the First Tank Guide! http://www.rpgcalendar.net/ - the Montana Role-Playing Calendar ---- http://www.miscon.org/ - Montana's Longest Running Science Fiction Convention From tlhackque at yahoo.com Thu Apr 19 12:50:59 2018 From: tlhackque at yahoo.com (tlhackque) Date: Thu, 19 Apr 2018 12:50:59 -0400 Subject: [Mailman-Users] Brute force attacks on mailman web ui In-Reply-To: <23256.15179.454906.332210@turnbull.sk.tsukuba.ac.jp> References: <00cfd5c3-c318-6714-214e-25c6b539a6b6@uni-konstanz.de> <20180416113840.GA10061@gsp.org> <23256.15179.454906.332210@turnbull.sk.tsukuba.ac.jp> Message-ID: <7c6639c7-df14-0bd4-6743-17ca5b58775a@yahoo.com> On 19-Apr-18 02:46, Stephen J. Turnbull wrote: > So here's my problem. A lot of my constituency resides in CN, > occasionally including people at frequently problematic domains like > 163.com. Do you know any resources (or keywords to start googling > even!) at subnational levels? KR and CN breakdowns would be most > useful to me; breakdowns for RU and former USSR would be appreciated > by many of my colleagues. > I'm not sure what you are looking for. Blocking by geography is a very crude tool - it turns out to be useful in that many hosts serve limited geographies, and it's pretty easy to identify countries that generate a lot of "bad" traffic.? E.g. RU & CN are widely believed to support intrusions by (pseudo/)government actors, and rarely prosecute.? As you discovered, below that level, you need to use other tools. There are a number of geolocating services that attempt to turn IP addresses into specific locations; for example maxmind offers a series of databases of increasing precision for increasing prices (starting with free). You can use these databases with your webserver (e.g. apache mod_geoip) and name server (BIND for sure).? There is also a GeoIP module for iptables.? (I use (and maintain) BlockCountries because it is more flexible and easier to use. YMMV). But the problem is that unless you know exactly where your users (and potential users) are located, this won't help.? Do you have a list of cities?? Streets?? I don't think that the criminal element has easily identifiable geographies. What you probably want is to identify the specific bad actors; for that the spamhaus and other "block lists" ("RBL") are helpful.? Most of these are distributed via DNS - which means that they aren't practical for firewalls.? You can configure your email server (e.g. sendmail/postfix) to use them.? But this happens inside your firewall.? These lists are fairly well curated, but certainly aren't perfect. As previously noted, fail2ban is one reactive means of dealing with these - it reads log files and dynamically blocks IP addresses that generate errors.? It can be resource intensive, especially if you want a reasonably fast reaction time.? And specifying bad behavior is somewhat of an art. One option is to provide a website for registering your users, then allow them access via some convenient token.??? A Captcha will help to reduce fraudulent registrations.? E.g., if they have a static IP address, register that.? Or provide a VPN (with just your web or email server as an endpoint).? Or use X.509 client authentication? - note that you can use this with your mailserver.? For this purpose, you want your own CA for X.509.? You can revoke abused tokens.? If your community is small (or willing to pay), you can look at hardware tokens, such a yubikey. That will work if you have a reasonably sized community - and people really want to use your service.? However, if you're trying to attract people who don't know if they are interested, the cost of connecting with you would probably turn many away. It's a balancing act, and your business (community, etc) needs will determine what is best for you. Note that I'm not exclusively endorsing any of the products/services mentioned - there are alternatives, and you need to evaluate what each offers against your needs. Unfortunately, there's no universal answer. Good luck. From jon.clements at umass.edu Thu Apr 19 13:09:57 2018 From: jon.clements at umass.edu (Jon Clements) Date: Thu, 19 Apr 2018 13:09:57 -0400 Subject: [Mailman-Users] Fwd: Mailman not sending mail??? In-Reply-To: References: <1414_1524143038_5AD893BE_1414_6671_1_ccf2878c-64b6-a8c1-d4a0-2b4fa6675383@uni-konstanz.de> Message-ID: This seems to be happening in smtp-failure log when the problem started ...??? And the failure log appears to be ongoing... JC Apr 05 01:07:46 2018 (210) delivery to john at dickiebros.com failed with code -1: Connection unexpectedly closed Apr 05 01:07:46 2018 (210) delivery to cortland9 at icloud.com failed with code -1: Connection unexpectedly closed Apr 05 01:07:46 2018 (210) delivery to tlr1 at cornell.edu failed with code -1: Connection unexpectedly closed: [Errno 54] Connection reset by peer Apr 05 01:07:46 2018 (210) delivery to finazzo at chibardun.net failed with code -1: Connection unexpectedly closed Apr 05 01:07:46 2018 (210) delivery to gpullano at greatamericanpublish.com failed with code -1: Connection unexpectedly closed Apr 05 01:07:46 2018 (210) delivery to dayna at applelanefarm.com failed with code -1: Connection unexpectedly closed Apr 05 01:07:46 2018 (210) delivery to rollinsorchards at gmail.com failed with code -1: Connection unexpectedly closed Apr 05 01:07:46 2018 (210) delivery to pippmsxx at yahoo.com failed with code -1: Connection unexpectedly closed Apr 05 01:07:46 2018 (210) delivery to gbmount at gmail.com failed with code -1: Connection unexpectedly closed Apr 05 01:07:46 2018 (210) delivery to sincuk at psis.umass.edu failed with code -1: Connection unexpectedly closed: [Errno 54] Connection reset by peer Apr 05 01:07:46 2018 (210) delivery to thelittlefarmer at rocketmail.com failed with code -1: Connection unexpectedly closed Apr 05 01:07:46 2018 (210) delivery to marion.murray at usu.edu failed with code -1: Connection unexpectedly closed: [Errno 54] Connection reset by peer Apr 05 01:07:46 2018 (210) delivery to appleman.maurice at gmail.com failed with code -1: Connection unexpectedly closed Apr 05 01:07:46 2018 (210) delivery to bdsparks at meistermedia.com failed with code -1: Connection unexpectedly closed Apr 05 01:07:46 2018 (210) delivery to wincowgill at mac.com failed with code -1: Connection unexpectedly closed Apr 05 01:07:46 2018 (210) delivery to george.brinson at nf.sympatico.ca failed with code -1: Connection unexpectedly closed: [Errno 54] Connection reset by peer On Thu, Apr 19, 2018 at 9:03 AM, mailman-admin < mailman-admin at uni-konstanz.de> wrote: > Am 19.04.2018 um 13:16 schrieb Jon Clements: > > Hi, my Mailman seems to have stopped sending mail. Posts to a list(s) are > > not going anywhere, nor for example, subscribe notifications, etc. are > not > > going out. It seems to be receiving mail as pending moderator requests > are > > there. Webadmin seems to be working normally (except for not sending mail > > when requested). Where do I start looking? Sorry, not very technically > > proficient, it's a wonder I got it (Mailman v. 2.1.20) up and running a > > couple years ago on Mac OS 10.10.5 (using Server v. 5.0.15) but it has > been > > running great since then (until now). I will say there was quite a bit of > > spam coming in before the problem started, and I stopped Mail service (on > > Server) for about a day to hopefully make it go away, and then the > Mailman > > sending problem cropped up after I re-started Mail service (I think). > > Local/normal Mail accounts seem to be working fine. > > > > Appreciate any help you can give me... > > > > I had a died OutgoingRunner a couple of times. > Check if all mailman processes are still running. > If not restart mailman. > > > Kind regards > Christian Mack > > > ------------------------------------------------------ > Mailman-Users mailing list Mailman-Users at python.org > https://mail.python.org/mailman/listinfo/mailman-users > Mailman FAQ: http://wiki.list.org/x/AgA3 > Security Policy: http://wiki.list.org/x/QIA9 > Searchable Archives: http://www.mail-archive.com/ma > ilman-users%40python.org/ > Unsubscribe: https://mail.python.org/mailman/options/mailman-users/jon. > clements%40umass.edu > -- JMCEXTMAN (aka Jon Clements) 413.478.7219 Verizon 413.378.3068 Project Fi UMass Cold Spring Orchard 393 Sabin Street Belchertown, MA 01007 http://umassfruit.com -- Jon Clements aka 'Mr Honeycrisp' University of Massachusetts Amherst Extension UMass Cold Spring Orchard 393 Sabin St. Belchertown, MA 01007 413-478-7219 Verizon 413-378-3068 Project Fi umassfruit.com From incoming-pythonlists at rjl.com Thu Apr 19 13:08:40 2018 From: incoming-pythonlists at rjl.com (Natu) Date: Thu, 19 Apr 2018 10:08:40 -0700 Subject: [Mailman-Users] Brute force attacks on mailman web ui In-Reply-To: References: <00cfd5c3-c318-6714-214e-25c6b539a6b6@uni-konstanz.de> <20180416113840.GA10061@gsp.org> <104b4fe8-ffa5-5b47-95f7-10de69a6e5ec@yahoo.com> <20180417142013.GA19660@gsp.org> Message-ID: On 04/17/2018 08:27 PM, Carl Zwanzig wrote: > On 4/17/2018 7:20 AM, Rich Kulawiec wrote: >> I stood up a new server last fall with *no* valid ssh access and logged >> about 750,000 attempts in a month.?? Similar patterns. > > There's a reason I don't put sshd on port 22; moving it elsewhere and > blackhole-ing 22 cut the auth log tremendously. > > ( If you have no users logging in remotely or if users are technical enough, consider using fwknop for ssh and other services.? I also use openvpn or openvpn with fwknop to access the vpn.? I've found fwknop to be rock solid, and I've never had even a single attack on services that use fwknop.? http://www.cipherdyne.org/fwknop/ Natu From fmouse at fmp.com Thu Apr 19 13:33:12 2018 From: fmouse at fmp.com (Lindsay Haisley) Date: Thu, 19 Apr 2018 12:33:12 -0500 Subject: [Mailman-Users] Brute force attacks on mailman web ui In-Reply-To: References: <00cfd5c3-c318-6714-214e-25c6b539a6b6@uni-konstanz.de> <20180416113840.GA10061@gsp.org> <104b4fe8-ffa5-5b47-95f7-10de69a6e5ec@yahoo.com> <20180417142013.GA19660@gsp.org> Message-ID: <1524159192.1052.13.camel@fmp.com> On Thu, 2018-04-19 at 10:08 -0700, Natu wrote: > On 04/17/2018 08:27 PM, Carl Zwanzig wrote: > > On 4/17/2018 7:20 AM, Rich Kulawiec wrote: > >> I stood up a new server last fall with *no* valid ssh access and logged > >> about 750,000 attempts in a month.?? Similar patterns. > > > > There's a reason I don't put sshd on port 22; moving it elsewhere and > > blackhole-ing 22 cut the auth log tremendously. > > > > ( > > If you have no users logging in remotely or if users are technical > enough, consider using fwknop for ssh and other services.? I also use > openvpn or openvpn with fwknop to access the vpn.? I've found fwknop to > be rock solid, and I've never had even a single attack on services that > use fwknop.? http://www.cipherdyne.org/fwknop/ Once again, do yourself a favor and check out fail2ban. It's in use on my company's server and works wonders on stopping brute force attacks on ALL services affected. -- Lindsay Haisley | "The first casualty when FMP Computer Services | war comes is truth." 512-259-1190 | http://www.fmp.com | -- Hiram W Johnson From ddewey at cyberthugs.com Thu Apr 19 13:53:20 2018 From: ddewey at cyberthugs.com (ddewey at cyberthugs.com) Date: Thu, 19 Apr 2018 13:53:20 -0400 Subject: [Mailman-Users] Brute force attacks on mailman web ui In-Reply-To: <20180416113840.GA10061@gsp.org> References: <00cfd5c3-c318-6714-214e-25c6b539a6b6@uni-konstanz.de> <20180416113840.GA10061@gsp.org> Message-ID: <20180419175320.GA29768@bianchi.cyberthugs.com> Quoting Rich Kulawiec (rsk at gsp.org): > On Mon, Apr 16, 2018 at 09:08:43AM +0200, mailman-admin wrote: > > Brute Force attempts can only be mitigated by e.g. fail2ban. > > Nope. There are other ways. > > Brute force attacks can be pre-emptively blocked by nearly everyone > operating a Mailman instance. (I say "nearly" for specific reasons > that will become clear below.) Great writeup. This is exactly how I've had my firewall configured for some time, with the drop/edrop and country block lists. I monitor for breakin attempts and add country blocks as needed... it's interesting that this seems to be somewhat cyclical in my experience, in that one month 80% of my brute force attacks are from Turkey, then the next month it shifts to Brazil (as examples, but I have both of these countries blocked now). From kaneko at yamachu-tokachi.co.jp Thu Apr 19 06:17:52 2018 From: kaneko at yamachu-tokachi.co.jp (kaneko at yamachu-tokachi.co.jp) Date: Thu, 19 Apr 2018 19:17:52 +0900 Subject: [Mailman-Users] 'from' header at delivered email from inside / outside organization Message-ID: <015001d3d7c7$ad1ab9f0$07502dd0$@yamachu-tokachi.co.jp> Hello Mailman experts, I created a mailing list (i.e. abc at ml.abc.co.jp ) with mailman in our organization. Does anyone know how to configure mailman to achieve expected behavior? Expected behavior: a. When a sender is outside of our organization (abc.co.jp), the received mail should show original sender's email address at 'from' header. b. When a sender is inside of our organization and receiver is outside of our organization, the received mail should show ML address (abc at ml.abc.co.jp ) at 'from' header. I want to know sender's email address if it's from outside our organization but do not want disclose employees' email address in our organization when they send emails to outside our organization. Thanks, Toshi Kaneko From gtaylor at tnetconsulting.net Thu Apr 19 15:16:20 2018 From: gtaylor at tnetconsulting.net (Grant Taylor) Date: Thu, 19 Apr 2018 13:16:20 -0600 Subject: [Mailman-Users] 'from' header at delivered email from inside / outside organization In-Reply-To: <015001d3d7c7$ad1ab9f0$07502dd0$@yamachu-tokachi.co.jp> References: <015001d3d7c7$ad1ab9f0$07502dd0$@yamachu-tokachi.co.jp> Message-ID: <15a31066-8910-9d8e-318d-db7b872b5487@spamtrap.tnetconsulting.net> On 04/19/2018 04:17 AM, kaneko at yamachu-tokachi.co.jp wrote: > Hello Mailman experts, I'm not an expert, but I've got questions. > I created a mailing list (i.e. abc at ml.abc.co.jp) with mailman in our > organization. I don't think it matters, but I want to make sure I'm not assuming anything incorrectly. It looks like your Mailman list is configured in a sub-domain of your copmpanies main domain. I'm assuming that means that email for the mailing list is routed to the server hosting Mailman independent of your main email server. Correct? > Does anyone know how to configure mailman to achieve expected behavior? > > Expected behavior: > > a. When a sender is outside of our organization (abc.co.jp), the > received mail should show original sender's email address at 'from' > header. Okay. No from header modification. > b. When a sender is inside of our organization and receiver is > outside of our organization, the received mail should show ML address > (abc at ml.abc.co.jp) at 'from' header. What from address should receivers inside the organization see from senders also inside the organization? Do they need to see the real internal from header? Or is it okay that they see the mailing list address? If it's okay that internal recipients see the mailing list instead of the actaul internal senders, then the problem can likely be simplified to be "internal senders should have their from address rewritten to the mailing list address." > I want to know sender's email address if it's from outside our > organization but do not want disclose employees' email address in our > organization when they send emails to outside our organization. This almost sounds like an MTA masquerading issue. I don't know if mailman can help with this or not. Especially if it's supposed to conditionally happen based on the sender and the recipient. -- Grant. . . . unix || die From mark at msapiro.net Thu Apr 19 19:03:45 2018 From: mark at msapiro.net (Mark Sapiro) Date: Thu, 19 Apr 2018 16:03:45 -0700 Subject: [Mailman-Users] Fwd: Mailman not sending mail??? In-Reply-To: References: <1414_1524143038_5AD893BE_1414_6671_1_ccf2878c-64b6-a8c1-d4a0-2b4fa6675383@uni-konstanz.de> Message-ID: On 04/19/2018 10:09 AM, Jon Clements wrote: > This seems to be happening in smtp-failure log when the problem started > ...??? And the failure log appears to be ongoing... > > JC > > Apr 05 01:07:46 2018 (210) delivery to john at dickiebros.com failed with code > -1: Connection unexpectedly closed ... Sometimes Mailman and the underlying Python smtplib module can get out of sync with each other or the MTA. Usually, this can be fixed by restarting Mailman. -- Mark Sapiro The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan From mark at msapiro.net Thu Apr 19 19:13:26 2018 From: mark at msapiro.net (Mark Sapiro) Date: Thu, 19 Apr 2018 16:13:26 -0700 Subject: [Mailman-Users] 'from' header at delivered email from inside / outside organization In-Reply-To: <015001d3d7c7$ad1ab9f0$07502dd0$@yamachu-tokachi.co.jp> References: <015001d3d7c7$ad1ab9f0$07502dd0$@yamachu-tokachi.co.jp> Message-ID: On 04/19/2018 03:17 AM, kaneko at yamachu-tokachi.co.jp wrote: > > Does anyone know how to configure mailman to achieve expected behavior? > > > > Expected behavior: > > a. When a sender is outside of our organization (abc.co.jp), the > received mail should show original sender's email address at 'from' header. > b. When a sender is inside of our organization and receiver is outside > of our organization, the received mail should show ML address > (abc at ml.abc.co.jp ) at 'from' header. This can't be done in Mailman without source code modification. You would have to modify Mailman's SMTPDirect.py module to always set deliveryfunc = verpdeliver and then modify verpdeliver itself to look at the sender and recipient and if the sender is local and the recipient remote, replace the From: The mods would not be difficult nor extensive, but this can't be done with configuration alone. -- Mark Sapiro The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan From turnbull.stephen.fw at u.tsukuba.ac.jp Thu Apr 19 23:32:30 2018 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Fri, 20 Apr 2018 12:32:30 +0900 Subject: [Mailman-Users] 'from' header at delivered email from inside / outside organization In-Reply-To: References: <015001d3d7c7$ad1ab9f0$07502dd0$@yamachu-tokachi.co.jp> Message-ID: <23257.24398.974423.619881@turnbull.sk.tsukuba.ac.jp> Mark Sapiro writes: > On 04/19/2018 03:17 AM, kaneko at yamachu-tokachi.co.jp wrote: > > Expected behavior: > > > > a. When a sender is outside of our organization (abc.co.jp), the > > received mail should show original sender's email address at 'from' header. > > b. When a sender is inside of our organization and receiver is outside > > of our organization, the received mail should show ML address > > (abc at ml.abc.co.jp ) at 'from' header. > > > This can't be done in Mailman without source code modification. > The mods would not be difficult nor extensive, but this can't be done > with configuration alone. Besides what Mark says, it looks to me like you're describing a customer relations application. "Mailman Banzai!" and all that, but Mailman is not designed as a CRM, and hard to tune to be one. If a CRM is what you want, I'm sure there are dedicated applications with more of the features you need. I'm sorry I don't have any concrete suggestions, I don't use or need CRMs. Steve From stephen at xemacs.org Thu Apr 19 23:33:32 2018 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Fri, 20 Apr 2018 12:33:32 +0900 Subject: [Mailman-Users] Brute force attacks on mailman web ui In-Reply-To: <7c6639c7-df14-0bd4-6743-17ca5b58775a@yahoo.com> References: <00cfd5c3-c318-6714-214e-25c6b539a6b6@uni-konstanz.de> <20180416113840.GA10061@gsp.org> <23256.15179.454906.332210@turnbull.sk.tsukuba.ac.jp> <7c6639c7-df14-0bd4-6743-17ca5b58775a@yahoo.com> Message-ID: <23257.24460.469807.637321@turnbull.sk.tsukuba.ac.jp> tlhackque via Mailman-Users writes: > I'm not sure what you are looking for. I'm looking for anything that will help block swaths of Chinese spammers and possibly attacks, while allowing me to do a better job of serving students vacationing at home in China than treating them the way the Chinese government does. A unicorn, or failing that, a pony. > There are a number of geolocating services that attempt to turn IP > addresses into specific locations; for example maxmind offers a series > of databases of increasing precision for increasing prices (starting > with free). I'll try their free offering. Thank you! > But the problem is that unless you know exactly where your users (and > potential users) are located, this won't help.? Do you have a list of > cities?? Streets? I can frequently get down to the street level for valid users, yes, at least after first contact. > What you probably want is to identify the specific bad actors; No, I want to identify good actors and block the rest. The problem I've had in the past is that I can't depend on static IPs because I'm dealing with people using telephones, mostly. > As previously noted, fail2ban is one reactive means of dealing with > these - it reads log files and dynamically blocks IP addresses that > generate errors.? It can be resource intensive, especially if you want a > reasonably fast reaction time.? And specifying bad behavior is somewhat > of an art. I wouldn't call it art, but a few years ago I had a 1MB .procmailrc. :-) > One option is to provide a website for registering your users, then > allow them access via some convenient token. I'm not sure what you're suggesting. That's what is being attacked here. > Or provide a VPN (with just your web or email server as an > endpoint). I believe the Chinese have outlawed VPNs, I assume they allow TLS still, though, given the size of ecommerce there. ?> Or use X.509 client authentication? - note that you can use this ?> with your mailserver. That's an interesting idea, but again my users will be mostly using phones, so I don't think this will work with mail very well, and I'm not sure how to set that up on a phone. ?> For this purpose, you want your own CA for X.509. Sure. > However, if you're trying to attract people who don't know if they > are interested, the cost of connecting with you would probably turn > many away. The prospect of graduate study outside of China seems to be a strong motivator so far. We'll see if it interests people in conforming to practices that increase my security. Interesting thoughts, anyway. Steve From tlhackque at yahoo.com Fri Apr 20 06:21:37 2018 From: tlhackque at yahoo.com (tlhackque) Date: Fri, 20 Apr 2018 06:21:37 -0400 Subject: [Mailman-Users] Brute force attacks on mailman web ui In-Reply-To: <23257.24460.469807.637321@turnbull.sk.tsukuba.ac.jp> References: <00cfd5c3-c318-6714-214e-25c6b539a6b6@uni-konstanz.de> <20180416113840.GA10061@gsp.org> <23256.15179.454906.332210@turnbull.sk.tsukuba.ac.jp> <7c6639c7-df14-0bd4-6743-17ca5b58775a@yahoo.com> <23257.24460.469807.637321@turnbull.sk.tsukuba.ac.jp> Message-ID: <82de023c-6644-1aad-0cab-8cc2d1e5b942@yahoo.com> On 19-Apr-18 23:33, Stephen J. Turnbull wrote: > tlhackque via Mailman-Users writes: > > > I'm not sure what you are looking for. > > I'm looking for anything that will help block swaths of Chinese > spammers and possibly attacks, while allowing me to do a better job of > serving students vacationing at home in China than treating them the > way the Chinese government does. A unicorn, or failing that, a pony. So you know exactly who your users are, and can pre-register them while they are not in China. Geographic IP address is the wrong hammer for this nail. Block the country, open pinholes for the users. > No, I want to identify good actors and block the rest. The problem > I've had in the past is that I can't depend on static IPs because I'm > dealing with people using telephones, mostly. GeoIP will never get you down to the level of granularity and accuracy that you want. Even if it did, people with phones move - apartment, coffee shop, etc.? > > One option is to provide a website for registering your users, then > > allow them access via some convenient token. > > I'm not sure what you're suggesting. That's what is being attacked > here. If you're seeing webserver attacks, you'll also see mailserver AUTH attacks. I assumed both. It's easier to protect a website than a mailserver.? Plus, registration is a one-time activity. And in your scenario, you can block all of China, since you can register your students while they are at your school (which presumably is not in China). So use the registration website to issue an X.509 certificate, register a hardware token, issue fwknop key - whatever you choose as your token (credential).? Then use that token to protect routine access to the mailman web ui AND mail servers. Even if you allow registration from China, you can make the registration website available at limited times.? E.g. odd days of the month from 1341 to 1417.? This drastically reduces your attack surface.? You tell your users the algorithm for when they can use it to register, or more likely, invalidate a lost key & get a new one, etc.? Change it every semester. > > Or provide a VPN (with just your web or email server as an > > endpoint). > > I believe the Chinese have outlawed VPNs, I assume they allow TLS > still, though, given the size of ecommerce there. I believe you are correct.? They may use spoofing on TLS, so unless you use some form of pinning, assume MITM.? > ?> Or use X.509 client authentication? - note that you can use this > ?> with your mailserver. > > That's an interesting idea, but again my users will be mostly using > phones, so I don't think this will work with mail very well, and I'm > not sure how to set that up on a phone. Mobile MUAs not my area of expertise, but it ought to be possible.? Both sendmail and postfix allow it to be configured. Even if you don't have a native MUA, you can provide a web-based e-mail account on your server for your users - e.g. squirrelmail, roundcube, etc.? Put client auth on that (easy); allow that server access to your e-mail server, perhaps just on localhost.? Mobile web browsers certainly support x.509 client auth.? If you go this route, you only need to open a TLS port; you don't need IMAP/POP3/SMTP/Submission.? You will still get kiddie script attempts, but TLS will block them for lack of a (valid) client certificate.? These are less common, and combined with fail2ban should work reasonably well. From your revised description, fwknop seems to be a good choice.? It's cheap, transparent, easy to setup, and should traverse the great firewall.? (No direct experience, but the technology is hard to block, especially if you overlay it on a web or mail server.)? You can use it to protect both.? Issue their keys before they go home, and you're done.? Optionally, provide some form of recovery/reissue for the "I lost my phone" case.? Or just say "don't lose your phone; if you do, you're out of luck for the rest of your trip." Providing an e-mail service as outlined above is more expensive, but provides an additional service for your users. In any case, I think we've probably exhausted the patience of mailman-users since we're off into the general problem of keeping our servers alive in the jungle... Good luck. From kaneko at yamachu-tokachi.co.jp Fri Apr 20 07:14:04 2018 From: kaneko at yamachu-tokachi.co.jp (kaneko at yamachu-tokachi.co.jp) Date: Fri, 20 Apr 2018 20:14:04 +0900 Subject: [Mailman-Users] 'from' header at delivered email from inside / outside organization In-Reply-To: <23257.24398.974423.619881@turnbull.sk.tsukuba.ac.jp> References: <015001d3d7c7$ad1ab9f0$07502dd0$@yamachu-tokachi.co.jp> <23257.24398.974423.619881@turnbull.sk.tsukuba.ac.jp> Message-ID: <002701d3d898$b1774a50$1465def0$@yamachu-tokachi.co.jp> Hi everyone, Many thanks for your answers! I understand your suggestions. We will change sender filed in our mail client. Thanks again, Toshi Kaneko -----Original Message----- From: Stephen J. Turnbull [mailto:turnbull.stephen.fw at u.tsukuba.ac.jp] Sent: Friday, April 20, 2018 12:33 PM To: Mark Sapiro Cc: mailman-users at python.org Subject: Re: [Mailman-Users] 'from' header at delivered email from inside / outside organization Mark Sapiro writes: > On 04/19/2018 03:17 AM, kaneko at yamachu-tokachi.co.jp wrote: > > Expected behavior: > > > > a. When a sender is outside of our organization (abc.co.jp), the > > received mail should show original sender's email address at 'from' header. > > b. When a sender is inside of our organization and receiver is outside > > of our organization, the received mail should show ML address > > (abc at ml.abc.co.jp ) at 'from' header. > > > This can't be done in Mailman without source code modification. > The mods would not be difficult nor extensive, but this can't be done > with configuration alone. Besides what Mark says, it looks to me like you're describing a customer relations application. "Mailman Banzai!" and all that, but Mailman is not designed as a CRM, and hard to tune to be one. If a CRM is what you want, I'm sure there are dedicated applications with more of the features you need. I'm sorry I don't have any concrete suggestions, I don't use or need CRMs. Steve From stephen at xemacs.org Sat Apr 21 12:51:58 2018 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sun, 22 Apr 2018 01:51:58 +0900 Subject: [Mailman-Users] [Suspected Spam]Re: Brute force attacks on mailman web ui In-Reply-To: <82de023c-6644-1aad-0cab-8cc2d1e5b942@yahoo.com> References: <00cfd5c3-c318-6714-214e-25c6b539a6b6@uni-konstanz.de> <20180416113840.GA10061@gsp.org> <23256.15179.454906.332210@turnbull.sk.tsukuba.ac.jp> <7c6639c7-df14-0bd4-6743-17ca5b58775a@yahoo.com> <23257.24460.469807.637321@turnbull.sk.tsukuba.ac.jp> <82de023c-6644-1aad-0cab-8cc2d1e5b942@yahoo.com> Message-ID: <23259.27694.876035.964841@turnbull.sk.tsukuba.ac.jp> tlhackque writes: > So you know exactly who your users are, and can pre-register them > while they are not in China. No. China may, or may not, block any given email provider without warning. They may need to provide a new address *from that address* (or their mother's, which I also don't know). If I can figure out how they can use X.509 auth with mail or thru the web, that will do the trick for authentication, of course. I might use fwknop to conceal authenticated services. > Geographic IP address is the wrong hammer for this nail. Yes, I understand that. Tell it to Chairman Xi, please. > GeoIP will never get you down to the level of granularity and accuracy > that you want. Sure it will. If I can block 95% of Chinese attempts to connect on SYN, that's a win. > Even if it did, people with phones move - apartment, coffee shop, > etc. Whether that means they'll be out of the geographic area allowed depends on how the provider allocates IP addresses. > And in your scenario, you can block all of China, since you can > register your students while they are at your school (which > presumably is not in China). It's not, but no, I'm not sure I can, for the reason given above. > So use the registration website to issue an X.509 certificate, > register a hardware token, issue fwknop key - whatever you choose > as your token (credential).? Then use that token to protect routine > access to the mailman web ui AND mail servers. I know how to issue such things, or can find out. What I don't know how to do is enable devices to use them, and whether I can configure once, or teach students to do it. I also wonder what Chinese immigration authorities would think of a fwknop app on an iPhone.... > Even if you don't have a native MUA, you can provide a web-based e-mail > account on your server for your users - e.g. squirrelmail, roundcube, > etc. That's a last resort. They won't use that account frequently (if at all), which really counts against it. I would want something that they use under normal circumstances that they're used to, and have some idea how to configure on a new phone, etc. > Mobile web browsers certainly support x.509 client auth. This seems like the route to go, then. Use a nonstandard port to screen out the dumbest kiddies, or maybe fwknop. > Issue their keys before they go home, and you're done.? Optionally, > provide some form of recovery/reissue for the "I lost my phone" case.? I'm not sanguine about "issue keys and you're done". There are users in the loop, and they're not terribly security-savvy. I'm not saying it's insanely difficult, but the system would be an unfamiliar one that is a SPOF. The recovery/reissue feature wouldn't be optional: the trips I'd be worrying about would be on location data-gathering trips. > In any case, I think we've probably exhausted the patience of > mailman-users since we're off into the general problem of keeping > our servers alive in the jungle... Well, the considerations of dealing with user-hostile environments like the Great Firewall are pretty special, but the jungle *is* general. I.e., it applies to Mailman servers too. I don't know anybody who runs a list who doesn't run into abuse from time to time. Thanks for the ideas and software suggestions! Steve From dmaziuk at bmrb.wisc.edu Sat Apr 21 13:14:26 2018 From: dmaziuk at bmrb.wisc.edu (Dmitri Maziuk) Date: Sat, 21 Apr 2018 12:14:26 -0500 Subject: [Mailman-Users] [Suspected Spam]Re: Brute force attacks on mailman web ui In-Reply-To: <23259.27694.876035.964841@turnbull.sk.tsukuba.ac.jp> References: <00cfd5c3-c318-6714-214e-25c6b539a6b6@uni-konstanz.de> <20180416113840.GA10061@gsp.org> <23256.15179.454906.332210@turnbull.sk.tsukuba.ac.jp> <7c6639c7-df14-0bd4-6743-17ca5b58775a@yahoo.com> <23257.24460.469807.637321@turnbull.sk.tsukuba.ac.jp> <82de023c-6644-1aad-0cab-8cc2d1e5b942@yahoo.com> <23259.27694.876035.964841@turnbull.sk.tsukuba.ac.jp> Message-ID: <380294d1-f37e-e560-8ca6-fcbc35ddbba6@bmrb.wisc.edu> On 4/21/2018 11:51 AM, Stephen J. Turnbull wrote: > I also wonder what Chinese > immigration authorities would think of a fwknop app on an iPhone.... They'll send your users to a re-education camp where they can learn about possession of restricted/illegal tech. Where they will also not have access to e-mail and won't be able to participate in your list. You can skip the middle part and just not let them subscribe to begin with. (FWIW I've recently firewalled off a bunch of Baidu IPs because their crawler was ignoring robots.txt and hitting CGI scripts that require a small-ish amount of processing... never attribute to malice that which can be adequately explained by stupidity.) Dima From heller at deepsoft.com Sat Apr 21 13:43:22 2018 From: heller at deepsoft.com (Robert Heller) Date: Sat, 21 Apr 2018 13:43:22 -0400 (EDT) Subject: [Mailman-Users] [Suspected Spam]Re: Brute force attacks on mailman web ui In-Reply-To: <380294d1-f37e-e560-8ca6-fcbc35ddbba6@bmrb.wisc.edu> References: <00cfd5c3-c318-6714-214e-25c6b539a6b6@uni-konstanz.de> <20180416113840.GA10061@gsp.org> <23256.15179.454906.332210@turnbull.sk.tsukuba.ac.jp> <7c6639c7-df14-0bd4-6743-17ca5b58775a@yahoo.com> <23257.24460.469807.637321@turnbull.sk.tsukuba.ac.jp> <82de023c-6644-1aad-0cab-8cc2d1e5b942@yahoo.com> <23259.27694.876035.964841@turnbull.sk.tsukuba.ac.jp> <380294d1-f37e-e560-8ca6-fcbc35ddbba6@bmrb.wisc.edu> Message-ID: <20180421174323.1E0CB26C2534@sharky3.deepsoft.com> At Sat, 21 Apr 2018 12:14:26 -0500 Dmitri Maziuk wrote: > > On 4/21/2018 11:51 AM, Stephen J. Turnbull wrote: > > > I also wonder what Chinese > > immigration authorities would think of a fwknop app on an iPhone.... > > They'll send your users to a re-education camp where they can learn > about possession of restricted/illegal tech. Where they will also not > have access to e-mail and won't be able to participate in your list. > > You can skip the middle part and just not let them subscribe to begin with. > > (FWIW I've recently firewalled off a bunch of Baidu IPs because their > crawler was ignoring robots.txt and hitting CGI scripts that require a > small-ish amount of processing... never attribute to malice that which > can be adequately explained by stupidity.) And *I* have a fail2ban rule just for BingBot. M$'s coders are somewhat clueless. > > Dima > ------------------------------------------------------ > Mailman-Users mailing list Mailman-Users at python.org > https://mail.python.org/mailman/listinfo/mailman-users > Mailman FAQ: http://wiki.list.org/x/AgA3 > Security Policy: http://wiki.list.org/x/QIA9 > Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ > Unsubscribe: https://mail.python.org/mailman/options/mailman-users/heller%40deepsoft.com > > -- Robert Heller -- 978-544-6933 Deepwoods Software -- Custom Software Services http://www.deepsoft.com/ -- Linux Administration Services heller at deepsoft.com -- Webhosting Services From kgordon2006 at frontier.com Sat Apr 21 13:46:01 2018 From: kgordon2006 at frontier.com (Kenneth G. Gordon) Date: Sat, 21 Apr 2018 10:46:01 -0700 Subject: [Mailman-Users] Filtering Chinese spam. Message-ID: <5ADB78D9.8031.F3A4565@kgordon2006.frontier.com> Hello: One of my mailman mailing lists has been suddenly afflicted with tons of Chinese spam. After reading an on-line discussion of this problem and how to handle it from a thread posted to this list in July of 2016, here: https://mail.python.org/pipermail/mailman-users/2016-July/080993.html I have modified my settings in Privacy Options/Spam Filters thusly: ^Subject: =?utf-8?B? ^Subject:.*\?{4,} from: .*@qq.com from: .*ebdoor.com from: .*126.com from: .*139.com from: .*136.com from: .*163.com from: .*193.com The "from" expressions have worked quite well, and using those I have cut the Chinese spam by over 90%, but I am still receiving periodic posts from other Chinese senders and I cannot keep adding more items to the "from" set for obvious reasons. Instead, as someone mentioned in that thread of July 2016, I find that all Chinese posts include the expression " =?utf-8?B?" following the word "Subject". Accordingly, I have included that in the first line of my Spam FIlter. However, it does not seem to be working as desired as I am still receiving Chinese spam containing that expression. >From that thread of July 2016, I have, today, added that second line above. Can someone here tell me what I have NOT done correctly with that first line which makes it NOT work as desired? Kenneth G. Gordon (Age 75) A Tired Old SYSAD. --- This email has been checked for viruses by Avast antivirus software. https://www.avast.com/antivirus From mark at msapiro.net Sun Apr 22 10:55:44 2018 From: mark at msapiro.net (Mark Sapiro) Date: Sun, 22 Apr 2018 07:55:44 -0700 Subject: [Mailman-Users] Filtering Chinese spam. In-Reply-To: <5ADB78D9.8031.F3A4565@kgordon2006.frontier.com> References: <5ADB78D9.8031.F3A4565@kgordon2006.frontier.com> Message-ID: <610440c3-89f7-0df4-6aee-46dcad402262@msapiro.net> On 04/21/2018 10:46 AM, Kenneth G. Gordon wrote: > > I have modified my settings in Privacy Options/Spam Filters thusly: > > ^Subject: =?utf-8?B? > ^Subject:.*\?{4,} > from: .*@qq.com > from: .*ebdoor.com > from: .*126.com > from: .*139.com > from: .*136.com > from: .*163.com > from: .*193.com I'm a bit confused as to where you are putting these. The ones starting with ^Subject: look like regexps that would be in header_filter_rules and the ones starting with from: would also work in header_filter_rules but look more like bounce_matching_headers entries. It is best to use header_filter_rules for everything as it gives more control over what to do with a matching message. In that case, it would be better if the from ones were like ^from: .*@qq.com to avoid a match on something like Subject: message from: someone @qq.com Also, if those from: lines are in bounce_matching_headers, it only results in those messages being held and presumably the same end result is obtained with Privacy options... -> Sender filters -> generic_nonmember_action = Hold > Instead, as someone mentioned in that thread of July 2016, I find that all Chinese posts > include the expression " =?utf-8?B?" following the word "Subject". > > Accordingly, I have included that in the first line of my Spam FIlter. However, it does not > seem to be working as desired as I am still receiving Chinese spam containing that > expression. You need to understand regular expressions . '?' has a special meaning in a regexp. You need ^Subject: =\?utf-8\?B\? to match something with a Subject beginning with =?utf-8?B? -- Mark Sapiro The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan From heller at deepsoft.com Sun Apr 22 11:22:02 2018 From: heller at deepsoft.com (Robert Heller) Date: Sun, 22 Apr 2018 11:22:02 -0400 (EDT) Subject: [Mailman-Users] Filtering Chinese spam. In-Reply-To: <5ADB78D9.8031.F3A4565@kgordon2006.frontier.com> References: <5ADB78D9.8031.F3A4565@kgordon2006.frontier.com> Message-ID: <20180422152203.31A3B26C264B@sharky3.deepsoft.com> At Sat, 21 Apr 2018 10:46:01 -0700 "Kenneth G. Gordon" wrote: > > Hello: > > One of my mailman mailing lists has been suddenly afflicted with tons of Chinese spam. > > After reading an on-line discussion of this problem and how to handle it from a thread > posted to this list in July of 2016, here: > > https://mail.python.org/pipermail/mailman-users/2016-July/080993.html > > I have modified my settings in Privacy Options/Spam Filters thusly: > > ^Subject: =?utf-8?B? > ^Subject:.*\?{4,} > from: .*@qq.com If you have access to the SMTP server itself, getting qq.com blocked at that point will help even more than blocking it in Mailman. > from: .*ebdoor.com > from: .*126.com > from: .*139.com > from: .*136.com > from: .*163.com > from: .*193.com > > The "from" expressions have worked quite well, and using those I have cut the Chinese > spam by over 90%, but I am still receiving periodic posts from other Chinese senders and I > cannot keep adding more items to the "from" set for obvious reasons. > > Instead, as someone mentioned in that thread of July 2016, I find that all Chinese posts > include the expression " =?utf-8?B?" following the word "Subject". > > Accordingly, I have included that in the first line of my Spam FIlter. However, it does not > seem to be working as desired as I am still receiving Chinese spam containing that > expression. > > >From that thread of July 2016, I have, today, added that second line above. > > Can someone here tell me what I have NOT done correctly with that first line which makes it > NOT work as desired? > > Kenneth G. Gordon (Age 75) > A Tired Old SYSAD. > > > > --- > This email has been checked for viruses by Avast antivirus software. > https://www.avast.com/antivirus > > ------------------------------------------------------ > Mailman-Users mailing list Mailman-Users at python.org > https://mail.python.org/mailman/listinfo/mailman-users > Mailman FAQ: http://wiki.list.org/x/AgA3 > Security Policy: http://wiki.list.org/x/QIA9 > Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ > Unsubscribe: https://mail.python.org/mailman/options/mailman-users/heller%40deepsoft.com > > -- Robert Heller -- 978-544-6933 Deepwoods Software -- Custom Software Services http://www.deepsoft.com/ -- Linux Administration Services heller at deepsoft.com -- Webhosting Services From turnbull.stephen.fw at u.tsukuba.ac.jp Sun Apr 22 11:54:50 2018 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Mon, 23 Apr 2018 00:54:50 +0900 Subject: [Mailman-Users] Filtering Chinese spam. In-Reply-To: <5ADB78D9.8031.F3A4565@kgordon2006.frontier.com> References: <5ADB78D9.8031.F3A4565@kgordon2006.frontier.com> Message-ID: <23260.45130.239011.312648@turnbull.sk.tsukuba.ac.jp> Kenneth G. Gordon writes: > One of my mailman mailing lists has been suddenly afflicted with > tons of Chinese spam. If you have access to the firewall in your mail system, or are friendly with its admin, the most efficient way to handle this is in the firewall by dropping all traffic from China, as described in a thread about "Brute force attacks" starting about a week ago on this list. That is fairly risky these days; you never know when one of your members is going to visit China and send mail from a friend's address. As Robert Heller suggests, the next most efficient thing is to drop all traffic from China in the MTA (mail server software) using similar techniques. More to my taste than either would be to install a programmable spam- checker like SpamAssassin or SpamBayes, and bump the cost of rules that catch Chinese spam if necessary. This is far more effective and efficient than doing it in Mailman, and if your system has mailboxes other than those for Mailman, they will also be protected. > I have modified my settings in Privacy Options/Spam Filters thusly: > > ^Subject: =?utf-8?B? > ^Subject:.*\?{4,} As Mark points out, the first is not going to work. You need to "escape" the question marks as in the second expression: ^Subject: =\?utf-8\?B\? Both of these filters are risky. The first is likely to catch any email whose subject starts with an emoji, smart quotes, or any other exotic characters such as math symbols or a complex smiley (such as table flipping or the 7-character shrug). The second will catch any mail with a subject containing 4 or more question marks in a row. Of course these may be desirable if you're running a mailing list for junior highschool students ;-), but such subjects are reasonably common among educated adults as well in my experience. How large that risk is, and whether or not to take it, is up to you and your subscribers, of course. Finally, a viable strategy would be to use these filters for now and explore the more capable methods at your leisure. Regards, Steve From kgordon2006 at frontier.com Sun Apr 22 20:50:02 2018 From: kgordon2006 at frontier.com (Kenneth G. Gordon) Date: Sun, 22 Apr 2018 17:50:02 -0700 Subject: [Mailman-Users] Filtering Chinese spam. In-Reply-To: <610440c3-89f7-0df4-6aee-46dcad402262@msapiro.net> References: <5ADB78D9.8031.F3A4565@kgordon2006.frontier.com>, <610440c3-89f7-0df4-6aee-46dcad402262@msapiro.net> Message-ID: <5ADD2DBA.27244.C60675@kgordon2006.frontier.com> On 22 Apr 2018 at 7:55, Mark Sapiro wrote: > On 04/21/2018 10:46 AM, Kenneth G. Gordon wrote: > > > > I have modified my settings in Privacy Options/Spam Filters thusly: > > > > ^Subject: =?utf-8?B? > > ^Subject:.*\?{4,} > > from: .*@qq.com > > from: .*ebdoor.com > > from: .*126.com > > from: .*139.com > > from: .*136.com > > from: .*163.com > > from: .*193.com > > > I'm a bit confused as to where you are putting these. The ones starting > with ^Subject: look like regexps that would be in header_filter_rules Correct. > and the ones starting with from: would also work in header_filter_rules > but look more like bounce_matching_headers entries. Yes. I took the first one from my bounce_matching_header rules, then added some others. Those worked to eliminate all the spam from qq.com so I continued with the others. > It is best to use header_filter_rules for everything as it gives more > control over what to do with a matching message. In that case, it would > be better if the from ones were like > > ^from: .*@qq.com OK. Thanks, I am not particularly good at operating mailman (!!) yet, but am trying to learn. > to avoid a match on something like > > Subject: message from: someone @qq.com My lists would never have anyone from such an address. They are very specifically limited to a very few folks who are interested in a rather arcane part of radio operations. Therefore, if I block everything from qq.com, that would be just fine. > Also, if those from: lines are in bounce_matching_headers, it only > results in those messages being held and presumably the same end result > is obtained with Privacy options... -> Sender filters -> > generic_nonmember_action = Hold Thanks again, I want ALL traffic from (for instance) qq.com to go into a big black hole. Ken Gordon --- This email has been checked for viruses by Avast antivirus software. https://www.avast.com/antivirus From kgordon2006 at frontier.com Sun Apr 22 20:55:59 2018 From: kgordon2006 at frontier.com (Kenneth G. Gordon) Date: Sun, 22 Apr 2018 17:55:59 -0700 Subject: [Mailman-Users] Filtering Chinese spam. In-Reply-To: <20180422152203.31A3B26C264B@sharky3.deepsoft.com> References: <5ADB78D9.8031.F3A4565@kgordon2006.frontier.com>, <20180422152203.31A3B26C264B@sharky3.deepsoft.com> Message-ID: <5ADD2F1F.17924.CB79F7@kgordon2006.frontier.com> On 22 Apr 2018 at 11:22, Robert Heller wrote: > If you have access to the SMTP server itself, getting qq.com blocked at that > point will help even more than blocking it in Mailman. I kinda don't think I do: I believe that is frontier.com. ;-) And Frontier uses Yahoo's mail server :-( Boo, hiss... Thanks, Ken Gordon --- This email has been checked for viruses by Avast antivirus software. https://www.avast.com/antivirus From kgordon2006 at frontier.com Sun Apr 22 21:20:51 2018 From: kgordon2006 at frontier.com (Kenneth G. Gordon) Date: Sun, 22 Apr 2018 18:20:51 -0700 Subject: [Mailman-Users] Filtering Chinese spam. In-Reply-To: <23260.45130.239011.312648@turnbull.sk.tsukuba.ac.jp> References: <5ADB78D9.8031.F3A4565@kgordon2006.frontier.com>, <23260.45130.239011.312648@turnbull.sk.tsukuba.ac.jp> Message-ID: <5ADD34F3.28128.E23FDC@kgordon2006.frontier.com> Thank you very much for the information below, Mr. Turnbull. Your last line pretty much says it all. I have much to learn yet, and what I am doing now, with the corrections I have received here, will serve in the meantime. Kenneth Gordon On 23 Apr 2018 at 0:54, Stephen J. Turnbull wrote: > Kenneth G. Gordon writes: > > > One of my mailman mailing lists has been suddenly afflicted with > > tons of Chinese spam. > > If you have access to the firewall in your mail system, or are > friendly with its admin, the most efficient way to handle this is in > the firewall by dropping all traffic from China, as described in a > thread about "Brute force attacks" starting about a week ago on this > list. That is fairly risky these days; you never know when one of > your members is going to visit China and send mail from a friend's > address. As Robert Heller suggests, the next most efficient thing is > to drop all traffic from China in the MTA (mail server software) using > similar techniques. > > More to my taste than either would be to install a programmable spam- > checker like SpamAssassin or SpamBayes, and bump the cost of rules > that catch Chinese spam if necessary. This is far more effective and > efficient than doing it in Mailman, and if your system has mailboxes > other than those for Mailman, they will also be protected. > > > I have modified my settings in Privacy Options/Spam Filters thusly: > > > > ^Subject: =?utf-8?B? > > ^Subject:.*\?{4,} > > As Mark points out, the first is not going to work. You need to > "escape" the question marks as in the second expression: > > ^Subject: =\?utf-8\?B\? > > Both of these filters are risky. The first is likely to catch any > email whose subject starts with an emoji, smart quotes, or any other > exotic characters such as math symbols or a complex smiley (such as > table flipping or the 7-character shrug). The second will catch any > mail with a subject containing 4 or more question marks in a row. Of > course these may be desirable if you're running a mailing list for > junior highschool students ;-), but such subjects are reasonably > common among educated adults as well in my experience. How large that > risk is, and whether or not to take it, is up to you and your > subscribers, of course. > > Finally, a viable strategy would be to use these filters for now and > explore the more capable methods at your leisure. > > Regards, > Steve > --- This email has been checked for viruses by Avast antivirus software. https://www.avast.com/antivirus From mark at msapiro.net Sun Apr 22 21:49:06 2018 From: mark at msapiro.net (Mark Sapiro) Date: Sun, 22 Apr 2018 18:49:06 -0700 Subject: [Mailman-Users] Filtering Chinese spam. In-Reply-To: <5ADD2DBA.27244.C60675@kgordon2006.frontier.com> References: <5ADB78D9.8031.F3A4565@kgordon2006.frontier.com> <610440c3-89f7-0df4-6aee-46dcad402262@msapiro.net> <5ADD2DBA.27244.C60675@kgordon2006.frontier.com> Message-ID: <62158992-487e-3c39-01d3-9de20e374a16@msapiro.net> On 04/22/2018 05:50 PM, Kenneth G. Gordon wrote: > On 22 Apr 2018 at 7:55, Mark Sapiro wrote: > >> Also, if those from: lines are in bounce_matching_headers, it only >> results in those messages being held and presumably the same end result >> is obtained with Privacy options... -> Sender filters -> >> generic_nonmember_action = Hold > > Thanks again, I want ALL traffic from (for instance) qq.com to go into a big black hole. So you have two choices. If you want all posts from non-list members to just disappear, set Privacy options... -> Sender filters -> generic_nonmember_action = Discard and you don't have to deal with any 'from' spam filters and all non-member posts will just disappear. If, on the other hand, you want some non-member posts to be held (or rejected or even accepted) while dealing with the Chinese spam separately, set Privacy options... -> Sender filters -> generic_nonmember_action = Hold or Reject or Accept as desired and create one header_filter_rule with a Discard action and a list of regexps like ^Subject: =\?utf-8\?B\? ^Subject:.*\?{4,} ^from: .*@qq\.com ^from: .*ebdoor\.com ^from: .*126\.com ^from: .*139\.com ^from: .*136\.com ^from: .*163\.com ^from: .*193\.com although note Steve's caveats about the first two. Note that you could handle all the 3-digit ones with a single ^From: .*@\d{3}\.com regexp which will match a From: header with anything followed by @ and 3 digits and .com. Also note, these tests are case insensitive so From vs. from is irrelevant and note the \ escape of the . so it matches a literal . and not any character -- Mark Sapiro The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan From domiman at gmail.com Mon Apr 23 08:06:07 2018 From: domiman at gmail.com (Dominik Reichardt) Date: Mon, 23 Apr 2018 14:06:07 +0200 Subject: [Mailman-Users] SourceForge Mailing list "moderator request(s) waiting Message-ID: <31E5DAFF-2EDD-4DB4-AAB9-E4B6A4C34473@gmail.com> Hi all, I recently became an admin of a Mailing list on Sourceforge and I recently changed something in our settings and now I get the daily digests of spam to our list. (This was when Sourceforge had the data migration hiccup not that long ago) Subject: 4 Exult-general moderator request(s) waiting Body: The Exult-general at lists.sourceforge.net mailing list has 4 request(s) waiting for your consideration at: https://lists.sourceforge.net/lists/admindb/exult-general Please attend to this at your earliest convenience. This notice of pending requests, if any, will be sent out daily. Pending posts:? (some posts with Cause: Post by non-member to a members-only list) And it?s driving me nuts because I can?t find the setting I changed. Just maybe it was always like this and I receive them now because I added my email address to admin/moderator panel. I?m not sure, I read through so much that day. I also didn?t find this in the mailman docs but there is so much to read and I might have missed it. Can you help me? Thanks a lot! Dominik From mark at msapiro.net Tue Apr 24 11:29:23 2018 From: mark at msapiro.net (Mark Sapiro) Date: Tue, 24 Apr 2018 08:29:23 -0700 Subject: [Mailman-Users] SourceForge Mailing list "moderator request(s) waiting In-Reply-To: <31E5DAFF-2EDD-4DB4-AAB9-E4B6A4C34473@gmail.com> References: <31E5DAFF-2EDD-4DB4-AAB9-E4B6A4C34473@gmail.com> Message-ID: <63714f22-505c-36bd-7df1-4d65770a827c@msapiro.net> On 04/23/2018 05:06 AM, Dominik Reichardt wrote: > > I recently became an admin of a Mailing list on Sourceforge and I recently changed something in our settings and now I get the daily digests of spam to our list. (This was when Sourceforge had the data migration hiccup not that long ago) > > Subject: 4 Exult-general moderator request(s) waiting > > Body: > The Exult-general at lists.sourceforge.net mailing list has 4 request(s) > waiting for your consideration at: > > https://lists.sourceforge.net/lists/admindb/exult-general > > Please attend to this at your earliest convenience. This notice of > pending requests, if any, will be sent out daily. > > > > Pending posts:? > (some posts with Cause: Post by non-member to a members-only list) > > And it?s driving me nuts because I can?t find the setting I changed. Just maybe it was always like this and I receive them now because I added my email address to admin/moderator panel. I?m not sure, I read through so much that day. Yes. This notice is sent daily to the list's owners and moderators. If you just added your email address as a moderator, that's why you started seeing these. The question is are these the same 4 requests daily? if so, just deal with them via the admindb UI. Once they have been accepted/rejected/discarded and are gone from the admindb UI, you will no longer be notified. If on the other hand, you are dealing with the posts and the next day, there are new ones, this is expected behavior. As long as there are outstanding requests, you will receive a notice about them. -- Mark Sapiro The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan From rafi at brijnet.org Wed Apr 25 17:02:52 2018 From: rafi at brijnet.org (Rafael Salasnik) Date: Wed, 25 Apr 2018 22:02:52 +0100 Subject: [Mailman-Users] Moving mailman to new server, mails not getting through Message-ID: <427abd42-9f6a-5267-4ed4-a1c25cd61bd9@brijnet.org> Hi I've moving my mailman lists to a new server. I've setup a list, added myself as a subscriber, received a welcome message, tried to post a test message but it disappears into the ether. I've looked at the settings & they seem fine. Any pointers on where I should be looking to see where the problem is? Thanks --- This email has been checked for viruses by AVG. http://www.avg.com From bo at bogusville.us Wed Apr 25 12:24:50 2018 From: bo at bogusville.us (Bo Gusman) Date: Wed, 25 Apr 2018 09:24:50 -0700 Subject: [Mailman-Users] Cannot discard pending requests Message-ID: Mailman v2.1.18 on Debian Jessie. Been running mailman (much older version) for years on an old CentOS box. Recently moved to a new machine as above, and migrated a list - no problems there. The list itself seems to be working just fine - mail flowing as expected. However, I've got a couple of pending moderator requests that simply refuse to go away. I can't Accept them, Reject them, or Discard them. Selecting Discard and clicking Submit All Data, the page simply refreshes itself and the items remain unaffected. I do not see any error message, cannot find any indication in any system log that there is a problem of any sort (it's possible I'm not looking in the right place?) I've run check_perms and there are 10 symlinks that are reported with bad groups (has root, expected list) but those cannot be changed, apparently. How can I fix this? ??? Bo From mark at msapiro.net Thu Apr 26 00:38:01 2018 From: mark at msapiro.net (Mark Sapiro) Date: Wed, 25 Apr 2018 21:38:01 -0700 Subject: [Mailman-Users] Moving mailman to new server, mails not getting through In-Reply-To: <427abd42-9f6a-5267-4ed4-a1c25cd61bd9@brijnet.org> References: <427abd42-9f6a-5267-4ed4-a1c25cd61bd9@brijnet.org> Message-ID: <3cfd9259-adb5-5091-790d-7fd73a1e912c@msapiro.net> On 04/25/2018 02:02 PM, Rafael Salasnik wrote: > Hi > > I've moving my mailman lists to a new server. I've setup a list, added > myself as a subscriber, received a welcome message, tried to post a test > message but it disappears into the ether. I've looked at the settings & > they seem fine. Any pointers on where I should be looking to see where > the problem is? Logs. Mailman's logs and the MTA logs. I'm guessing, but my guess is you subscribed yourself via the web UI and not email and you have overlooked something in the MTA setup so mail to mailman is not being delivered. Also, see . -- Mark Sapiro The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan From mark at msapiro.net Thu Apr 26 00:41:38 2018 From: mark at msapiro.net (Mark Sapiro) Date: Wed, 25 Apr 2018 21:41:38 -0700 Subject: [Mailman-Users] Cannot discard pending requests In-Reply-To: References: Message-ID: <7feb215c-2506-c923-f3a0-78f5f5fc0ff0@msapiro.net> On 04/25/2018 09:24 AM, Bo Gusman wrote: > > The list itself seems to be working just fine - mail flowing as > expected. However, I've got a couple of pending moderator requests that > simply refuse to go away. I can't Accept them, Reject them, or Discard > them. Selecting Discard and clicking Submit All Data, the page simply > refreshes itself and the items remain unaffected. I suspect this is one of the issues described at . Most likely the reason is your site redirects http to https and you haven't done steps 2 and 3 at . -- Mark Sapiro The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan From rafi at brijnet.org Sun Apr 29 09:00:15 2018 From: rafi at brijnet.org (Rafael Salasnik) Date: Sun, 29 Apr 2018 14:00:15 +0100 Subject: [Mailman-Users] Giving access to moderators using hostgator Message-ID: Hi Any idea of how to give moderators access to mailing list administration in hostgator? Thanks --- This email has been checked for viruses by AVG. http://www.avg.com From mark at msapiro.net Sun Apr 29 11:13:36 2018 From: mark at msapiro.net (Mark Sapiro) Date: Sun, 29 Apr 2018 08:13:36 -0700 Subject: [Mailman-Users] Giving access to moderators using hostgator In-Reply-To: References: Message-ID: On 04/29/2018 06:00 AM, Rafael Salasnik wrote: > Hi > > Any idea of how to give moderators access to mailing list administration > in hostgator? Give them the list admin password. This can be confusing as there are 4 independent list attributes involved. There are the 'owner' and 'moderator' lists of addresses. These only control where various messages and notices are sent. I.e. mail to LISTNAME-owner and most Mailman generated notices go to both the owner and the moderator addresses, but a few notices go to owner addresses only. There are also administrator and moderator passwords. From the point of view of the web UI and administrator is one who visits the 'admin' pages and authenticates with the administrator password. A moderator is one who visits the 'admindb' pages and authenticates with either the administrator password or the moderator password. I.e. authentication with the moderator password allows access only to the 'admindb' pages. Authentication with the administrator password allows access to both the 'admin' and the 'admindb' pages. These access rights have nothing to do with any addresses in the 'owner' and 'moderator' lists. All the above is 'generic' GNU Mailman. If Hostgator has made changes to their Mailman, things could be different there. I do see that Hostgator appears to use cPanel and I know that cPanel Mailman has some changes in the area of web authentication, but even so I suspect that if you want to give someone access to the 'admin' UI, they have to authenticate as an admin. -- Mark Sapiro The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan From enseikou at gmail.com Sun Apr 29 17:13:51 2018 From: enseikou at gmail.com (Rubeno =?ISO-8859-1?Q?Fern=E1ndez?=) Date: Sun, 29 Apr 2018 23:13:51 +0200 Subject: [Mailman-Users] Wrong language sent to subscribers Message-ID: <8477589.BCy93dUXjQ@fractal> Hello all, I'm using Mailman 2.1.20 on Ubuntu 16, which I installed from their repositories; Python is 2.7. I've set up DEFAULT_SERVER_LANGUAGE to 'ca' in my /etc/mailman/mm_cfg.py file. All the language files can be found at /usr/ share/mailman/ca. However, subscription confirmation notices and acknowledgements are sent only in English. Strangely enough, the subjects are in Catalan, though. Is this a bug? How can I fix it? Rubeno From mark at msapiro.net Mon Apr 30 11:23:08 2018 From: mark at msapiro.net (Mark Sapiro) Date: Mon, 30 Apr 2018 08:23:08 -0700 Subject: [Mailman-Users] Wrong language sent to subscribers In-Reply-To: <8477589.BCy93dUXjQ@fractal> References: <8477589.BCy93dUXjQ@fractal> Message-ID: On 04/29/2018 02:13 PM, Rubeno Fern?ndez wrote: > Hello all, > I'm using Mailman 2.1.20 on Ubuntu 16, which I installed from their > repositories; Python is 2.7. I've set up DEFAULT_SERVER_LANGUAGE to 'ca' in > my /etc/mailman/mm_cfg.py file. All the language files can be found at /usr/ > share/mailman/ca. I'm unclear on how this is configured in your installation. In a 'normal' Debian/Ubuntu package, everything is in /var/lib/mailman/. In particular, the various templates such as subscribeack.txt are in /var/lib/mailman/templates/LC/, possibly overridden by list specific, domain specific, or site specific templates. See for info on the location of these. > However, subscription confirmation notices and acknowledgements are sent only > in English. Strangely enough, the subjects are in Catalan, though. > Is this a bug? How can I fix it? You need to verify that the templates in /var/lib/mailman/templates/ca/ exist and are appropriate and if there are any overrides in /var/lib/mailman/lists/LISTNAME/ca/, /var/lib/mailman/templates/DOMAIN/ca/ or /var/lib/mailman/templates/site/ca/ that they are appropriate. -- Mark Sapiro The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan