Filtering Chinese spam.
Hello:
One of my mailman mailing lists has been suddenly afflicted with tons of Chinese spam.
After reading an on-line discussion of this problem and how to handle it from a thread posted to this list in July of 2016, here:
https://mail.python.org/pipermail/mailman-users/2016-July/080993.html
I have modified my settings in Privacy Options/Spam Filters thusly:
^Subject: =?utf-8?B? ^Subject:.*\?{4,} from: .*@qq.com from: .*ebdoor.com from: .*126.com from: .*139.com from: .*136.com from: .*163.com from: .*193.com
The "from" expressions have worked quite well, and using those I have cut the Chinese spam by over 90%, but I am still receiving periodic posts from other Chinese senders and I cannot keep adding more items to the "from" set for obvious reasons.
Instead, as someone mentioned in that thread of July 2016, I find that all Chinese posts include the expression " =?utf-8?B?" following the word "Subject".
Accordingly, I have included that in the first line of my Spam FIlter. However, it does not seem to be working as desired as I am still receiving Chinese spam containing that expression.
From that thread of July 2016, I have, today, added that second line above.
Can someone here tell me what I have NOT done correctly with that first line which makes it NOT work as desired?
Kenneth G. Gordon (Age 75) A Tired Old SYSAD.
This email has been checked for viruses by Avast antivirus software. https://www.avast.com/antivirus
On 04/21/2018 10:46 AM, Kenneth G. Gordon wrote:
I have modified my settings in Privacy Options/Spam Filters thusly:
^Subject: =?utf-8?B? ^Subject:.*\?{4,} from: .*@qq.com from: .*ebdoor.com from: .*126.com from: .*139.com from: .*136.com from: .*163.com from: .*193.com
I'm a bit confused as to where you are putting these. The ones starting with ^Subject: look like regexps that would be in header_filter_rules and the ones starting with from: would also work in header_filter_rules but look more like bounce_matching_headers entries.
It is best to use header_filter_rules for everything as it gives more control over what to do with a matching message. In that case, it would be better if the from ones were like
^from: .*@qq.com
to avoid a match on something like
Subject: message from: someone @qq.com
Also, if those from: lines are in bounce_matching_headers, it only results in those messages being held and presumably the same end result is obtained with Privacy options... -> Sender filters -> generic_nonmember_action = Hold
Instead, as someone mentioned in that thread of July 2016, I find that all Chinese posts include the expression " =?utf-8?B?" following the word "Subject".
Accordingly, I have included that in the first line of my Spam FIlter. However, it does not seem to be working as desired as I am still receiving Chinese spam containing that expression.
You need to understand regular expressions https://docs.python.org/2/library/re.html.
'?' has a special meaning in a regexp. You need
^Subject: =\?utf-8\?B\?
to match something with a Subject beginning with =?utf-8?B?
-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 22 Apr 2018 at 7:55, Mark Sapiro wrote:
On 04/21/2018 10:46 AM, Kenneth G. Gordon wrote:
I have modified my settings in Privacy Options/Spam Filters thusly:
^Subject: =?utf-8?B? ^Subject:.*\?{4,} from: .*@qq.com from: .*ebdoor.com from: .*126.com from: .*139.com from: .*136.com from: .*163.com from: .*193.com
I'm a bit confused as to where you are putting these. The ones starting with ^Subject: look like regexps that would be in header_filter_rules
Correct.
and the ones starting with from: would also work in header_filter_rules but look more like bounce_matching_headers entries.
Yes. I took the first one from my bounce_matching_header rules, then added some others. Those worked to eliminate all the spam from qq.com so I continued with the others.
It is best to use header_filter_rules for everything as it gives more control over what to do with a matching message. In that case, it would be better if the from ones were like
^from: .*@qq.com
OK. Thanks, I am not particularly good at operating mailman (!!) yet, but am trying to learn.
to avoid a match on something like
Subject: message from: someone @qq.com
My lists would never have anyone from such an address. They are very specifically limited to a very few folks who are interested in a rather arcane part of radio operations. Therefore, if I block everything from qq.com, that would be just fine.
Also, if those from: lines are in bounce_matching_headers, it only results in those messages being held and presumably the same end result is obtained with Privacy options... -> Sender filters -> generic_nonmember_action = Hold
Thanks again, I want ALL traffic from (for instance) qq.com to go into a big black hole.
Ken Gordon
This email has been checked for viruses by Avast antivirus software. https://www.avast.com/antivirus
On 04/22/2018 05:50 PM, Kenneth G. Gordon wrote:
On 22 Apr 2018 at 7:55, Mark Sapiro wrote:
Also, if those from: lines are in bounce_matching_headers, it only results in those messages being held and presumably the same end result is obtained with Privacy options... -> Sender filters -> generic_nonmember_action = Hold
Thanks again, I want ALL traffic from (for instance) qq.com to go into a big black hole.
So you have two choices. If you want all posts from non-list members to just disappear, set Privacy options... -> Sender filters -> generic_nonmember_action = Discard and you don't have to deal with any 'from' spam filters and all non-member posts will just disappear.
If, on the other hand, you want some non-member posts to be held (or rejected or even accepted) while dealing with the Chinese spam separately, set Privacy options... -> Sender filters -> generic_nonmember_action = Hold or Reject or Accept as desired and create one header_filter_rule with a Discard action and a list of regexps like
^Subject: =\?utf-8\?B\? ^Subject:.*\?{4,} ^from: .*@qq\.com ^from: .*ebdoor\.com ^from: .*126\.com ^from: .*139\.com ^from: .*136\.com ^from: .*163\.com ^from: .*193\.com
although note Steve's caveats about the first two.
Note that you could handle all the 3-digit ones with a single
^From: .*@\d{3}\.com
regexp which will match a From: header with anything followed by @ and 3 digits and .com.
Also note, these tests are case insensitive so From vs. from is irrelevant and note the \ escape of the . so it matches a literal . and not any character
-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
At Sat, 21 Apr 2018 10:46:01 -0700 "Kenneth G. Gordon" kgordon2006@frontier.com wrote:
Hello:
One of my mailman mailing lists has been suddenly afflicted with tons of Chinese spam.
After reading an on-line discussion of this problem and how to handle it from a thread posted to this list in July of 2016, here:
https://mail.python.org/pipermail/mailman-users/2016-July/080993.html
I have modified my settings in Privacy Options/Spam Filters thusly:
^Subject: =?utf-8?B? ^Subject:.*\?{4,} from: .*@qq.com
If you have access to the SMTP server itself, getting qq.com blocked at that point will help even more than blocking it in Mailman.
from: .*ebdoor.com from: .*126.com from: .*139.com from: .*136.com from: .*163.com from: .*193.com
The "from" expressions have worked quite well, and using those I have cut the Chinese spam by over 90%, but I am still receiving periodic posts from other Chinese senders and I cannot keep adding more items to the "from" set for obvious reasons.
Instead, as someone mentioned in that thread of July 2016, I find that all Chinese posts include the expression " =?utf-8?B?" following the word "Subject".
Accordingly, I have included that in the first line of my Spam FIlter. However, it does not seem to be working as desired as I am still receiving Chinese spam containing that expression.
From that thread of July 2016, I have, today, added that second line above.
Can someone here tell me what I have NOT done correctly with that first line which makes it NOT work as desired?
Kenneth G. Gordon (Age 75) A Tired Old SYSAD.
This email has been checked for viruses by Avast antivirus software. https://www.avast.com/antivirus
Mailman-Users mailing list Mailman-Users@python.org https://mail.python.org/mailman/listinfo/mailman-users Mailman FAQ: http://wiki.list.org/x/AgA3 Security Policy: http://wiki.list.org/x/QIA9 Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/ Unsubscribe: https://mail.python.org/mailman/options/mailman-users/heller%40deepsoft.com
-- Robert Heller -- 978-544-6933 Deepwoods Software -- Custom Software Services http://www.deepsoft.com/ -- Linux Administration Services heller@deepsoft.com -- Webhosting Services
On 22 Apr 2018 at 11:22, Robert Heller wrote:
If you have access to the SMTP server itself, getting qq.com blocked at that point will help even more than blocking it in Mailman.
I kinda don't think I do: I believe that is frontier.com. ;-)
And Frontier uses Yahoo's mail server :-(
Boo, hiss...
Thanks,
Ken Gordon
This email has been checked for viruses by Avast antivirus software. https://www.avast.com/antivirus
Kenneth G. Gordon writes:
One of my mailman mailing lists has been suddenly afflicted with tons of Chinese spam.
If you have access to the firewall in your mail system, or are friendly with its admin, the most efficient way to handle this is in the firewall by dropping all traffic from China, as described in a thread about "Brute force attacks" starting about a week ago on this list. That is fairly risky these days; you never know when one of your members is going to visit China and send mail from a friend's address. As Robert Heller suggests, the next most efficient thing is to drop all traffic from China in the MTA (mail server software) using similar techniques.
More to my taste than either would be to install a programmable spam- checker like SpamAssassin or SpamBayes, and bump the cost of rules that catch Chinese spam if necessary. This is far more effective and efficient than doing it in Mailman, and if your system has mailboxes other than those for Mailman, they will also be protected.
I have modified my settings in Privacy Options/Spam Filters thusly:
^Subject: =?utf-8?B? ^Subject:.*\?{4,}
As Mark points out, the first is not going to work. You need to "escape" the question marks as in the second expression:
^Subject: =\?utf-8\?B\?Both of these filters are risky. The first is likely to catch any email whose subject starts with an emoji, smart quotes, or any other exotic characters such as math symbols or a complex smiley (such as table flipping or the 7-character shrug). The second will catch any mail with a subject containing 4 or more question marks in a row. Of course these may be desirable if you're running a mailing list for junior highschool students ;-), but such subjects are reasonably common among educated adults as well in my experience. How large that risk is, and whether or not to take it, is up to you and your subscribers, of course.
Finally, a viable strategy would be to use these filters for now and explore the more capable methods at your leisure.
Regards, Steve
Thank you very much for the information below, Mr. Turnbull. Your last line pretty much says it all. I have much to learn yet, and what I am doing now, with the corrections I have received here, will serve in the meantime.
Kenneth Gordon
On 23 Apr 2018 at 0:54, Stephen J. Turnbull wrote:
Kenneth G. Gordon writes:
One of my mailman mailing lists has been suddenly afflicted with tons of Chinese spam.
If you have access to the firewall in your mail system, or are friendly with its admin, the most efficient way to handle this is in the firewall by dropping all traffic from China, as described in a thread about "Brute force attacks" starting about a week ago on this list. That is fairly risky these days; you never know when one of your members is going to visit China and send mail from a friend's address. As Robert Heller suggests, the next most efficient thing is to drop all traffic from China in the MTA (mail server software) using similar techniques.
More to my taste than either would be to install a programmable spam- checker like SpamAssassin or SpamBayes, and bump the cost of rules that catch Chinese spam if necessary. This is far more effective and efficient than doing it in Mailman, and if your system has mailboxes other than those for Mailman, they will also be protected.
I have modified my settings in Privacy Options/Spam Filters thusly:
^Subject: =?utf-8?B? ^Subject:.*\?{4,}
As Mark points out, the first is not going to work. You need to "escape" the question marks as in the second expression:
^Subject: =\?utf-8\?B\?Both of these filters are risky. The first is likely to catch any email whose subject starts with an emoji, smart quotes, or any other exotic characters such as math symbols or a complex smiley (such as table flipping or the 7-character shrug). The second will catch any mail with a subject containing 4 or more question marks in a row. Of course these may be desirable if you're running a mailing list for junior highschool students ;-), but such subjects are reasonably common among educated adults as well in my experience. How large that risk is, and whether or not to take it, is up to you and your subscribers, of course.
Finally, a viable strategy would be to use these filters for now and explore the more capable methods at your leisure.
Regards, Steve
This email has been checked for viruses by Avast antivirus software. https://www.avast.com/antivirus
participants (4)
-
Kenneth G. Gordon -
Mark Sapiro -
Robert Heller -
Stephen J. Turnbull