periodic shunts might have been undetected bad bounce_matching_headers
Hi mailman-users@python.org I hope I may have solved some postings being shunted on a few of my lists, necessitating occasional cd /usr/local/mailman; unshunt qfiles/shunt
I discovered 3 lists had missing '\n', shown with cd /usr/local/mailman/lists; config_list -o - ListName bounce_matching_headers = """# Lines that *start* with a '#' are comments.
Fixed with cd /usr/local/mailman/lists ; config_list -o ~/tmp/ListName ListName vi ~/tmp/ListName ; config_list -i ~/tmp/ListName ListName & I hope the shunting will stop.
I did autopsy snapshots of /usr/local/mailman before & after last unshunt, if anyone feels like telling me what to look for.
Unfortunately the corrupt bounce_matching_headers lines in configuration were undetected by these automatic checks cd /usr/local/etc/rc.d; ./mailman stop ; ./mailman start cd /usr/local/mailman/lists ; ../bin/check_db -a cd /usr/local/mailman/lists ; ../bin/check_perms
The errors probably derive from a fumbled edit by me using an older mailman. I'll do more visual inspection on my most troublesome list.
Cheers, Julian
Julian H. Stacey, Computer Consultant, BSD Linux Unix Systems Engineer, Munich Reply below, Prefix '> '. Plain text, No .doc, base64, HTML, quoted-printable. http://berklix.eu/brexit/#3,500,000_stolen_votes_inc_700,000_in_EU
On 08/25/2017 06:29 AM, Julian H. Stacey wrote:
Hi mailman-users@python.org I hope I may have solved some postings being shunted on a few of my lists, necessitating occasional cd /usr/local/mailman; unshunt qfiles/shunt
I discovered 3 lists had missing '\n', shown with cd /usr/local/mailman/lists; config_list -o - ListName bounce_matching_headers = """# Lines that *start* with a '#' are comments.
What is the rest of this up to the terminating """?
Fixed with cd /usr/local/mailman/lists ; config_list -o ~/tmp/ListName ListName vi ~/tmp/ListName ; config_list -i ~/tmp/ListName ListName & I hope the shunting will stop.
I did autopsy snapshots of /usr/local/mailman before & after last unshunt, if anyone feels like telling me what to look for.
What is the traceback from the 'shunt' message in Mailman's error log?
Unfortunately the corrupt bounce_matching_headers lines in configuration were undetected by these automatic checks cd /usr/local/etc/rc.d; ./mailman stop ; ./mailman start cd /usr/local/mailman/lists ; ../bin/check_db -a cd /usr/local/mailman/lists ; ../bin/check_perms
If bounce_matching_headers contains an invalid line, there should be entries in Mailman's 'config' log like
'bad bounce_matching_header line: LISTNAME LINE
or
bad regexp in bounce_matching_header line: LISTNAME REGEXP (cause: ERROR)
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Hi Mark, thanks for reply. PS you'r doing a great job answering so many of us :-)
Mark Sapiro wrote:
On 08/25/2017 06:29 AM, Julian H. Stacey wrote:
Hi mailman-users@python.org I hope I may have solved some postings being shunted on a few of my lists, necessitating occasional cd /usr/local/mailman; unshunt qfiles/shunt
I discovered 3 lists had missing '\n', shown with cd /usr/local/mailman/lists; config_list -o - ListName bounce_matching_headers = """# Lines that *start* with a '#' are comments.
What is the rest of this up to the terminating """?
Ah yes, I see what you mean, Its not a '' line, but a """ block: Now I have: # Note that leading whitespace is trimmed from the regexp. This can be # circumvented in a number of ways, e.g. by escaping or bracketing it. bounce_matching_headers = """ # Lines that *start* with a '#' are comments. to: friend@public.com message-id: relay.comanche.denmark.eu from: list@listme.com from: .*@uplinkpro.com"""
## Bounce optionsbefore it was # Note that leading whitespace is trimmed from the regexp. This can be # circumvented in a number of ways, e.g. by escaping or bracketing it. bounce_matching_headers = """# Lines that *start* with a '#' are comments. to: friend@public.com message-id: relay.comanche.denmark.eu from: list@listme.com from: .*@uplinkpro.com"""
## Bounce optionsSo fixed, but I guess that wasnt the cause of the shunts. Especially as I rcall a shunt on one of the other lists that didnt have that damage.
Fixed with cd /usr/local/mailman/lists ; config_list -o ~/tmp/ListName ListName vi ~/tmp/ListName ; config_list -i ~/tmp/ListName ListName & I hope the shunting will stop.
I did autopsy snapshots of /usr/local/mailman before & after last unshunt, if anyone feels like telling me what to look for.
What is the traceback from the 'shunt' message in Mailman's error log?
/usr/local/mailman/logs/error (Aside: I also spotted a few other unassociated errors such as Hostile listname: & fixed pages that pointed to non existant local Mailman list names)
All my remaining errors now seem to be the same sort:
NameError: global name 'syslog' is not defined
Aug 23 13:37:16 2017 (82399) SHUNTING: 1503488012.441064+20509f89ec545c49966a4f1d747c804ea1df5852
Aug 24 20:56:44 2017 (18093) Uncaught runner exception: global name 'syslog' is not defined
Aug 24 20:56:44 2017 (18093) Traceback (most recent call last):
File "/usr/local/mailman/Mailman/Queue/Runner.py", line 119, in _oneloop
self._onefile(msg, msgdata)
File "/usr/local/mailman/Mailman/Queue/Runner.py", line 190, in _onefile
keepqueued = self._dispose(mlist, msg, msgdata)
File "/usr/local/mailman/Mailman/Queue/IncomingRunner.py", line 130, in _dispose
more = self._dopipeline(mlist, msg, msgdata, pipeline)
File "/usr/local/mailman/Mailman/Queue/IncomingRunner.py", line 153, in _dopipeline
sys.modules[modname].process(mlist, msg, msgdata)
File "/usr/local/mailman/Mailman/Handlers/SpamDetect.py", line 136, in process
syslog('vette',
NameError: global name 'syslog' is not defined
Aug 24 20:56:44 2017 (18093) SHUNTING: 1503601003.748515+0146d084002ddd93cacadc41656c7632efc83301/usr/local/mailman/Mailman/Handlers/SpamDetect.py line 136 syslog('vette', '%s: Automatically Moderated %s for verbose postings.', mlist.real_name, sender)
Line 34: from Mailman import mm_cfg
Mailman/mm_cfg.py has no syslog
So I'm searching for syslog in my FreeBSD /usr/ports/mail/mailman
Got a bit lost maybe this might be FreeBSD specific,
Added maintainer of the FreeBSD ports wrapper in case of a hint ?Unfortunately the corrupt bounce_matching_headers lines in configuration were undetected by these automatic checks cd /usr/local/etc/rc.d; ./mailman stop ; ./mailman start cd /usr/local/mailman/lists ; ../bin/check_db -a cd /usr/local/mailman/lists ; ../bin/check_perms
If bounce_matching_headers contains an invalid line, there should be entries in Mailman's 'config' log like
'bad bounce_matching_header line: LISTNAME LINE
or
bad regexp in bounce_matching_header line: LISTNAME REGEXP (cause: ERROR)
cd /usr/local/mailman/logs grep bad * grep header * No other errors, apart from admin: No such list "admin":
Thanks, Julian
Julian H. Stacey, Computer Consultant, BSD Linux Unix Systems Engineer, Munich Reply below, Prefix '> '. Plain text, No .doc, base64, HTML, quoted-printable. http://berklix.eu/brexit/#3,500,000_stolen_votes_inc_700,000_in_EU
On 08/29/2017 03:59 PM, Julian H. Stacey wrote:
Now I have: # Note that leading whitespace is trimmed from the regexp. This can be # circumvented in a number of ways, e.g. by escaping or bracketing it. bounce_matching_headers = """ # Lines that *start* with a '#' are comments. to: friend@public.com message-id: relay.comanche.denmark.eu from: list@listme.com from: .*@uplinkpro.com"""
## Bounce options before it was # Note that leading whitespace is trimmed from the regexp. This can be # circumvented in a number of ways, e.g. by escaping or bracketing it. bounce_matching_headers = """# Lines that *start* with a '#' are comments. to: friend@public.com message-id: relay.comanche.denmark.eu from: list@listme.com from: .*@uplinkpro.com"""
Actually, the only difference between the two is "Now" has a leading blank line which won't make any difference as blank lines are ignored.
What is the traceback from the 'shunt' message in Mailman's error log?
/usr/local/mailman/logs/error (Aside: I also spotted a few other unassociated errors such as Hostile listname: & fixed pages that pointed to non existant local Mailman list names)
All my remaining errors now seem to be the same sort:
NameError: global name 'syslog' is not defined
Aug 23 13:37:16 2017 (82399) SHUNTING: 1503488012.441064+20509f89ec545c49966a4f1d747c804ea1df5852 Aug 24 20:56:44 2017 (18093) Uncaught runner exception: global name 'syslog' is not defined Aug 24 20:56:44 2017 (18093) Traceback (most recent call last): File "/usr/local/mailman/Mailman/Queue/Runner.py", line 119, in _oneloop self._onefile(msg, msgdata) File "/usr/local/mailman/Mailman/Queue/Runner.py", line 190, in _onefile keepqueued = self._dispose(mlist, msg, msgdata) File "/usr/local/mailman/Mailman/Queue/IncomingRunner.py", line 130, in _dispose more = self._dopipeline(mlist, msg, msgdata, pipeline) File "/usr/local/mailman/Mailman/Queue/IncomingRunner.py", line 153, in _dopipeline sys.modules[modname].process(mlist, msg, msgdata) File "/usr/local/mailman/Mailman/Handlers/SpamDetect.py", line 136, in process syslog('vette', NameError: global name 'syslog' is not defined
Aug 24 20:56:44 2017 (18093) SHUNTING: 1503601003.748515+0146d084002ddd93cacadc41656c7632efc83301
/usr/local/mailman/Mailman/Handlers/SpamDetect.py line 136 syslog('vette', '%s: Automatically Moderated %s for verbose postings.', mlist.real_name, sender)
Line 34: from Mailman import mm_cfg
That and the next few lines should be
from Mailman import mm_cfg from Mailman import Errors from Mailman import i18n from Mailman import Utils from Mailman.Handlers.Hold import hold_for_approval from Mailman.Logging.Syslog import syslog
It appears the 'from Mailman.Logging.Syslog import syslog' line is missing.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Mark Sapiro wrote:
On 08/29/2017 03:59 PM, Julian H. Stacey wrote:
Now I have: # Note that leading whitespace is trimmed from the regexp. This can be # circumvented in a number of ways, e.g. by escaping or bracketing it. bounce_matching_headers = """ # Lines that *start* with a '#' are comments. to: friend@public.com message-id: relay.comanche.denmark.eu from: list@listme.com from: .*@uplinkpro.com"""
## Bounce options before it was # Note that leading whitespace is trimmed from the regexp. This can be # circumvented in a number of ways, e.g. by escaping or bracketing it. bounce_matching_headers = """# Lines that *start* with a '#' are comments. to: friend@public.com message-id: relay.comanche.denmark.eu from: list@listme.com from: .*@uplinkpro.com"""
Actually, the only difference between the two is "Now" has a leading blank line which won't make any difference as blank lines are ignored.
OK,
What is the traceback from the 'shunt' message in Mailman's error log?
/usr/local/mailman/logs/error (Aside: I also spotted a few other unassociated errors such as Hostile listname: & fixed pages that pointed to non existant local Mailman list names)
All my remaining errors now seem to be the same sort:
NameError: global name 'syslog' is not defined
Aug 23 13:37:16 2017 (82399) SHUNTING: 1503488012.441064+20509f89ec545c49966a4f1d747c804ea1df5852 Aug 24 20:56:44 2017 (18093) Uncaught runner exception: global name 'syslog' is not defined Aug 24 20:56:44 2017 (18093) Traceback (most recent call last): File "/usr/local/mailman/Mailman/Queue/Runner.py", line 119, in _oneloop self._onefile(msg, msgdata) File "/usr/local/mailman/Mailman/Queue/Runner.py", line 190, in _onefile keepqueued = self._dispose(mlist, msg, msgdata) File "/usr/local/mailman/Mailman/Queue/IncomingRunner.py", line 130, in _dispose more = self._dopipeline(mlist, msg, msgdata, pipeline) File "/usr/local/mailman/Mailman/Queue/IncomingRunner.py", line 153, in _dopipeline sys.modules[modname].process(mlist, msg, msgdata) File "/usr/local/mailman/Mailman/Handlers/SpamDetect.py", line 136, in process syslog('vette', NameError: global name 'syslog' is not defined
Aug 24 20:56:44 2017 (18093) SHUNTING: 1503601003.748515+0146d084002ddd93cacadc41656c7632efc83301
/usr/local/mailman/Mailman/Handlers/SpamDetect.py line 136 syslog('vette', '%s: Automatically Moderated %s for verbose postings.', mlist.real_name, sender)
Line 34: from Mailman import mm_cfg
That and the next few lines should be
from Mailman import mm_cfg from Mailman import Errors from Mailman import i18n from Mailman import Utils from Mailman.Handlers.Hold import hold_for_approval from Mailman.Logging.Syslog import syslog
It appears the 'from Mailman.Logging.Syslog import syslog' line is missing.
Yes, it's missing. Well analysed without seeing it, Thanks !
So I patched just that line in.
Mailman 2.1.22 source misses that line. 2.1.23 2.1.24 have it.
2.1.23/NEWS has "Bug Fixes and other patches" ... SpamDetect.py I'd best upgrade from 2.1.22 to 2.1.24 when I can monitor traffic after.
I checked in case anything else missing, but all seems OK :-) I compared 2.1.22 & 2.1.24, all Handlers/*.py , & SpamDetect.py is the only one varying between versions with Mailman.Logging.Syslog . Various other Handlers/*.py do not have Mailman.Logging.Syslog in 2.1.22 & 2.1.24, but this quick check passes: cd /usr/local/mailman/Mailman/Handlers grep -l syslog *.py > ~/tmp/1 grep -l Mailman.Logging.Syslog *.py > ~/tmp/2 cmp ~/tmp/1 ~/tmp/2 (But I don't know Python, & not considered other directories, includes, libs).
Thanks Mark!
Cheers, Julian
Julian H. Stacey, Computer Consultant, BSD Linux Unix Systems Engineer, Munich Reply below, Prefix '> '. Plain text, No .doc, base64, HTML, quoted-printable. http://berklix.eu/brexit/#3,500,000_stolen_votes_inc_700,000_in_EU
On 08/30/2017 08:36 AM, Julian H. Stacey wrote:
Mark Sapiro wrote:
It appears the 'from Mailman.Logging.Syslog import syslog' line is missing.
Yes, it's missing. Well analysed without seeing it, Thanks !
So I patched just that line in.
Mailman 2.1.22 source misses that line. 2.1.23 2.1.24 have it.
This is a bug introduced in 2.1.21, never noticed and inadvertently fixed in 2.1.23. Presumably, it was never noticed before now because it was introduced with the member verbosity feature and only triggered if Privacy options... -> Sender filters -> member_verbosity_threshold is set greater than the default 0 and a "verbose" member is actually set to moderated.
2.1.23/NEWS has "Bug Fixes and other patches" ... SpamDetect.py
But that is in the 2.1.16 section of NEWS.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Mark Sapiro wrote:
On 08/30/2017 08:36 AM, Julian H. Stacey wrote:
Mark Sapiro wrote:
It appears the 'from Mailman.Logging.Syslog import syslog' line is missing.
Yes, it's missing. Well analysed without seeing it, Thanks !
So I patched just that line in.
Mailman 2.1.22 source misses that line. 2.1.23 2.1.24 have it.This is a bug introduced in 2.1.21, never noticed and inadvertently fixed in 2.1.23. Presumably, it was never noticed before now because it was introduced with the member verbosity feature and only triggered if Privacy options... -> Sender filters -> member_verbosity_threshold is set greater than the default 0 and a "verbose" member is actually set to moderated.
2.1.24 has defaults 0 300. My most troublesome list had max 5/week = member_verbosity_threshold: 5 member_verbosity_interval: 604800 which may have occasionaly triggered correctly, but at least one of the shunted people was someone who very rarely posts so at least that shunt was unintended by traffic criteria.
2.1.23/NEWS has "Bug Fixes and other patches" ... SpamDetect.py
But that is in the 2.1.16 section of NEWS.
Oh yes, I hadn't noticed it was in an old section. So this is a till now undocumented old fix :-)
Cheers, Julian
Julian H. Stacey, Computer Consultant, BSD Linux Unix Systems Engineer, Munich Reply below, Prefix '> '. Plain text, No .doc, base64, HTML, quoted-printable. http://berklix.eu/brexit/#3,500,000_stolen_votes_inc_700,000_in_EU
On 08/30/2017 05:13 PM, Julian H. Stacey wrote:
2.1.24 has defaults 0 300. My most troublesome list had max 5/week = member_verbosity_threshold: 5 member_verbosity_interval: 604800 which may have occasionaly triggered correctly, but at least one of the shunted people was someone who very rarely posts so at least that shunt was unintended by traffic criteria.
With the bug, the shunting will occur whenever a member's moderate flag is set for verbosity. The exception occurs after the flag is set in the attempt to log the fact.
Note that with the above settings on the list flagging doesn't require
5 posts to this list within a week, but >5 posts combined to all lists with member_verbosity_threshold > 0.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Mark Sapiro wrote:
On 08/30/2017 05:13 PM, Julian H. Stacey wrote:
2.1.24 has defaults 0 300. My most troublesome list had max 5/week = member_verbosity_threshold: 5 member_verbosity_interval: 604800 which may have occasionaly triggered correctly, but at least one of the shunted people was someone who very rarely posts so at least that shunt was unintended by traffic criteria.
With the bug, the shunting will occur whenever a member's moderate flag is set for verbosity. The exception occurs after the flag is set in the attempt to log the fact.
Note that with the above settings on the list flagging doesn't require
5 posts to this list within a week, but >5 posts combined to all lists with member_verbosity_threshold > 0.
I hadn't realised that. Explains why I saw so many shunts. I've reverted my lists to defaults. Thanks.
Cheers, Julian
Julian H. Stacey, Computer Consultant, BSD Linux Unix Systems Engineer, Munich Reply below, Prefix '> '. Plain text, No .doc, base64, HTML, quoted-printable. http://berklix.eu/brexit/#3,500,000_stolen_votes_inc_700,000_in_EU
participants (2)
-
Julian H. Stacey -
Mark Sapiro