Re: [Mailman-Users] Archive merge and search
On 07/11/2014 19:42, Mark Sapiro wrote:
On 11/07/2014 03:52 AM, Hal wrote:
For not allowing text/html [snip] But does the above apply to *all* archived postings, or does it only filter anything that comes in from now on?
It applies to all posts that arrive after you make those settings, buth in the archive and in the messages delivered to the list. It won't affect messages already archived or in the mylist.mbox file, even if you rebuild the archive.
That's fine, but good to know for later. So for any new messages from now on I want my list to work this way:
HTML formatted postings should be converted to plain text before reaching other members.
HTML formatted postings can retain their formatting for the archive (I believe the archive is in the HTML format anyway?), but if it only archives whatever is sent to list members I don't mind. The important thing is that members receive plain text messages.
Since many people have their email programs set by default to send in HTML these days I just want Mailman to do its filtering, then continue by sending the posting as plain text without any moderator request or alerting the sender.
I'd like to block all attachements (list members should only receive plain text files). 40kb is already set for Max_message_size (in "General options" within the list administration web interface) which seems to have worked fine (as far as I know).
Furthermore I understand that Filter_filename_extensions (in the "Content filtering" section) in addition removes any attachements based on specific filename *extensions* regardless of their file size?
I see exe, bat, cmd and a bunch of other filetypes I've never heard of (geared towards Windows/DOS users I suppose -I'm a Mac user) are listed, but I suppose I could block .zip and those pesky .vcf/.vcard and "winmail.dat" files the same way. When such extensions are encountered, are they just removed from the messages while the message posting itself is passed on to list members, or is the whole posting stopped for approval first?
I'm thinking out loud here, so feel free to chime in for better ideas, but I'm thinking there are two kind of attachement groups which need different actions to be taken:
Deliberate attachements: zip files, gif/jpg images etc. which a poster wants to share. The message/attachement should be stopped from reaching the list and an email sent to the poster with a "your message has been blocked. Please resend your message, this time without an attachement" type of message.
Accidental attachements: winmail.dat, .vcf or .vcard an so on. Many users don't know (as with HTML postings) that their email program is set up to send this stuff. IMHO those attachements don't have anything to do with the actual content of their postings, so Mailman should just remove the attachement(s), then pass on the rest of the message to the list.
Having said that, have I understood things correctly by setting up my "Content filtering" options as follows? (based on what you've said and what I've read here: http://wiki.list.org/pages/viewpage.action?pageId=4030684):
Edit_filter_content: YES Filter_mime_types: (left blank) Pass_mime_types: multipart message/rfc822 text/plain text/html filter_filename_ext.: exe bat cmd com pif scr vbs cpl zip dat vcf vcard pass_filename_ext.: (left blank) Collapse_alternatives: YES conv_html_to_plaintext: YES Filter_action: DISCARD
A different obfuscation for email addresses would require source code modification. I.e., there's no 'plugin' for it.
Is this a feature that could be suggested for the upcoming Mailman 3? Perhaps an optional user-configuration through the web admin interface?
Mailman 3 uses different and 'pluggable' archiving. The archiver that is bundled with MM 3 is called Hyperkitty. I'm not sure what address obfuscation it does.
Thanks. I'll look into it.
Failing that, is there a way I could have the (currently private) archive have a filter before HTTP access?
You could create your own CGI or other web process to access the archives and present them any way you want.
Being ignorant on the subject, what kind of pre-written CGI script should I try to find (i.e. "search engine to web archive gateway" or something like that?).
You previously suggested htdig (http://www.htdig.org/) with your patches for allowing my visitors to search through both the Mailman archives and my website. Assuming this is a more ready-to-use solution than the other search engines out there, are there features I will be missing out on (e.g. the ability to use CSS and Ajax for making its search results appear more in line with the rest of my website) and is it still secure? I've read that malicious code can sometimes be entered as search phrases and damage the database if the search engine isn't using "parametrized queries".
I've found other search engines (Nutch, Lucene, Solr, Tipue search, Xapian and Ajax live search) but I have no idea if they're suitable for my use and how well they work or how difficult they are to set up. Opinions from anyone are highly appreciated. Thanks.
Hal
On 11/18/2014 06:35 AM, Hal wrote:
So for any new messages from now on I want my list to work this way:
- HTML formatted postings should be converted to plain text before reaching other members.
In Mailman's Content filtering msection you want the following:
filter_content: Yes filter_mime_types: empty pass_mime_types: multipart text/plain text/html filter_filename_extensions: irrelevant, default list OK pass_filename_extensions: empty collapse_alternatives: Yes convert_html_to_plaintext: Yes filter_action: as desired, this will only apply to a message which contains no text/html or text/plain part.
- HTML formatted postings can retain their formatting for the archive (I believe the archive is in the HTML format anyway?), but if it only archives whatever is sent to list members I don't mind. The important thing is that members receive plain text messages.
What will be archived is what was delivered to list members.
- Since many people have their email programs set by default to send in HTML these days I just want Mailman to do its filtering, then continue by sending the posting as plain text without any moderator request or alerting the sender.
Settings in 1) do that.
- I'd like to block all attachements (list members should only receive plain text files). 40kb is already set for Max_message_size (in "General options" within the list administration web interface) which seems to have worked fine (as far as I know).
'attachments' is an imprecise word, but settings in 1) will do what you want.
Furthermore I understand that Filter_filename_extensions (in the "Content filtering" section) in addition removes any attachements based on specific filename *extensions* regardless of their file size?
I see exe, bat, cmd and a bunch of other filetypes I've never heard of (geared towards Windows/DOS users I suppose -I'm a Mac user) are listed, but I suppose I could block .zip and those pesky .vcf/.vcard and "winmail.dat" files the same way.
They will all be removed anyway unless they have a MIME Content-Type of text/plain or text/html which is unlikely.
When such extensions are encountered, are they just removed from the messages while the message posting itself is passed on to list members, or is the whole posting stopped for approval first?
They are just removed.
I'm thinking out loud here, so feel free to chime in for better ideas, but I'm thinking there are two kind of attachement groups which need different actions to be taken:
Deliberate attachements: zip files, gif/jpg images etc. which a poster wants to share. The message/attachement should be stopped from reaching the list and an email sent to the poster with a "your message has been blocked. Please resend your message, this time without an attachement" type of message.
Content filtering will just remove them.
Accidental attachements: winmail.dat, .vcf or .vcard an so on. Many users don't know (as with HTML postings) that their email program is set up to send this stuff. IMHO those attachements don't have anything to do with the actual content of their postings, so Mailman should just remove the attachement(s), then pass on the rest of the message to the list.
winmail.dat is really more of a 'deliberate' attachment. It is a message part with MIME type application/vnd.ms-tnef which is a Microsoft Outlook/Exchange 'transport neutral encapsulation format' way of encoding attachments.
.vcf and .vcard 'attachments' have Content-Type text/vcard or possibly application/vcard+json or application/vcard+xml.
In any case, since these do not have Content-Type text/plain or text/html, they will be removed.
Having said that, have I understood things correctly by setting up my "Content filtering" options as follows? (based on what you've said and what I've read here: http://wiki.list.org/pages/viewpage.action?pageId=4030684):
Edit_filter_content: YES Filter_mime_types: (left blank) Pass_mime_types: multipart message/rfc822 text/plain text/html filter_filename_ext.: exe bat cmd com pif scr vbs cpl zip dat vcf vcard pass_filename_ext.: (left blank) Collapse_alternatives: YES conv_html_to_plaintext: YES Filter_action: DISCARD
Maybe. The only difference between this and 1) above is message/rfc822.
If I forward a message to your list as an 'attachment', do you want to remove that forwarded message from my post or do you want to accept the plain text or possibly HTML converted to plain text parts of that forwarded message?
If you want to remove it, leave message/rfc822 out of the list, if you want to accept the result of applying contentent filtering to it, put message/rfc822 in the list.
Failing that, is there a way I could have the (currently private) archive have a filter before HTTP access?
You could create your own CGI or other web process to access the archives and present them any way you want.
Being ignorant on the subject, what kind of pre-written CGI script should I try to find (i.e. "search engine to web archive gateway" or something like that?).
I doubt very much that you'll find anything pre-written that will meet your needs.
You previously suggested htdig (http://www.htdig.org/) with your patches for allowing my visitors to search through both the Mailman archives and my website.
To be clear, htdig is a search engine that can index and search all or a portion of your web site. The patches developed by Richard Barrett and currently supported by me add a search form to the main archive table of contents page for a list and invoke htdig to do the search. This search is only of the archive of that list.
Assuming this is a more ready-to-use solution than the other search engines out there,
For a general search of your web site, probably not a good assumption.
are there features I will be missing out on (e.g. the ability to use CSS and Ajax for making its search results appear more in line with the rest of my website) and is it still secure? I've read that malicious code can sometimes be entered as search phrases and damage the database if the search engine isn't using "parametrized queries".
I don't think that malicious search phrases is an issue with htdig, but I don't know for sure that it isn't.
It probably wouldn't be too difficult to incorporate CSS into the search results pages, but I've never tried it. Ajax might be more problematic.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 18/11/2014 17:31, Mark Sapiro wrote:
On 11/18/2014 06:35 AM, Hal wrote:
So for any new messages from now on I want my list to work this way:
- HTML formatted postings should be converted to plain text before reaching other members.
In Mailman's Content filtering msection you want the following:
filter_content: Yes filter_mime_types: empty pass_mime_types: multipart text/plain text/html filter_filename_extensions: irrelevant, default list OK pass_filename_extensions: empty collapse_alternatives: Yes convert_html_to_plaintext: Yes filter_action: as desired, this will only apply to a message which contains no text/html or text/plain part.
All my settings were the same as above except "pass_mime_types:" where "message/rfc822" was also included. My list isn't very active but has come alive in the past few days, but strangely none of those messages are available in the archive. So I rechecked my settings and quickly removed "message/rfc822" once I found it. Still, this hasn't made any difference (new messages have been posted to the list since I made that change -I believe you've already pointed out that any changes don't affect messages already posted) and as far as I can see this shouldn't be the cause of this problem anyway (as you explained earlier, including "message/rfc822" only defines if attachements should be content filtered or not.
Any idea what could be causing list messages not to be archived? My "Archiving options" seem fine:
archive: yes archive_private: private archive_volume_freq.: monthly
Hal
On 12/10/2014 01:41 AM, Hal wrote:
On 18/11/2014 17:31, Mark Sapiro wrote:
In Mailman's Content filtering msection you want the following:
filter_content: Yes filter_mime_types: empty pass_mime_types: multipart text/plain text/html filter_filename_extensions: irrelevant, default list OK pass_filename_extensions: empty collapse_alternatives: Yes convert_html_to_plaintext: Yes filter_action: as desired, this will only apply to a message which contains no text/html or text/plain part.
All my settings were the same as above except "pass_mime_types:" where "message/rfc822" was also included.
And that's OK. If you want to allow the text content from a post with an attached message, then include message/rfc822. If you want to remove everything from a message attached to a post (e.g. a message forwarded "as attachment" to the list), then don't include message/rfc822.
My list isn't very active but has come alive in the past few days, but strangely none of those messages are available in the archive. So I rechecked my settings and quickly removed "message/rfc822" once I found it. Still, this hasn't made any difference (new messages have been posted to the list since I made that change -I believe you've already pointed out that any changes don't affect messages already posted) and as far as I can see this shouldn't be the cause of this problem anyway (as you explained earlier, including "message/rfc822" only defines if attachements should be content filtered or not.
Yes. the presence or absence of message/rfc822 in pass_mime_types doesn't affect whether or not messages which are delivered to the list are archived.
Any idea what could be causing list messages not to be archived? My "Archiving options" seem fine:
archive: yes archive_private: private archive_volume_freq.: monthly
Lot's of things. ArchRunner not running, permissions issues in the archive, other things causing exceptions in ArchRunner.
Check that ArchRunner is running (ps -fAww|grep ArchRunner).
Look in Mailman's error log. If you find errors and shunted messages, you can archive the shunted messages by running 'bin/unshunt' after fixing the underlying problem and after removing any unwanted message entries from qfiles/shunt. Run 'bin/show_qfiles qfiles/shunt/*' to see what's there/
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Mark Sapiro writes:
And that's OK. If you want to allow the text content from a post with an attached message, then include message/rfc822. If you want to remove everything from a message attached to a post (e.g. a message forwarded "as attachment" to the list), then don't include message/rfc822.
I don't know how many RFC pedants are actually out there, but if you're downstream from a third party that wraps messages From: Yahoo! et amis (ie, DMARC "p=reject" domains), you'll lose that content, too.
I suspect there are *very* few sites that wrap, but it's a potential concern.
On 10/12/2014 20:07, Mark Sapiro wrote:
On 12/10/2014 01:41 AM, Hal wrote:
Any idea what could be causing list messages not to be archived? My "Archiving options" seem fine:
archive: yes archive_private: private archive_volume_freq.: monthly
Lot's of things. ArchRunner not running, permissions issues in the archive, other things causing exceptions in ArchRunner.
Check that ArchRunner is running (ps -fAww|grep ArchRunner).
Here's what I get:
$ ps -fAww|grep ArchRunner mailman 1368 1346 0 Aug01 ? 00:36:10 /usr/bin/python /usr/lib/mailman/bin/qrunner --runner=ArchRunner:0:1 -s hal 20027 20009 0 20:32 pts/3 00:00:00 grep ArchRunner
Does this mean it's been running since august 1st this year and it's still running as it should?
Look in Mailman's error log. If you find errors and shunted messages, you can archive the shunted messages by running 'bin/unshunt' after fixing the underlying problem and after removing any unwanted message entries from qfiles/shunt. Run 'bin/show_qfiles qfiles/shunt/*' to see what's there/
$ /usr/lib/mailman/bin/show_qfiles qfiles/shunt/* ====================> qfiles/shunt/* Traceback (most recent call last): File "/usr/lib/mailman/bin/show_qfiles", line 95, in <module> main() File "/usr/lib/mailman/bin/show_qfiles", line 81, in main fp = open(filename) IOError: [Errno 2] No such file or directory: 'qfiles/shunt/*' $
I'm not sure how to interpret the above though.
I located the /var/log/mailman/ directory which is where I suppose the mentioned logs are located? There's a whole bunch of files there:
$ ls total 144 -rw-rw-r-- 1 mailman mailman 16289 Dec 10 21:07 bounce -rw-rw-r-- 1 mailman mailman 83 Nov 13 23:22 bounce-20141116 -rw-rw-r-- 1 mailman mailman 249 Nov 20 16:17 bounce-20141123 -rw-rw-r-- 1 mailman mailman 83 Nov 28 15:55 bounce-20141130 -rw-rw-r-- 1 mailman mailman 83 Dec 2 11:56 bounce-20141207 -rw-rw-r-- 1 apache mailman 15801 Dec 10 20:51 error -rw-rw-r-- 1 apache mailman 128 Nov 10 12:03 error-20141116 -rw-rw-r-- 1 apache mailman 0 Nov 16 03:34 error-20141123 -rw-rw-r-- 1 apache mailman 0 Nov 23 03:32 error-20141130 -rw-rw-r-- 1 apache mailman 0 Nov 30 03:12 error-20141207 -rw-rw-r-- 1 apache mailman 279 May 24 2014 mischief -rw-rw-r-- 1 mailman mailman 1843 Dec 10 20:51 post -rw-rw-r-- 1 mailman mailman 0 Nov 9 03:35 post-20141116 -rw-rw-r-- 1 mailman mailman 0 Nov 16 03:34 post-20141123 -rw-rw-r-- 1 mailman mailman 0 Nov 23 03:32 post-20141130 -rw-rw-r-- 1 mailman mailman 0 Nov 30 03:12 post-20141207 -rw-rw-r-- 1 mailman mailman 729 Dec 7 03:08 qrunner -rw-rw-r-- 1 mailman mailman 729 Nov 9 03:35 qrunner-20141116 -rw-rw-r-- 1 mailman mailman 729 Nov 16 03:34 qrunner-20141123 -rw-rw-r-- 1 mailman mailman 729 Nov 23 03:32 qrunner-20141130 -rw-rw-r-- 1 mailman mailman 729 Nov 30 03:12 qrunner-20141207 -rw-rw-r-- 1 mailman mailman 5834 Dec 11 12:00 smtp -rw-rw-r-- 1 mailman mailman 2235 Nov 15 08:00 smtp-20141116 -rw-rw-r-- 1 mailman mailman 4178 Nov 22 08:00 smtp-20141123 -rw-rw-r-- 1 mailman mailman 2643 Nov 29 08:00 smtp-20141130 -rw-rw-r-- 1 mailman mailman 2792 Dec 6 08:00 smtp-20141207 -rw-rw-r-- 1 mailman mailman 2460 Dec 10 20:51 smtp-failure -rw-rw-r-- 1 mailman mailman 0 Nov 9 03:35 smtp-failure-20141116 -rw-rw-r-- 1 mailman mailman 0 Nov 16 03:34 smtp-failure-20141123 -rw-rw-r-- 1 mailman mailman 0 Nov 23 03:32 smtp-failure-20141130 -rw-rw-r-- 1 mailman mailman 0 Nov 30 03:12 smtp-failure-20141207 -rw-rw-r-- 1 apache mailman 184 Dec 8 02:17 subscribe -rw-rw-r-- 1 apache mailman 96 Nov 13 23:22 subscribe-20141116 -rw-rw-r-- 1 apache mailman 96 Nov 20 16:17 subscribe-20141123 -rw-rw-r-- 1 apache mailman 96 Nov 28 15:55 subscribe-20141130 -rw-rw-r-- 1 apache mailman 0 Nov 30 03:12 subscribe-20141207 -rw-rw-r-- 1 mailman mailman 206 Dec 8 02:51 vette -rw-rw-r-- 1 mailman mailman 0 Nov 9 03:35 vette-20141116 -rw-rw-r-- 1 mailman mailman 374 Nov 20 15:14 vette-20141123 -rw-rw-r-- 1 mailman mailman 94 Nov 25 10:34 vette-20141130 -rw-rw-r-- 1 mailman mailman 204 Dec 6 01:57 vette-20141207 $
I had a look at the most obvious file "error" (with yesterday's date) which shows a whole lot of permissions errors:
$ more error Dec 09 12:58:27 2014 (1368) Archive file access failure: /var/lib/mailman/archives/private/my_list_name.mbox/my_list_name.mbox [Errno 13] Permission denied: '/var/lib/mailman/archives/private/my_list_name.mbox/my_list_name.mbox' Dec 09 12:58:27 2014 (1368) Uncaught runner exception: [Errno 13] Permission denied: '/var/lib/mailman/archives/private/my_list_name.mbo x/my_list_name.mbox' Dec 09 12:58:27 2014 (1368) Traceback (most recent call last): File "/usr/lib/mailman/Mailman/Queue/Runner.py", line 120, in _oneloop self._onefile(msg, msgdata) File "/usr/lib/mailman/Mailman/Queue/Runner.py", line 191, in _onefile keepqueued = self._dispose(mlist, msg, msgdata) File "/usr/lib/mailman/Mailman/Queue/ArchRunner.py", line 73, in _dispose mlist.ArchiveMail(msg) File "/usr/lib/mailman/Mailman/Archiver/Archiver.py", line 200, in ArchiveMail self.__archive_to_mbox(msg) File "/usr/lib/mailman/Mailman/Archiver/Archiver.py", line 169, in __archive_to_mbox mbox = self.__archive_file(afn) File "/usr/lib/mailman/Mailman/Archiver/Archiver.py", line 157, in __archive_file return Mailbox.Mailbox(open(afn, 'a+')) IOError: [Errno 13] Permission denied: '/var/lib/mailman/archives/private/my_list_name.mbox/my_list_name.mbox'
Dec 09 12:58:27 2014 (1368) SHUNTING: 1418083106.1332631+b14c9e32a3019daba3fe4613ed678dd0ea9ef638 Dec 09 13:25:58 2014 (1368) Archive file access failure: /var/lib/mailman/archives/private/my_list_name.mbox/my_list_name.mbox [Errno 13] Permission denied: '/var/lib/mailman/archives/privat e/my_list_name.mbox/my_list_name.mbox' Dec 09 13:25:58 2014 (1368) Uncaught runner exception: [Errno 13] Permission denied: '/var/lib/mailman/archives/private/my_list_name.mbo x/my_list_name.mbox' Dec 09 13:25:58 2014 (1368) Traceback (most recent call last): File "/usr/lib/mailman/Mailman/Queue/Runner.py", line 120, in _oneloop self._onefile(msg, msgdata) File "/usr/lib/mailman/Mailman/Queue/Runner.py", line 191, in _onefile keepqueued = self._dispose(mlist, msg, msgdata) File "/usr/lib/mailman/Mailman/Queue/ArchRunner.py", line 73, in _dispose mlist.ArchiveMail(msg) File "/usr/lib/mailman/Mailman/Archiver/Archiver.py", line 200, in ArchiveMail self.__archive_to_mbox(msg) File "/usr/lib/mailman/Mailman/Archiver/Archiver.py", line 169, in __archive_to_mbox mbox = self.__archive_file(afn) File "/usr/lib/mailman/Mailman/Archiver/Archiver.py", line 157, in __archive_file return Mailbox.Mailbox(open(afn, 'a+')) IOError: [Errno 13] Permission denied: '/var/lib/mailman/archives/private/my_list_name.mbox/my_list_name.mbox'
Dec 09 13:25:58 2014 (1368) SHUNTING: 1418084757.3008399+c295e9e54242ea28350cd2bc2b722c9e1c599436 Dec 09 14:26:30 2014 (1368) Archive file access failure: /var/lib/mailman/archives/private/my_list_name.mbox/my_list_name.mbox [Errno 13] Permission denied: '/var/lib/mailman/archives/privat e/my_list_name.mbox/my_list_name.mbox' Dec 09 14:26:30 2014 (1368) Uncaught runner exception: [Errno 13] Permission denied: '/var/lib/mailman/archives/private/my_list_name.mbo x/my_list_name.mbox' Dec 09 14:26:30 2014 (1368) Traceback (most recent call last): File "/usr/lib/mailman/Mailman/Queue/Runner.py", line 120, in _oneloop self._onefile(msg, msgdata) File "/usr/lib/mailman/Mailman/Queue/Runner.py", line 191, in _onefile
I've obviously messed something up while importing all my MBOX files into the archive. Here's what comes up when I go to the archives to see the mbox file mentioned in the error log above:
$ cd /var/lib/mailman/archives $ ls total 8 drwxr-s--x 6 hal mailman 4096 Nov 19 00:41 private/ drwxr-sr-x 2 hal mailman 4096 Nov 8 15:39 public/ $ cd private/ $ ls total 32 drwxr-sr-x 2 hal mailman 4096 Nov 8 15:39 mailman/ drwxr-sr-x 2 hal mailman 4096 Nov 8 15:39 mailman.mbox/ drwxrwsr-x 176 hal mailman 20480 Nov 19 03:27 my_list_name/ drwxr-sr-x 2 hal mailman 4096 Nov 8 15:39 my_list_name.mbox/ $ cd my_list_name.mbox/ $ ls total 808 -rw-r--r-- 1 hal mailman 825894 Nov 8 15:39 my_list_name.mbox $
Hal
On 12/11/2014 01:50 AM, Hal wrote:
$ ps -fAww|grep ArchRunner mailman 1368 1346 0 Aug01 ? 00:36:10 /usr/bin/python /usr/lib/mailman/bin/qrunner --runner=ArchRunner:0:1 -s hal 20027 20009 0 20:32 pts/3 00:00:00 grep ArchRunner
Does this mean it's been running since august 1st this year and it's still running as it should?
Yes.
Look in Mailman's error log. If you find errors and shunted messages, you can archive the shunted messages by running 'bin/unshunt' after fixing the underlying problem and after removing any unwanted message entries from qfiles/shunt. Run 'bin/show_qfiles qfiles/shunt/*' to see what's there/
$ /usr/lib/mailman/bin/show_qfiles qfiles/shunt/* ====================> qfiles/shunt/* Traceback (most recent call last): File "/usr/lib/mailman/bin/show_qfiles", line 95, in <module> main() File "/usr/lib/mailman/bin/show_qfiles", line 81, in main fp = open(filename) IOError: [Errno 2] No such file or directory: 'qfiles/shunt/*' $
I'm not sure how to interpret the above though.
You have to point it at Mailman's qfiles/shunt/ directory. Depending on what packaged Mailman this is, that may be /var/lib/mailman/qfiles/shunt or /var/spool/mailman/shunt/ or somewhere else.
I located the /var/log/mailman/ directory which is where I suppose the mentioned logs are located? There's a whole bunch of files there:
$ ls total 144 -rw-rw-r-- 1 mailman mailman 16289 Dec 10 21:07 bounce -rw-rw-r-- 1 mailman mailman 83 Nov 13 23:22 bounce-20141116 -rw-rw-r-- 1 mailman mailman 249 Nov 20 16:17 bounce-20141123 -rw-rw-r-- 1 mailman mailman 83 Nov 28 15:55 bounce-20141130 -rw-rw-r-- 1 mailman mailman 83 Dec 2 11:56 bounce-20141207
That's a lot of recent bounce activity. You might look at that.
-rw-rw-r-- 1 apache mailman 15801 Dec 10 20:51 error
This is the one we need to see.
...
I had a look at the most obvious file "error" (with yesterday's date) which shows a whole lot of permissions errors:
$ more error Dec 09 12:58:27 2014 (1368) Archive file access failure: /var/lib/mailman/archives/private/my_list_name.mbox/my_list_name.mbox [Errno 13] Permission denied: '/var/lib/mailman/archives/private/my_list_name.mbox/my_list_name.mbox' Dec 09 12:58:27 2014 (1368) Uncaught runner exception: [Errno 13] Permission denied: '/var/lib/mailman/archives/private/my_list_name.mbo x/my_list_name.mbox' Dec 09 12:58:27 2014 (1368) Traceback (most recent call last): File "/usr/lib/mailman/Mailman/Queue/Runner.py", line 120, in _oneloop self._onefile(msg, msgdata) File "/usr/lib/mailman/Mailman/Queue/Runner.py", line 191, in _onefile keepqueued = self._dispose(mlist, msg, msgdata) File "/usr/lib/mailman/Mailman/Queue/ArchRunner.py", line 73, in _dispose mlist.ArchiveMail(msg) File "/usr/lib/mailman/Mailman/Archiver/Archiver.py", line 200, in ArchiveMail self.__archive_to_mbox(msg) File "/usr/lib/mailman/Mailman/Archiver/Archiver.py", line 169, in __archive_to_mbox mbox = self.__archive_file(afn) File "/usr/lib/mailman/Mailman/Archiver/Archiver.py", line 157, in __archive_file return Mailbox.Mailbox(open(afn, 'a+')) IOError: [Errno 13] Permission denied: '/var/lib/mailman/archives/private/my_list_name.mbox/my_list_name.mbox' ... I've obviously messed something up while importing all my MBOX files into the archive. Here's what comes up when I go to the archives to see the mbox file mentioned in the error log above:
$ cd /var/lib/mailman/archives $ ls total 8 drwxr-s--x 6 hal mailman 4096 Nov 19 00:41 private/ drwxr-sr-x 2 hal mailman 4096 Nov 8 15:39 public/ $ cd private/ $ ls total 32 drwxr-sr-x 2 hal mailman 4096 Nov 8 15:39 mailman/ drwxr-sr-x 2 hal mailman 4096 Nov 8 15:39 mailman.mbox/ drwxrwsr-x 176 hal mailman 20480 Nov 19 03:27 my_list_name/ drwxr-sr-x 2 hal mailman 4096 Nov 8 15:39 my_list_name.mbox/ $ cd my_list_name.mbox/ $ ls total 808 -rw-r--r-- 1 hal mailman 825894 Nov 8 15:39 my_list_name.mbox
And here's the problem.
sudo chmod g+w /var/lib/mailman/archives/private/my_list_name.mbox/my_list_name.mbox
will fix it. Then you can run bin/unshunt which will put the shunted messages in the archive, but as I said before, it's a good idea to first examine the contents of qfiles/shunt/ to be sure there aren't other messages there that you don't want requeued.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 11/12/2014 19:12, Mark Sapiro wrote:
$ ls total 808 -rw-r--r-- 1 hal mailman 825894 Nov 8 15:39 my_list_name.mbox
And here's the problem.
sudo chmod g+w /var/lib/mailman/archives/private/my_list_name.mbox/my_list_name.mbox
will fix it. Then you can run bin/unshunt which will put the shunted messages in the archive, but as I said before, it's a good idea to first examine the contents of qfiles/shunt/ to be sure there aren't other messages there that you don't want requeued.
Got it! Thanks. The messages have entered the archive now and I'll keep a watch when new ones arrive to see if they're archived or bounced. So in summary, this is what I did:
- permission fixing for the archive
$ sudo chmod g+w /var/lib/mailman/archives/private/my_list_name.mbox/my_list_name.mbox
So this adds write permission to the group ("hal" and "mailman"), right? Here's a listing before issuing the above chmod command:
$ pwd /var/lib/mailman/archives/private/my_list_name.mbox $ ls total 808 -rw-r--r-- 1 hal mailman 825894 Nov 8 15:39 my_list_name.mbox $
..... then after doing "chmod g+w":
$ pwd /var/lib/mailman/archives/private/my_list_name.mbox $ ls total 860 -rw-rw-r-- 1 hal mailman 876548 Dec 12 23:22 my_list_name.mbox $
- Viewing the "error" Mailman logfile (to see what's wrong)
$ cd /var/log/mailman/ $ more error
- See what's inside the shunt directory and view their contents
(shunt=messages set aside instead of being archived) $ cd /var/spool/mailman/shunt/ $ ls $ /usr/lib/mailman/bin/show_qfiles /var/spool/mailman/shunt/*
- "Unshunt" messages
(put the set aside messages into the archive) $ /usr/lib/mailman/bin/unshunt
Hal
On 12/12/2014 02:37 AM, Hal wrote:
- permission fixing for the archive
$ sudo chmod g+w /var/lib/mailman/archives/private/my_list_name.mbox/my_list_name.mbox
So this adds write permission to the group ("hal" and "mailman"), right? Here's a listing before issuing the above chmod command:
...
$ pwd /var/lib/mailman/archives/private/my_list_name.mbox $ ls total 860 -rw-rw-r-- 1 hal mailman 876548 Dec 12 23:22 my_list_name.mbox $
Yes. All of Mailman's file access is based on group permissions. The actual owner and owner permissions are irrelevant in most cases. Everything should be in Mailman's group (mailman in this case) and the desired access permitted for that group.
...
- "Unshunt" messages
(put the set aside messages into the archive) $ /usr/lib/mailman/bin/unshunt
Yes, and since this succeeded in adding the previously shunted messages to the archive, you can be assured that future messages will not be shunted, at least not for this reason.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
participants (3)
-
Hal -
Mark Sapiro -
Stephen J. Turnbull