archive processing problem (mmarch w/ MIME)
I've checked the FAQ and Googled this before coming here to ask for advice. Background: Running Debian 4.0, x86_64, MTA is Postfix, packaged Debian Mailman (version listed "2.1.9-7"), Python version 2.4.4.
I'm having a problem where the archives are not ingesting MIME email for various mailing lists that I maintain. A message goes to the mailing list, is processed properly and gets reflected to the recipients. All email to the list winds up in /var/lib/mailman/archives/private/listname.mbox/listname.mbox. However, messages that have MIME encoding don't make it to /var/lib/mailman/archives/private/listname for inclusion in the web page.
I tried running mmarch to rebuild the archive, and got the following output:
"mmarch" seems to be having trouble with the Content-Type header line.
If I pull out that line for a given message, it is ingested, though it
displays the full MIME text.
Any direction on where to look would be appreciated.
-John
-- John Reddy jreddy@bnl.gov (631) 344-3284 Technology Architect ITD Unix Services Brookhaven National Labs
What exactly was the Content-Type header line ?
-- Tokio Kikuchi, tkikuchi@is.kochi-u.ac.jp http://weather.is.kochi-u.ac.jp/
On 6/6/2007 8:08 PM Tokio Kikuchi said the following: line as is, this is how it exists in the message header. The mbox is readable by mutt (my text email reader of choice).
Content-Transfer-Encoding: binary Content-Type: multipart/alternative; boundary="_----------=_1136314441199510" MIME-Version: 1.0
If I delete these lines from the mbox, the process will continue to the next entry in the mbox and then it will fail with the same set of error messages again:
Content-Transfer-Encoding: binary Content-Type: multipart/alternative; boundary="_----------=_1136319711206230" MIME-Version: 1.0
Doesn't make much sense to me why it's having trouble ingesting MIME.
-John
-- John Reddy jreddy@bnl.gov (631) 344-3284 Technology Architect ITD Unix Services Brookhaven National Labs
John Reddy wrote:
How about deleteing the first line (Content-Transfer-Encoding: binary) and leave the rest.
I believe CTE should not be in the main section of message header if Content-Type is multipart.
-John
-- Tokio Kikuchi, tkikuchi@is.kochi-u.ac.jp http://weather.is.kochi-u.ac.jp/
On 6/7/07, Tokio Kikuchi wrote:
I believe CTE should not be in the main section of message header if Content-Type is multipart.
In addition, would this not point to a potential bug in the "email" module that we inherit from Python (see <http://docs.python.org/lib/module-email.html>), as opposed to a potential bug within Mailman itself?
-- Brad Knowles <brad@shub-internet.org>, Consultant & Author LinkedIn Profile: <http://tinyurl.com/y8kpxu> Slides from Invited Talks: <http://tinyurl.com/tj6q4>
09 F9 11 02 9D 74 E3 5B D8 41 56 C5 63 56 88 C0
On 6/7/07, John Reddy wrote:
Now that you remind me that this is Debian and you're using the packaged version of Mailman, I am having this vague feeling that we've run into problems before with Debian and the way they handle (mishandle) the email module within Python.
I'm not sure this is related, but check out the thread at <http://mail.python.org/pipermail/mailman-users/2007-April/056659.html>.
If this does turn out to be related, then it sounds like we have another issue to put into the FAQ Wizard.
-- Brad Knowles <brad@shub-internet.org>, Consultant & Author LinkedIn Profile: <http://tinyurl.com/y8kpxu> Slides from Invited Talks: <http://tinyurl.com/tj6q4>
09 F9 11 02 9D 74 E3 5B D8 41 56 C5 63 56 88 C0
On 6/7/2007 9:03 AM Tokio Kikuchi said the following: problem. Only removing the Content-Type header does. It's not that the mails suddenly changed. We have dozens of mailing lists with mbox files going back a couple of years. It's just that I've been looking into this recently and just noticed it.
-John
-- John Reddy jreddy@bnl.gov (631) 344-3284 Technology Architect ITD Unix Services Brookhaven National Labs
Just as a follow-up to this problem from earlier this month. Turns out the cause of this headache was related to a migration my group had performed. We built a completely new server for mailman. As part of the migration, the admin performing the move copied files from the old server to the new server by way of rsync.
In the mix, /var/lib/mailman/pythonlib, which the debian package installs as a symlink to /usr/lib/mailman/pythonlib got erased by the rsync process. Without /var/lib/mailman/pythonlib available in the expected location, mmarch failed on any MIME encoding.
Thanks to everyone who tried helping figure out the problem
-John Reddy
On 6/6/2007 5:34 PM John Reddy said the following:
-- John Reddy jreddy@bnl.gov (631) 344-3284 Technology Architect ITD Unix Services Brookhaven National Labs
What exactly was the Content-Type header line ?
-- Tokio Kikuchi, tkikuchi@is.kochi-u.ac.jp http://weather.is.kochi-u.ac.jp/
On 6/6/2007 8:08 PM Tokio Kikuchi said the following: line as is, this is how it exists in the message header. The mbox is readable by mutt (my text email reader of choice).
Content-Transfer-Encoding: binary Content-Type: multipart/alternative; boundary="_----------=_1136314441199510" MIME-Version: 1.0
If I delete these lines from the mbox, the process will continue to the next entry in the mbox and then it will fail with the same set of error messages again:
Content-Transfer-Encoding: binary Content-Type: multipart/alternative; boundary="_----------=_1136319711206230" MIME-Version: 1.0
Doesn't make much sense to me why it's having trouble ingesting MIME.
-John
-- John Reddy jreddy@bnl.gov (631) 344-3284 Technology Architect ITD Unix Services Brookhaven National Labs
John Reddy wrote:
How about deleteing the first line (Content-Transfer-Encoding: binary) and leave the rest.
I believe CTE should not be in the main section of message header if Content-Type is multipart.
-John
-- Tokio Kikuchi, tkikuchi@is.kochi-u.ac.jp http://weather.is.kochi-u.ac.jp/
On 6/7/07, Tokio Kikuchi wrote:
I believe CTE should not be in the main section of message header if Content-Type is multipart.
In addition, would this not point to a potential bug in the "email" module that we inherit from Python (see <http://docs.python.org/lib/module-email.html>), as opposed to a potential bug within Mailman itself?
-- Brad Knowles <brad@shub-internet.org>, Consultant & Author LinkedIn Profile: <http://tinyurl.com/y8kpxu> Slides from Invited Talks: <http://tinyurl.com/tj6q4>
09 F9 11 02 9D 74 E3 5B D8 41 56 C5 63 56 88 C0
On 6/7/07, John Reddy wrote:
Now that you remind me that this is Debian and you're using the packaged version of Mailman, I am having this vague feeling that we've run into problems before with Debian and the way they handle (mishandle) the email module within Python.
I'm not sure this is related, but check out the thread at <http://mail.python.org/pipermail/mailman-users/2007-April/056659.html>.
If this does turn out to be related, then it sounds like we have another issue to put into the FAQ Wizard.
-- Brad Knowles <brad@shub-internet.org>, Consultant & Author LinkedIn Profile: <http://tinyurl.com/y8kpxu> Slides from Invited Talks: <http://tinyurl.com/tj6q4>
09 F9 11 02 9D 74 E3 5B D8 41 56 C5 63 56 88 C0
On 6/7/2007 9:03 AM Tokio Kikuchi said the following: problem. Only removing the Content-Type header does. It's not that the mails suddenly changed. We have dozens of mailing lists with mbox files going back a couple of years. It's just that I've been looking into this recently and just noticed it.
-John
-- John Reddy jreddy@bnl.gov (631) 344-3284 Technology Architect ITD Unix Services Brookhaven National Labs
Just as a follow-up to this problem from earlier this month. Turns out the cause of this headache was related to a migration my group had performed. We built a completely new server for mailman. As part of the migration, the admin performing the move copied files from the old server to the new server by way of rsync.
In the mix, /var/lib/mailman/pythonlib, which the debian package installs as a symlink to /usr/lib/mailman/pythonlib got erased by the rsync process. Without /var/lib/mailman/pythonlib available in the expected location, mmarch failed on any MIME encoding.
Thanks to everyone who tried helping figure out the problem
-John Reddy
On 6/6/2007 5:34 PM John Reddy said the following:
-- John Reddy jreddy@bnl.gov (631) 344-3284 Technology Architect ITD Unix Services Brookhaven National Labs
participants (4)
-
Brad Knowles -
John Reddy -
Mark Sapiro -
Tokio Kikuchi