Unknown encoding error
Hello list,
A week or so ago I ran across this problem on one of our mailing lists: A user had sent an email with an apparently bad subject line. This has caused mailman to stop delivering emails for that specific mailning list. Here's the error from the logfile:
Jan 05 21:25:10 2009 (23055) Uncaught runner exception: unknown encoding: Re: DDS 26073 : Imap - installation d Jan 05 21:25:10 2009 (23055) Traceback (most recent call last): File "/usr/lib/mailman/Mailman/Queue/Runner.py", line 111, in _oneloop self._onefile(msg, msgdata) File "/usr/lib/mailman/Mailman/Queue/Runner.py", line 167, in _onefile keepqueued = self._dispose(mlist, msg, msgdata) File "/usr/lib/mailman/Mailman/Queue/IncomingRunner.py", line 130, in _dispose more = self._dopipeline(mlist, msg, msgdata, pipeline) File "/usr/lib/mailman/Mailman/Queue/IncomingRunner.py", line 153, in _dopipeline sys.modules[modname].process(mlist, msg, msgdata) File "/var/lib/mailman/Mailman/Handlers/ToDigest.py", line 91, in process send_digests(mlist, mboxfp) File "/var/lib/mailman/Mailman/Handlers/ToDigest.py", line 132, in send_digests send_i18n_digests(mlist, mboxfp) File "/var/lib/mailman/Mailman/Handlers/ToDigest.py", line 306, in send_i18n_digests msg = scrubber(mlist, msg) File "/var/lib/mailman/Mailman/Handlers/Scrubber.py", line 240, in process url = save_attachment(mlist, part, dir) File "/var/lib/mailman/Mailman/Handlers/Scrubber.py", line 373, in save_attachment fnext = os.path.splitext(msg.get_filename(''))[1] File "/usr/lib/python2.3/email/Message.py", line 731, in get_filename return unicode(newvalue[2], newvalue[0] or 'us-ascii') LookupError: unknown encoding: Re: DDS 26073 : Imap - installation d
Jan 05 21:25:10 2009 (23055) SHUNTING: 1231190705.198385+f75914e7ce7a9daf6e70914e0ed49923a47d2dd1
The full subject line should be:
RE: DDS 26073 : Imap - installation d'application 'vacation'
so I think the apostrophe is causing havoc here...
I have searched mailing list and the search engines, and it looks like this error was a problem for people but has been solved in subsequent versions of mailman (we are running 2.1.5 on Debian Sarge). What I haven't been able to find out is how to purge out the offending email, and continue life as normal until we upgrade to Etch!
I should say that I have tried removing the offending file from $MAILMANDIR/qfiles/shunt and running "unshunt", but nothing changed.
Any suggestions are most appreciated. /Khosrow
Khosrow Ebrahimpour wrote:
A week or so ago I ran across this problem on one of our mailing lists: A user had sent an email with an apparently bad subject line. This has caused mailman to stop delivering emails for that specific mailning list. Here's the error from the logfile:
Jan 05 21:25:10 2009 (23055) Uncaught runner exception: unknown encoding: Re: DDS 26073 : Imap - installation d Jan 05 21:25:10 2009 (23055) Traceback (most recent call last): File "/usr/lib/mailman/Mailman/Queue/Runner.py", line 111, in _oneloop self._onefile(msg, msgdata) File "/usr/lib/mailman/Mailman/Queue/Runner.py", line 167, in _onefile keepqueued = self._dispose(mlist, msg, msgdata) File "/usr/lib/mailman/Mailman/Queue/IncomingRunner.py", line 130, in _dispose more = self._dopipeline(mlist, msg, msgdata, pipeline) File "/usr/lib/mailman/Mailman/Queue/IncomingRunner.py", line 153, in _dopipeline sys.modules[modname].process(mlist, msg, msgdata) File "/var/lib/mailman/Mailman/Handlers/ToDigest.py", line 91, in process send_digests(mlist, mboxfp) File "/var/lib/mailman/Mailman/Handlers/ToDigest.py", line 132, in send_digests send_i18n_digests(mlist, mboxfp) File "/var/lib/mailman/Mailman/Handlers/ToDigest.py", line 306, in send_i18n_digests msg = scrubber(mlist, msg) File "/var/lib/mailman/Mailman/Handlers/Scrubber.py", line 240, in process url = save_attachment(mlist, part, dir) File "/var/lib/mailman/Mailman/Handlers/Scrubber.py", line 373, in save_attachment fnext = os.path.splitext(msg.get_filename(''))[1] File "/usr/lib/python2.3/email/Message.py", line 731, in get_filename return unicode(newvalue[2], newvalue[0] or 'us-ascii') LookupError: unknown encoding: Re: DDS 26073 : Imap - installation d
Jan 05 21:25:10 2009 (23055) SHUNTING: 1231190705.198385+f75914e7ce7a9daf6e70914e0ed49923a47d2dd1
The message that causes this problem is not the message that is shunted. All the shunted messages are (presumably) valid messages that arrived after the issue started.
The problem is that a message was posted to the list and probably delivered normally and stored in the digest.mbox file for the list. Then, eventually, the digest.mbox grew big enough that every subsequent message triggers a digest delivery on size, but the bad message in digest.mbox throws the above exception in the ToDigest handler, and the digest is not produced and the current message is shunted.
The full subject line should be:
RE: DDS 26073 : Imap - installation d'application 'vacation'
so I think the apostrophe is causing havoc here...
Yes, it is an apostrophe that is the problem, but it is in a 'name' or 'filename' parameter of a Content-Type: or Content-Disposition: header.
I have searched mailing list and the search engines, and it looks like this error was a problem for people but has been solved in subsequent versions of mailman (we are running 2.1.5 on Debian Sarge). What I haven't been able to find out is how to purge out the offending email, and continue life as normal until we upgrade to Etch!
I should say that I have tried removing the offending file from $MAILMANDIR/qfiles/shunt and running "unshunt", but nothing changed.
You don't want to remove the shunted files as they are not the problem, and you can't successfully unshunt until you fix the digest.mbox.
You need to edit the lists/LISTNAME/digest.mbox. You can edit it by hand or with an MUA like mutt for example.
The offending message has an RFC 2231 enncoded 'name' or 'filename' parameter with an apostrophe in the name. It will look something like
Content-Disposition: attachment; filename*0*='charset'lang'text.................................... filename*1=more.............................
And text will have an apostrophe.
You can simply remove those parameters (filename and/or name) or delete the entire message.
You can also just delete the lists/LISTNAME/digest.mbox file, but then you'd lose those messages from the digest.
After fixing the digest.mbox, you can run bin/unshunt to finish processing the shunted messages.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 6 January 2009 11:36, Mark Sapiro wrote:
You need to edit the lists/LISTNAME/digest.mbox. You can edit it by hand or with an MUA like mutt for example.
Thank you Mark. That solution worked perfectly. I modified the digest.mbox and changed only the offending attachment content-types. After running "unshunt" everything behaved nicely.
A more general question. Has this issue been fixed in subsequent releases of mailman? We are running 2.1.5 on Sarge at the moment. I suspect that we will be seeing this error occasionally, although I'm surprised it hasn't been an issue so far.
Cheers, Khosrow
Khosrow Ebrahimpour wrote:
A more general question. Has this issue been fixed in subsequent releases of mailman? We are running 2.1.5 on Sarge at the moment. I suspect that we will be seeing this error occasionally, although I'm surprised it hasn't been an issue so far.
Yes. The problem of unanticipated exceptions in digest processing stopping a list was fixed in 2.1.7.
The problem of apostrophes in RFC 2231 encoded parameters was fixed in the underlying Python email library. I don't remember exactly when, but I'm pretty sure it was fixed in email 2.5.8 which has shipped with Mailman starting with 2.1.9.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
participants (2)
-
Khosrow Ebrahimpour -
Mark Sapiro