[Mailman-Users] Attachments are unexpectedly re-created.

[Tokio Kikuchi]

(10/03/03 16:54), Masaharu Kawada wrote:
> Hi Mark-san,
> 
> I got a reply from the customer, but it was not possbile for the customer
> to provide the whole digest.mbox to us due to their policy.  However,
> sevral parts of the error log and digest.mbox's content have been provided.
> As for the error messages, please see the attachment.  And as for the
> digest.mbox which might be the problematic one, I see lots of messages look
> like below.
> 

Hi, Kawada san,

The problem is that the Japanese mail users/MUA developers use CP-932
charset as Shift-JIS and its derivatives (ISO-2022-JP/EUC-JP).  CP-932
contains more (extended) characters than Shift-JIS while Python codec is
strict on the latter.  The characters like circled numbers fail to be
decoded in unicode and cause error.

Similar errors are reported in Japanese Mailman users, like the thread
starting from:
http://mm.tkikuchi.net/pipermail/mmjp-users/2009-February/002487.html

A workaround is to patch charset.py in Python email library as:

--- Lib/email/charset.py	2009-09-22 08:59:56.000000000 +0900
+++ Lib/email/charset.py.orig	2009-09-22 08:58:36.000000000 +0900
@@ -264,8 +264,7 @@
     def convert(self, s):
         """Convert a string from the input_codec to the output_codec."""
         if self.input_codec != self.output_codec:
-            return unicode(s, self.input_codec, 'replace'
-                     ).encode(self.output_codec, 'replace')
+            return unicode(s, self.input_codec).encode(self.output_codec)
         else:
             return s



-- 
Tokio Kikuchi, tkikuchi at is.kochi-u.ac.jp
http://weather.is.kochi-u.ac.jp/