[Mailman-Users] Charset problem. cmd_confirm assumes ascii code.

[Osmo Kujala]

Hello

We have problem with confirm mail when mail body contains
characters out of ascii code (range(128)). Mailman version
is 2.1.5 . I think Mailman could leave the body unread
when Subject contains the confirm code or at least it should
stop to "end" line. If Mailman wants to check through the body
it would be fair not to bug with the non-ascii codes.
People have auto-signatures with different character codes and
they may have automatic copy of the original mail as citation
(quotation). Mailmans request for confirmation contains non-ascii
code if the language code of the list is such.

Excerpt of the error log:

Apr 26 12:53:40 2006 (11216) Uncaught runner exception: 'ascii' codec can't decode byte 0xe4 in position 3: ordinal not in range(128)
Apr 26 12:53:40 2006 (11216) Traceback (most recent call last):
  File "/usr/lib/mailman/Mailman/Queue/Runner.py", line 111, in _oneloop
    self._onefile(msg, msgdata)
  File "/usr/lib/mailman/Mailman/Queue/Runner.py", line 167, in _onefile
    keepqueued = self._dispose(mlist, msg, msgdata)
  File "/usr/lib/mailman/Mailman/Queue/CommandRunner.py", line 237, in _dispose
    res.process()
  File "/usr/lib/mailman/Mailman/Queue/CommandRunner.py", line 110, in process
    stop = self.do_command(cmd, args)
  File "/usr/lib/mailman/Mailman/Queue/CommandRunner.py", line 135, in do_command
    return self.do_command(cmd, args)
  File "/usr/lib/mailman/Mailman/Queue/CommandRunner.py", line 137, in do_command
    return handler.process(self, args)
  File "/usr/lib/mailman/Mailman/Commands/cmd_confirm.py", line 86, in process
    if line.lstrip() == match:
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe4 in position 3: ordinal not in range(128)

Apr 26 12:53:40 2006 (11216) SHUNTING: 1146045219.7761891+6b2f2afceea57f26e6a9302e6a615bbd8266d493


Copy of the shunted file:

/usr/lib~mailman/bin/dumpdb -p /var/spool/mailman/shunt/1146045219.7761891+6b2f2afceea57f26e6a9302e6a615bbd8266d493.pck

[----- start pickle file -----]
<----- start object 1 ----->
>From KUJALA at jylk.jyu.fi Wed Apr 26 12:53:39 2006
Received: from posti1.jyu.fi (posti1.jyu.fi [130.234.4.32])
        by lists.jyu.fi (8.13.1/8.13.1) with ESMTP id k3Q9rdOt029242
        for <ok-testi-request at lists.jyu.fi>; Wed, 26 Apr 2006 12:53:39 +0300
Received: from localhost (localhost.localdomain [127.0.0.1])
        by posti1.jyu.fi (8.13.6/8.13.4) with ESMTP id k3Q9rdAc030411
        for <ok-testi-request at lists.jyu.fi>; Wed, 26 Apr 2006 12:53:39 +0300
Received: from jylk.jyu.fi (jylk.jyu.fi [130.234.4.141])
        by posti1.jyu.fi (8.13.6/8.13.6) with ESMTP id k3Q9rcUj030404
        for <ok-testi-request at lists.jyu.fi>; Wed, 26 Apr 2006 12:53:38 +0300
Received: from jylk.jyu.fi by jylk.jyu.fi (PMDF V4.3-7 #2482)
        id <01M1PGKIPOB4ECWZZ9 at jylk.jyu.fi>; Wed, 26 Apr 2006 12:54:35 EET
Date: Wed, 26 Apr 2006 12:54:11 +0200 (EET)
From: Osmo Kujala <KUJALA at jylk.jyu.fi>
Subject: Re: confirm 8e7af57ba7a721f68c96b0111f988ad36c8804f7
In-reply-to: Your message dated "Wed, 26 Apr 2006 12:52:38 +0300"
        <mailman.0.1146045158.29214.ok-testi at lists.jyu.fi>
To: ok-testi-request at lists.jyu.fi
Message-id: <01M1PQ8OVNMYECWZZ9 at jylk.jyu.fi>
X-Envelope-to: ok-testi-request at lists.jyu.fi
MIME-version: 1.0
Content-type: TEXT/PLAIN; CHARSET=ISO-8859-1
Content-transfer-encoding: 8BIT
X-Virus-Scanned: amavisd-new at cc.jyu.fi
X-Spam-Status: No, hits=-5.794 required=5 tests=[ALL_TRUSTED=-3.3, AWL=0.105,
        BAYES_00=-2.599]
X-Spam-Level:


end

123ä
23456ö ÖÄ


<----- start object 2 ----->
{   '_parsemsg': False,
    'lang': 'en',
    'listname': 'ok-testi',
    'received_time': 1146045219.7761891,
    'torequest': 1,
    'version': 3,
    'whichq': '/var/spool/mailman/commands'}
[----- end pickle file -----]


What's especially nasty is that the waiting time for confirmation
stops with this error and the entry disappears from the list.
So it's hard to find out what happened.

I tried to find help from mailman-users archives. Found some
identical cases from years 2003 and 2004 by searching keywords
"ascii ordinal confirm" but no solution to the problem.

best regars

Osmo Kujala   (Sorry about the typo's and I hope the scandinavian
               characters won't stop you reading this)