[Mailman-Users] Encoding problem with 2.15 to 2.18 upgrade with Finnish

[Mark Sapiro]

On 01/07/2016 04:40 AM, Eva Isaksson wrote:
> 
> Also:
> http://sources.debian.net/patches/patch/mailman/1:2.1.18-2/91_utf8.patch/
> <https://mail.google.com/>
> 
> If this means that debian is forcing everything to utf-8, what should I do,
> if every package comes with this forced utf-8 encoding?


This is a Debian issue. The FAQ at <http://wiki.list.org/x/12812344>
addresses that, but probably won't be much comfort to you.

You could install Mailman from source. See the FAQ at
<http://wiki.list.org/x/17891606> for info.

More below ...


> Here is the trackback we got in October:
> 
> Oct 28 10:04:28 2015 (22144) Uncaught runner exception: 'utf8' codec
> can't decode byte 0xe4 in position 11: invalid continuation byt
> e
> Oct 28 10:04:28 2015 (22144) Traceback (most recent call last):
>   File "/var/lib/mailman/Mailman/Queue/Runner.py", line 119, in _oneloop
>     self._onefile(msg, msgdata)
>   File "/var/lib/mailman/Mailman/Queue/Runner.py", line 190, in _onefile
>     keepqueued = self._dispose(mlist, msg, msgdata)
>   File "/var/lib/mailman/Mailman/Queue/IncomingRunner.py", line 130, in
> _dispose
>     more = self._dopipeline(mlist, msg, msgdata, pipeline)
>   File "/var/lib/mailman/Mailman/Queue/IncomingRunner.py", line 153, in
> _dopipeline
>     sys.modules[modname].process(mlist, msg, msgdata)
>   File "/var/lib/mailman/Mailman/Handlers/CookHeaders.py", line 239, in
> process
>     i18ndesc = uheader(mlist, mlist.description, 'List-Id', maxlinelen=998)
>   File "/var/lib/mailman/Mailman/Handlers/CookHeaders.py", line 65, in
> uheader
>     return Header(s, charset, maxlinelen, header_name, continuation_ws)
>   File "/usr/lib/python2.7/email/header.py", line 183, in __init__
>     self.append(s, charset, errors)
>   File "/usr/lib/python2.7/email/header.py", line 267, in append
>     ustr = unicode(s, incodec, errors)
> UnicodeDecodeError: 'utf8' codec can't decode byte 0xe4 in position 11:
> invalid continuation byte


This is exactly the issue described at
<https://bugs.launchpad.net/mailman/+bug/1462755>. Note that the person
who reported that is Thijs Kinkhorst who is the Debian maintainer of
Mailman, so they are well aware of this issue yet they continue to
distribute a package which causes this issue for their users. See in
particular <https://bugs.launchpad.net/mailman/+bug/1462755/comments/3>.
Had I realized when I wrote that that it was Debian and not the user
that had changed the encoding for the language, that reply would have
been much stronger.

The problem in your case is that even though the character set for
Finnish in your Mailman is UTF-8, your list(s) still have strings in
their attributes which are ISO-8859-1 encoded. In this particular case
the list's General Options -> description contains one or more
characters encoded in ISO-8859-1.

I think simply re-entering the description and saving changes will fix
this, but you should probably go through the entire web admin UI and do
the same for any other non-ascii strings.

-- 
Mark Sapiro <mark at msapiro.net>        The highway is for gamblers,
San Francisco Bay Area, California    better use your sense - B. Dylan