[Mailman-Users] Encoding problem with 2.15 to 2.18 upgrade with Finnish

[Eva Isaksson]

I did the 2.1.18 upgrade this weekend, after some weighing of the options.
In the long run, using the Debian package is less work intensive, so
in the end my choice was the utf-8 conversion.

> The script is at <https://www.msapiro.net/scripts/recode_list> (mirrored
> at <http://fog.ccsf.edu/~msapiro/scripts/recode_list>)

To cut the long story short, the script worked without any problems.
Taking backups took more time than running the script and doing the
package upgrade.

Such a relief - thank you.

- Eva Isaksson



2016-01-08 2:55 GMT+02:00 Mark Sapiro <mark at msapiro.net>:

> On 01/07/2016 01:20 PM, Mark Sapiro wrote:
> >
> > You can keep the Debian package with it's utf-8 character sets and
> > programmatically change all your lists so they work. In MM 2.1.19 I
> > changed the character sets for Russian and Romanian from koi8-r and
> > iso-8859-2 respectively to utf-8 because I was convinced that the former
> > character sets were not appropriate for those languages. In so doing, I
> > augmented Mailman's version updater to identify strings in list
> > attributes that were not valid utf-8 and convert them from the former
> > character set.
> >
> > This is an imperfect process. It does guarantee that all the strings in
> > list attributes are valid utf-8 encodings, but not necessarily that they
> > are 'correct'.
> >
> > I am working on a stand-alone script to do this. I will post it later
> today.
>
>
> There is now a first cut at this script. The functions that do the
> recoding are lifted straight from Mailman/versions.py so they have been
> fairly well exercised. I didn't just import them because I wanted the
> caller to know if they had changed anything and that required a minor
> addition to the top level function to return a flag.
>
> The script is at <https://www.msapiro.net/scripts/recode_list> (mirrored
> at <http://fog.ccsf.edu/~msapiro/scripts/recode_list>)
>
> Install the script in Mailman's bin/ directory and run it with the -h
> option for full info. It uses Python's argparse function and as such
> requires python 2.7.
>
> In your case, to convert all Finnish preferred_language lists strings
> from iso-8859-1 encoding to utf-8
>
> recode_list --language=fi
>
> would suffice. There are options to specify other than iso-8859-1 to
> utf-8 and to produce output about skipped or unchanged lists, but the
> defaults should be good for you and will report which lists were
> changed. In no case does it report the actual changes, just what lists
> were changed.
>
> I recommend making backups of your lists/*/config.pck files before
> running this in case something goes wrong.
>
> --
> Mark Sapiro <mark at msapiro.net>        The highway is for gamblers,
> San Francisco Bay Area, California    better use your sense - B. Dylan
>