Hey all,
I first thought about opening a bug for this, but I think it needs a small discussion first. In Mailman3 the message templates are stored on disk (welcome.txt, footer-generic.txt, ...) However, unless I missed something, there is no hint about the encoding of those files. As a result, when I try the decorate() function (from mailman.handlers.decorate) on a non-ascii file, it crashes with a classic UnicodeDecodeError: 'ascii' codec can't decode byte [...]. Note: if the file contains no string to replace (like $fqdn_listname), it is passed through unchanged and it works.
So how should we deal with this? I think that the TemplateLoader in mailman.app.templates should return unicode strings, because that's the closest to the moment when files are read, and unicode conversion should happen on the "external borders" of the application. Thus the TemplateLoader's get() method seems to be the right place. I see two options:
We require that all template files are stored in either ascii or utf-8. That's the easiest way to go, and we just decode the text after getting the file.
We use the fact that our Language entities contain encoding values. When the template is loaded from an internal URL containing the language, we add the corresponding encoding to the result metadata (to be retrieved with the info() method) and use that to decode the contents. This means that templates in non-localized directories still have to be ascii-only, and the same goes for templates retrieved from non-internal URLs (not starting with mailman://). It's more complex but it may seem more natural to the administrator, since he won't have to force UTF-8 encoding when editing a file. We must also think of not making it too hard for Postorius, which will probably only get UTF-8 posted from the webpage (since it's always displayed in UTF-8 IIRC).
I would vote for the easy way for just requiring UTF-8 encoding, but I'd like to hear your thoughts on this.
Thanks, Aurélien
http://aurelien.bompard.org ~~~~~~ xmpp:aurelien@bompard.org Tell me and I will forget. Show me and I will remember. Involve me and I will understand. -- Chinese proverb