Hi all,
due to Python defaults, mailman exhibits strange behaviour when processing UTF-8 emails. When no header/footer is configured, mailman passes UTF-8 emails in original form, i.e. 8bit. However, when either header or footer is configured in mailman, it uses Python's libraries to add them and as a side effect it converts 8bit emails into 7bit base64 encoded ones.
This is highly undesirable in some cases. For instance, mailinglist might be used to distribute trouble tickets or other content which is expected to be easily parsable by automated text-based utilities. With base64, emails grow in size by 33 % and such emails are getting much higher spam scores since base64 is typically used by spammers to obfuscate the payload. There are of course much more reasons for not using base64 as the primary encoding method for UTF-8 email.
The fix is quite simple and is already widely used by other projects. All that needs to be done is to redefine Python's UTF-8 charset properties, i.e. in every place where you have
from email.Charset import Charset
you need to add:
email.Charset.add_charset('utf-8',email.Charset.SHORTEST, None, None)
With such setting, mailman will keep the 8bit encoding also when it's adding header/footer and won't downconvert to 7bit+base64. So I'd like to propose the above addition, at least as a configurable option if there's any fear that enabling it by default could cause some problems.
Thanks, Petr