Polish language template files
Hi Mark, I've been in touch with the Polish Language maintainer (Stefan Plewako) regarding the non-ascii characters that I am seeing. In the language template files for Polish (Mailman 2.1.23) I see that the non-ascii characters (letters with diacritics) are replaced with question marks. These question marks get displayed in the Mailman web pages for lists using Polish. Stefan directed me to GitHub for the Polish language files and in those I files I can see the Polish letters okay. (letters with diacritics) I loaded Stefan's files into Mailman (replacing the existing) and all is now well. I was surprised as I was thinking that the non-ascii characters would need to be replaced with HTML entities - as you had done for the Hungarian files a couple of months ago. Stefan had advised me that doing that shouldn't be needed, and it seems he might be correct. Is there a mismatch with the language files that got shipped in the Mailman language files and Stefan's files on GitHub - or am I just barking up the wrong tree? I'm seeing the same issue for other languages, and I'm happy to chase down files for those if that is what is needed (assuming they have been created). Czech Greek Estonia Finnish Croation --Hungarian-- (discussed) Japanese Korean Lithuanian Polish Portuguese Slovenian Turkish I have close to zero understanding of the subject of character sets, but I include this reference below in case it has any bearing on things. https://mail.python.org/pipermail/mailman-i18n/2015-February/001854.html Regards, Mark Dale ***** SNIP from Stefan's GitHub files ***** <td colspan="2"> Wiadomości do wszystkich prenumeratorów listy wysyłaj na adres: <A HREF="mailto:<MM-Posting-Addr>"><MM-Posting-Addr></A>. <p>Możesz zapisać się na listę lub zmienić opcje prenumeraty korzystając z poniższych sekcji. </td> ***** SNIP from mailman-2.1.23/templates/pl/listinfo.html ***** <td colspan="2"> Wiadomo�ci do wszystkich prenumeratorów listy wysy�aj na adres: <A HREF="mailto:<MM-Posting-Addr>"><MM-Posting-Addr></A>. <p>Mo�esz zapisa� si� na list� lub zmieni� opcje prenumeraty korzystaj�c z poni�szych sekcji. </td> ========================================
On 01/17/2017 01:05 PM, Mark Dale wrote:
Hi Mark,
I've been in touch with the Polish Language maintainer (Stefan Plewako) regarding the non-ascii characters that I am seeing.
In the language template files for Polish (Mailman 2.1.23) I see that the non-ascii characters (letters with diacritics) are replaced with question marks. These question marks get displayed in the Mailman web pages for lists using Polish.
Stefan directed me to GitHub for the Polish language files and in those I files I can see the Polish letters okay. (letters with diacritics)
I loaded Stefan's files into Mailman (replacing the existing) and all is now well. I was surprised as I was thinking that the non-ascii characters would need to be replaced with HTML entities - as you had done for the Hungarian files a couple of months ago. Stefan had advised me that doing that shouldn't be needed, and it seems he might be correct.
I can easily convert all the non-ascii in the Polish language templates to html entities which is the correct way to deal with this. I have been reluctant to do this for certain languages in the past because of the sheer numbers of html entities involved, essentially every character in Greek for example. Polish is not so bad, but the majority of non-ascii characters have only numeric html entities. For example, the snippet you quote becomes
<td colspan="2"> Wiadomości do wszystkich prenumeratorów listy wysyłaj na adres: <A HREF="mailto:<MM-Posting-Addr>"><MM-Posting-Addr></A>.
<p>Możesz zapisać się na listę lub zmienić op cje prenumeraty korzystając z poniższych sekcji. </td>
which will render correctly in a browser that recognizes those entities but is no more readable to humans in other contexts than the � characters are. The underlying issue here is Mailman's character set for Polish is iso-8859-2. Mailman sends those web pages built from those templates with a Content-Type: text/html; charset=iso-8859-2 header, but some web servers are configured to override that. E.g., see <http://httpd.apache.org/docs/2.4/mod/core.html#adddefaultcharset> for a description of the Apache directive. Stefan's templates are UTF-8 encoded and the html templates will work in an environment where the web server 'forces' utf-8, but the .txt templates if utf-8 encoded will break in a Mailman whose character set for Polish is still iso-8859-2, because they will be sent in email with Content-Type: text/plain; charset=iso-8859-2 but with utf-8 encoded characters. The ultimate solution is to make everything utf-8 encoded. Individual sites can do this, but I can't for the reasons discussed at <https://mail.python.org/pipermail/mailman-i18n/2015-February/001854.html>. Also see see the thread "Encoding problem with 2.15 to 2.18 upgrade with Finnish" beginning at <https://mail.python.org/pipermail/mailman-users/2015-December/080221.html> and continuing at <https://mail.python.org/pipermail/mailman-users/2016-January/080275.html> for some of the fallout after Debian arbitrarily changed the character set for several languages to utf-8 in their Mailman package. Bottom line is I have converted the Polish html templates to use html entities at <http://bazaar.launchpad.net/~mailman-coders/mailman/2.1/revision/1688> and will install those at mail.python.org with the intent of releasing that with 2.1.24. It should be OK, but if I get pushback from the Polish lists on mpo, I may have to reverse. -- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Wiadomość napisana przez Mark Sapiro <mark@msapiro.net> w dniu 18.01.2017, o godz. 00:56:
Bottom line is I have converted the Polish html templates to use html entities at <http://bazaar.launchpad.net/~mailman-coders/mailman/2.1/revision/1688> and will install those at mail.python.org with the intent of releasing that with 2.1.24. It should be OK, but if I get pushback from the Polish lists on mpo, I may have to reverse.
Wow! Even if this change may not break anything I'm not sure that it solves any problem and quite frankly, I don't like seeing changes being made this way… stef
On 01/18/2017 05:45 AM, Stefan Plewako wrote:
Even if this change may not break anything I'm not sure that it solves any problem
It solves the problem of garbled characters in the web UI in installations that maintain Mailman's character set for Polish as iso-8859-2 but have web servers that force utf-8 or other non-iso-8859-2 charactersets
and quite frankly, I don't like seeing changes being made this way…
I don't understand. Can you elaborate on what exactly you don't like and what you would prefer? -- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Wiadomość napisana przez Mark Sapiro <mark@msapiro.net> w dniu 18.01.2017, o godz. 18:45:
It solves the problem of garbled characters in the web UI in installations that maintain Mailman's character set for Polish as iso-8859-2 but have web servers that force utf-8 or other non-iso-8859-2 charactersets
Is it? Do I understand correctly that Mailman will on the fly detect encoding enforced by web server and convert texts provided for placeholders (also content?) in changed templates to that enforced encoding? I'm sure there are other good questions.
I don't understand. Can you elaborate on what exactly you don't like and what you would prefer?
I don't like rushed, not exactly understood changes to work of others without even attempting to consult them.
On 01/18/2017 10:21 AM, Stefan Plewako wrote:
Wiadomość napisana przez Mark Sapiro <mark@msapiro.net> w dniu 18.01.2017, o godz. 18:45:
It solves the problem of garbled characters in the web UI in installations that maintain Mailman's character set for Polish as iso-8859-2 but have web servers that force utf-8 or other non-iso-8859-2 charactersets
Is it? Do I understand correctly that Mailman will on the fly detect encoding enforced by web server and convert texts provided for placeholders (also content?) in changed templates to that enforced encoding? I'm sure there are other good questions.
No. Mailman does not detect the encoding enforced by the web server. Mailman builds pages from the templates as provided and sends them with a Content-Type header specifying the encoding as Mailman's configured character set for the language. The problem occurs when the web server tries to enforce a different character encoding causing the encoding specified to the browser to not match that of the page.
I don't understand. Can you elaborate on what exactly you don't like and what you would prefer?
I don't like rushed, not exactly understood changes to work of others without even attempting to consult them.
I have explained in the past my reasons for not accepting your utf-8 encoded templates and message catalog as is and have always recoded them to iso-8859-2. You have been informed of this. See, e.g., <https://mail.python.org/pipermail/mailman-i18n/2015-February/001854.html>. What I have done here is only an extension of that recoding which replaces iso-8859-2 encoded characters in html templates with equivalent html entities. Please understand that I appreciate your work in keeping the Polish translation up to date and I hope you continue, but there are various technical problems that can occur that would be disruptive to existing Polish language lists if I just arbitrarily switched Mailman's encoding for Polish to utf-8. -- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Wiadomość napisana przez Mark Sapiro <mark@msapiro.net> w dniu 18.01.2017, o godz. 20:36:
On 01/18/2017 10:21 AM, Stefan Plewako wrote:
Wiadomość napisana przez Mark Sapiro <mark@msapiro.net> w dniu 18.01.2017, o godz. 18:45:
It solves the problem of garbled characters in the web UI in installations that maintain Mailman's character set for Polish as iso-8859-2 but have web servers that force utf-8 or other non-iso-8859-2 charactersets
Is it? Do I understand correctly that Mailman will on the fly detect encoding enforced by web server and convert texts provided for placeholders (also content?) in changed templates to that enforced encoding? I'm sure there are other good questions.
No. Mailman does not detect the encoding enforced by the web server. Mailman builds pages from the templates as provided and sends them with a Content-Type header specifying the encoding as Mailman's configured character set for the language. The problem occurs when the web server tries to enforce a different character encoding causing the encoding specified to the browser to not match that of the page.
OK, so the change doesn't solve any problem that I know off and especially not the problem of garbled characters in the web UI. In fact it hides the real problem making discovering it harder (not much but still) as some (touched) characters will display correctly.
I don't understand. Can you elaborate on what exactly you don't like and what you would prefer?
I don't like rushed, not exactly understood changes to work of others without even attempting to consult them.
I have explained in the past my reasons for not accepting your utf-8 encoded templates and message catalog as is and have always recoded them to iso-8859-2. You have been informed of this. See, e.g., <https://mail.python.org/pipermail/mailman-i18n/2015-February/001854.html>.
Yes and I have no idea how that relates to the current topic.
What I have done here is only an extension of that recoding which replaces iso-8859-2 encoded characters in html templates with equivalent html entities.
In my opinion you fail to accept that in best case you didn't change nothing visibly, in worst case you introduced new bugs.
Please understand that I appreciate your work in keeping the Polish translation up to date and I hope you continue, but there are various technical problems that can occur that would be disruptive to existing Polish language lists if I just arbitrarily switched Mailman's encoding for Polish to utf-8.
Quite frankly I doubt that and please stop suggesting that I requested switching default encoding for Polish to UTF-8 again (*after* the mentioned discussion in early 2015).
On 01/18/2017 12:39 PM, Stefan Plewako wrote:
OK, so the change doesn't solve any problem that I know off and especially not the problem of garbled characters in the web UI. In fact it hides the real problem making discovering it harder (not much but still) as some (touched) characters will display correctly.
Perhaps I don't understand, but the OP in this thread had a problem with garbled characters in the web UI of a Polish language list which he said was solved by installing your utf-8 encoded templates. This says to me that he has a Mailman installation where the character set for Polish is iso-8859-2, and his web server was overriding that and forcing utf-8 thus sending iso-8859-2 encoded characters to the browser and claiming they were utf-8. This issue can be fixed in general by encoding the problem characters as html entities which will render properly in the browser regardless of the encoding of the page. That is the change I made. Granted, there may still be problems with characters from strings in the message catalog or in list attributes, but these won't be fixed by encoding the template as utf-8 either.
What I have done here is only an extension of that recoding which replaces iso-8859-2 encoded characters in html templates with equivalent html entities.
In my opinion you fail to accept that in best case you didn't change nothing visibly, in worst case you introduced new bugs.
I think in the best case, I have addressed the issue described above, and if I have introduced new bugs, I hope they will be reported and I can either fix them or reverse my change.
Please understand that I appreciate your work in keeping the Polish translation up to date and I hope you continue, but there are various technical problems that can occur that would be disruptive to existing Polish language lists if I just arbitrarily switched Mailman's encoding for Polish to utf-8.
Quite frankly I doubt that and please stop suggesting that I requested switching default encoding for Polish to UTF-8 again (*after* the mentioned discussion in early 2015).
I'm sorry you doubt that and I'm sorry that I implied you were requesting switching the default encoding for Polish to UTF-8. -- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Wiadomość napisana przez Mark Sapiro <mark@msapiro.net> w dniu 18.01.2017, o godz. 22:46:
On 01/18/2017 12:39 PM, Stefan Plewako wrote:
OK, so the change doesn't solve any problem that I know off and especially not the problem of garbled characters in the web UI. In fact it hides the real problem making discovering it harder (not much but still) as some (touched) characters will display correctly.
Perhaps I don't understand, but the OP in this thread had a problem with garbled characters in the web UI of a Polish language list which he said was solved by installing your utf-8 encoded templates.
After reading first post in the thread, it is not clear to me what exactly was replaced and if things were sufficiently tested after. Using files (not only templates) from https://github.com/aviarypl/mailman-l10n-pl is one way to display things correctly, using standard files with ISO-8859-2 web server encoding is another (official). I don't believe that such post alone should be sufficient to change anything.
This issue can be fixed in general by encoding the problem characters as html entities which will render properly in the browser regardless of the encoding of the page. That is the change I made.
Granted, there may still be problems with characters from strings in the message catalog or in list attributes, but these won't be fixed by encoding the template as utf-8 either.
You did change only part of the things needed and can't really change the rest - the change doesn't make real (positive) difference.
I think in the best case, I have addressed the issue described above
No, you didn't. We have non-ASCII characters in strings that replace placeholders in changed templates.
On 01/18/2017 02:20 PM, Stefan Plewako wrote:
Wiadomość napisana przez Mark Sapiro <mark@msapiro.net> w dniu 18.01.2017, o godz. 22:46:
Granted, there may still be problems with characters from strings in the message catalog or in list attributes, but these won't be fixed by encoding the template as utf-8 either.
You did change only part of the things needed and can't really change the rest - the change doesn't make real (positive) difference.
I think in the best case, I have addressed the issue described above
No, you didn't. We have non-ASCII characters in strings that replace placeholders in changed templates.
I fully agree that what I did does not address the issue of non-ascii in strings in list attributes, nor did I say nor mean to imply that it did. It also turns out that I mis-diagnosed the OP's original issue, and that it was in fact not caused by the web server enforcing a different character set, but was apparently caused by an incompletely applied Debian update. In spite of that, I don't feel it is wrong to represent non-ascii characters in html templates as html entities, and I don't think it will cause any loss of information, at least not with any browser that recognizes html entities. -- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
At the risk of adding uninformed comments into this discussion, I have installed and tested using the Polish language files in: 1. Mailman 2.1.23 release 2. Stefan's files 2. Mark's files The test was to view what a user would see when they subscribed to list that was using Polish - on the webpages and the email they received. The server is Debian running Nginx, with Postfix and Mailman 2.1.23. The email used to view the "Subscribed" emails was webmail and desktop "IceDove". When using the Mailman 2.1.23 release language files, the "question marks" appear in both the webpages and the emails. When using Stefan's files, both the webpages and emails display correctly, showing the Polish letters with their diacritics. When using Mark's files, the webpages display correctly, but the emails show the "question marks" in place of the Polish letters. I thank you both for delving into this issue and I apologise if my remarks above add nothing to the conversation. I'm way out of my depth on the subject. Regards, Mark
Perhaps I don't understand, but the OP in this thread had a problem with garbled characters in the web UI of a Polish language list which he said was solved by installing your utf-8 encoded templates.
This says to me that he has a Mailman installation where the character set for Polish is iso-8859-2, and his web server was overriding that and forcing utf-8 thus sending iso-8859-2 encoded characters to the browser and claiming they were utf-8.
On 01/18/2017 02:50 PM, Mark Dale wrote:
At the risk of adding uninformed comments into this discussion, I have installed and tested using the Polish language files in:
1. Mailman 2.1.23 release 2. Stefan's files 2. Mark's files
The test was to view what a user would see when they subscribed to list that was using Polish - on the webpages and the email they received.
Thank you for testing and reporting. I think your report is helpful.
The server is Debian running Nginx, with Postfix and Mailman 2.1.23.
The email used to view the "Subscribed" emails was webmail and desktop "IceDove".
When using the Mailman 2.1.23 release language files, the "question marks" appear in both the webpages and the emails.
I am surprised that there are issues with the emails. The emails should be sent with RFC 2047 encoded subjects similar to Subject: =?iso-8859-2?q?Witaj_na_li=B6cie_=22List1=22_?= which renders as Subject: Witaj na liście "List1" and with the header Content-Type: text/plain; charset="iso-8859-2" which declares the body character set to be iso-8859-2
When using Stefan's files, both the webpages and emails display correctly, showing the Polish letters with their diacritics.
And in this case, with utf-8 encoded email templates, the utf-8 characters should be garbled because of the charset="iso-8859-2" in the Content-Type: header.
When using Mark's files, the webpages display correctly, but the emails show the "question marks" in place of the Polish letters.
I didn't change any of the templates used in emails. Since they aren't html, html entities don't work for them. So, the change that I made replacing non-ascii iso-8859-2 characters in the html templates improves the display of those web pages which is exactly what it was intended to do. What I don't understand is why the emails are garbled with iso-8859-2 templates and not with utf-8 templates unless you changed Mailman's character set for Polish to utf-8. If in fact you did that, then you do need utf-8 encoded templates and message catalog for Polish. So the question remains, what is the character set in the definition add_language('pl', _('Polish'), 'iso-8859-2', 'ltr') in Defaults.py or overridden in mm_cfg.py. If it is 'iso-8859-2' as shown and not 'utf-8' I am at a loss to understand why your emails are garbled with iso-8859-2 templates and not with utf-8 templates. -- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
I checked on two servers where I have tested (both with the same setup of Debian, Nginx, etc) and gotten the same results. I haven't made changes any language character settings on either. Both are out-of-the box regarding that. In Defaults.py for both, I see: add_language('pl', _('Polish'), 'iso-8859-2', 'ltr') add_language('pl', _('Polish'), 'iso-8859-2', 'ltr') And in mm_cfg.py, the default language is "en" on both, and there are no over-rides. If there is any testing I can do to that might contribute to some understanding, I'm happy to do so. On 19/01/17 10:31, Mark Sapiro wrote:
What I don't understand is why the emails are garbled with iso-8859-2 templates and not with utf-8 templates unless you changed Mailman's character set for Polish to utf-8. If in fact you did that, then you do need utf-8 encoded templates and message catalog for Polish.
So the question remains, what is the character set in the definition
add_language('pl', _('Polish'), 'iso-8859-2', 'ltr')
in Defaults.py or overridden in mm_cfg.py. If it is 'iso-8859-2' as shown and not 'utf-8' I am at a loss to understand why your emails are garbled with iso-8859-2 templates and not with utf-8 templates.
On 01/18/2017 04:18 PM, Mark Dale wrote:
In Defaults.py for both, I see:
add_language('pl', _('Polish'), 'iso-8859-2', 'ltr') add_language('pl', _('Polish'), 'iso-8859-2', 'ltr')
And in mm_cfg.py, the default language is "en" on both, and there are no over-rides.
If there is any testing I can do to that might contribute to some understanding, I'm happy to do so.
What are the headers from a list welcome or other similar message from a Polish language list? -- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 19/01/17 11:41, Mark Sapiro wrote:
What are the headers from a list welcome or other similar message from a Polish language list?
On the list-welcome emails the content-type in the header is: Content-Type: text/plain; charset="utf-8" It is the same for both the emails, the ones that display question marks and the one that displays the Polish letters. The full headers are: *************************************** **** EMAIL SHOWING QUESTION MARKS ***** ***************************************
From - Thu Jan 19 08:49:02 2017 X-Account-Key: account14 X-UIDL: 1484776110.24176.takara,S=4707 X-Mozilla-Status: 0001 X-Mozilla-Status2: 00000000 X-Mozilla-Keys:
From singapore-bounces@mailmanlists.sg Wed Jan 18 22:48:30 2017 Return-path: <singapore-bounces@mailmanlists.sg> Received: from [10.9.9.210] (helo=mailfront10.runbox.com) by takara.runbox.com with esmtp (Exim 4.69) id 1cTy5m-0006Hs-CK for mark%mailmanlists.net@runbox.com; Wed, 18 Jan 2017 22:48:30 +0100 Received: from exim by mailfront10.runbox.com with dspam-scanned (Exim 4.82) id 1cTy5k-0004cF-2H for mark%mailmanlists.net@runbox.com; Wed, 18 Jan 2017 22:48:30 +0100 Received: from exim by mailfront10.runbox.com with sa-scanned (Exim 4.82) id 1cTy5j-0004bx-LZ for mark%mailmanlists.net@runbox.com; Wed, 18 Jan 2017 22:48:28 +0100 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on antispam02.runbox.com X-Spam-Level: X-Spam-Status: No, score=0.1 required=4.0 tests=DKIM_SIGNED,T_DKIM_INVALID shortcircuit=no autolearn=disabled version=3.4.1 Received: from mailmanlists.sg ([128.199.127.214] helo=www.mailmanlists.sg) by mailfront10.runbox.com with esmtp (Exim 4.82) id 1cTy4m-0003rH-Jv for mark@mailmanlists.net; Wed, 18 Jan 2017 22:47:29 +0100 Received: from [127.0.1.1] (localhost [127.0.0.1]) by www.mailmanlists.sg (Postfix) with ESMTP id 44E9C5FECA for <mark@mailmanlists.net>; Thu, 19 Jan 2017 05:47:27 +0800 (SGT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=mailmanlists.sg; s=mail; t=1484776047; bh=vTkBfI6zV9V+cQAIFmIAjSkXSmNYsyevuuwu6J7GWeA=; h=Subject:From:To:Date:List-Id:From; b=k30Je9DQdskC8gJbCs6jDmenwwb685FOPNikNufnONzJaDxILEuNSAbsINpm2LK4p Ct+zlNK+fGcf6JoDgP5QvJWvh8dCHYH7mE4YppyUAbZABusunRwEcM3V4ktRsjbMDt jqvHPUIaw1a4yDGj4gixH/ywWOloXXv7jUyBrlL0= MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Subject: =?utf-8?q?Witaj_na_li=C5=9Bcie_=22Singapore=22_?= From: singapore-request@mailmanlists.sg To: mark@mailmanlists.net X-No-Archive: yes Message-ID: <mailman.0.1484776046.22108.singapore@mailmanlists.sg> Date: Thu, 19 Jan 2017 05:47:26 +0800 Precedence: bulk X-BeenThere: singapore@mailmanlists.sg X-Mailman-Version: 2.1.23 List-Id: <singapore.mailmanlists.sg> X-List-Administrivia: yes Errors-To: singapore-bounces@mailmanlists.sg X-FILTER-DSPAM: by mailfront10.runbox.com X-DSPAM-Factors: 27, To*mailmanlists+net, 0.00010, To*mark+mailmanlists, 0.00010, X-FILTER-DSPAM*runbox+com, 0.00010, Content-Type*text, 0.00010, X-FILTER-DSPAM*com, 0.00010, X-FILTER-DSPAM*by, 0.00010, X-FILTER-DSPAM*runbox, 0.00010, Content-Type*charset, 0.00010, To*mailmanlists, 0.00010, To*mark, 0.00010, To*net, 0.00010, mailmanlists, 0.00020, mailmanlists, 0.00020, do, 0.00020, do, 0.00020, www, 0.00020, www, 0.00020, X-FILTER-DSPAM*mailfront10+runbox, 0.00020, X-FILTER-DSPAM*by+mailfront10, 0.00020, Content-Type*charset+utf, 0.00020, DKIM-Signature*v1a+rsa, 0.00020, Content-Type*plain+charset, 0.00020, Content-Type*text+plain, 0.00020, X-FILTER-DSPAM*mailfront10, 0.00020, DKIM-Signature*To, 0.00020, Date*Jan, 0.00020, net, 0.00020 X-DSPAM-Result: Innocent X-DSPAM-Confidence: 0.9997 X-DSPAM-Probability: 0.0000
V2l0YW15IG5hIGxptmNpZSBTaW5nYXBvcmVAbWFpbG1hbmxpc3RzLnNnIQoKV2lhZG9tb7ZjaSBz *************************************** **** EMAIL SHOWING POLISH LETTERS ***** ***************************************
From - Thu Jan 19 09:09:43 2017 X-Account-Key: account14 X-UIDL: 1484777340.16071.takara,S=4770 X-Mozilla-Status: 0001 X-Mozilla-Status2: 00000000 X-Mozilla-Keys:
From singapore-bounces@mailmanlists.sg Wed Jan 18 23:09:00 2017 Return-path: <singapore-bounces@mailmanlists.sg> Received: from [10.9.9.211] (helo=mailfront11.runbox.com) by takara.runbox.com with esmtp (Exim 4.69) id 1cTyPc-0004B9-Hq for mark%mailmanlists.net@runbox.com; Wed, 18 Jan 2017 23:09:00 +0100 Received: from exim by mailfront11.runbox.com with dspam-scanned (Exim 4.82) id 1cTyPb-0005m8-4n for mark%mailmanlists.net@runbox.com; Wed, 18 Jan 2017 23:09:00 +0100 Received: from exim by mailfront11.runbox.com with sa-scanned (Exim 4.82) id 1cTyPa-0005lx-Th for mark%mailmanlists.net@runbox.com; Wed, 18 Jan 2017 23:08:59 +0100 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on antispam03.runbox.com X-Spam-Level: X-Spam-Status: No, score=0.1 required=4.0 tests=DKIM_SIGNED,T_DKIM_INVALID shortcircuit=no autolearn=disabled version=3.4.1 Received: from mailmanlists.sg ([128.199.127.214] helo=www.mailmanlists.sg) by mailfront11.runbox.com with esmtp (Exim 4.82) id 1cTyPH-0005dk-DU for mark@mailmanlists.net; Wed, 18 Jan 2017 23:08:40 +0100 Received: from [127.0.1.1] (localhost [127.0.0.1]) by www.mailmanlists.sg (Postfix) with ESMTP id 230A35FF5E for <mark@mailmanlists.net>; Thu, 19 Jan 2017 06:08:38 +0800 (SGT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=mailmanlists.sg; s=mail; t=1484777318; bh=ifwoNAeBj28OXU2B9hF6MqUDfqG3tN5O4yAkQUui8xk=; h=Subject:From:To:Date:List-Id:From; b=JHM4CNQgodqdYkj3hHmuOlkNYAjKvJ+rEDIJefamsytT2VWgpbcLklOf2gbD2yEzp 0TswmQcSsmGVGEJxKNUwydvJrhqg50kbmLFjSiz9Xsljus5J/v4EZ+B7a2sRS0TRA7 8IZU6oWFSRuy/KYq7wtz5lU5fB/cDi/R1tRs9mPU= MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Subject: =?utf-8?q?Witaj_na_li=C5=9Bcie_=22Singapore=22_?= From: singapore-request@mailmanlists.sg To: mark@mailmanlists.net X-No-Archive: yes Message-ID: <mailman.0.1484777317.22243.singapore@mailmanlists.sg> Date: Thu, 19 Jan 2017 06:08:37 +0800 Precedence: bulk X-BeenThere: singapore@mailmanlists.sg X-Mailman-Version: 2.1.23 List-Id: <singapore.mailmanlists.sg> X-List-Administrivia: yes Errors-To: singapore-bounces@mailmanlists.sg X-FILTER-DSPAM: by mailfront11.runbox.com X-DSPAM-Factors: 27, To*mailmanlists+net, 0.00010, To*mark+mailmanlists, 0.00010, X-FILTER-DSPAM*runbox+com, 0.00010, Content-Type*text, 0.00010, X-FILTER-DSPAM*com, 0.00010, X-FILTER-DSPAM*by, 0.00010, X-FILTER-DSPAM*runbox, 0.00010, Content-Type*charset, 0.00010, To*mailmanlists, 0.00010, To*mark, 0.00010, To*net, 0.00010, mailmanlists, 0.00020, mailmanlists, 0.00020, https+www, 0.00020, https+www, 0.00020, do, 0.00020, do, 0.00020, https, 0.00020, https, 0.00020, www, 0.00020, www, 0.00020, Content-Type*charset+utf, 0.00020, X-FILTER-DSPAM*by+mailfront11, 0.00020, DKIM-Signature*v1a+rsa, 0.00020, Content-Type*plain+charset, 0.00020, X-FILTER-DSPAM*mailfront11+runbox, 0.00020, Content-Type*text+plain, 0.00020 X-DSPAM-Result: Innocent X-DSPAM-Confidence: 0.9997 X-DSPAM-Probability: 0.0000
V2l0YW15IG5hIGxpxZtjaWUgU2luZ2Fwb3JlQG1haWxtYW5saXN0cy5zZyEKCldpYWRvbW/Fm2Np
On 01/18/2017 04:58 PM, Mark Dale wrote:
On 19/01/17 11:41, Mark Sapiro wrote:
What are the headers from a list welcome or other similar message from a Polish language list?
On the list-welcome emails the content-type in the header is:
Content-Type: text/plain; charset="utf-8"
It is the same for both the emails, the ones that display question marks and the one that displays the Polish letters.
The full headers are:
*************************************** **** EMAIL SHOWING QUESTION MARKS ***** *************************************** ... Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Subject: =?utf-8?q?Witaj_na_li=C5=9Bcie_=22Singapore=22_?=
I'm beginning to see the light here. Your Mailman installation is broken by Debian. Recent Debian Mailman packages have changed Mailman's character set for all languages to utf-8. They have changed the definition of add_language in Defaults.py from def add_language(code, description, charset, direction='ltr'): LC_DESCRIPTIONS[code] = (description, charset, direction) to def add_language(code, description, charset, direction='ltr'): LC_DESCRIPTIONS[code] = (description, 'utf-8', direction) thus ignoring the specified charset and made other changes to recode all the templates and message catalogs to utf-8. However, these changes are not sufficient as they did nothing about non-ascii string values in attributes of existing lists leading to <https://bugs.launchpad.net/mailman/+bug/1462755> and the issue in the thread mentioned at <https://bugs.launchpad.net/mailman/+bug/1462755/comments/6> So, basically in your Debian Mailman, everything is supposed to be utf-8 encoded. In your case however, something in the way the Mailman package was installed did not pick up the utf-8 encoded templates for Polish and perhaps other languages. Do you have overriding templates in a templates/site/pl or templates/<domain_name>/pl or even in lists/<listname>/pl directories that are not utf-8 encoded? So I think the bottom line here is you have Debian's <https://anonscm.debian.org/viewvc/pkg-mailman/trunk/debian/patches/91_utf8.patch?view=markup&pathrev=788> patch in your installation, but somehow you didn't have Debian's recoded utf-8 templates. -- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Thank you Mark. This has been an education. Although I may be headed in the opposite direction of what you have been saying, my thinking is that Debian is seemingly geared for UTF-8, so with that in mind I've loaded the language templates and message files (UTF-8) from the latest Debian Mailman package (extracted from the .deb file) to the server. Things now appear to be okay, with the webpages, emails, message archives - now displaying correctly for all the languages - including Greek and Japanese. I read the threads that you pointed me too, in particular the one regarding the qrunner. My understanding from reading it is that the issue of bad unicode characters causing problems doesn't apply after version 2.1.21 due to the patch you committed. Did I get that right? On 19/01/17 12:51, Mark Sapiro wrote:
Recent Debian Mailman packages have changed Mailman's character set for all languages to utf-8. They have changed the definition of add_language in Defaults.py from
Yes, I am seeing that in the Defaults.py def add_language(code, description, charset, direction='ltr'): LC_DESCRIPTIONS[code] = (description, 'utf-8', direction) ... add_language('pl', _('Polish'), 'iso-8859-2', 'ltr') ... *****
So, basically in your Debian Mailman, everything is supposed to be utf-8 encoded. In your case however, something in the way the Mailman package was installed did not pick up the utf-8 encoded templates for Polish and perhaps other languages.
This explains why I'm seeing the issue also in: Czech Greek Estonia Finnish Croation Hungarian Japanese Korean Lithuanian Polish Portuguese Slovenian Turkish
Do you have overriding templates in a templates/site/pl or templates/<domain_name>/pl or even in lists/<listname>/pl directories that are not utf-8 encoded?
No, there are no <site> or <listname> directories or files under templates/ - only the <language code> directories *****
So I think the bottom line here is you have Debian's <https://anonscm.debian.org/viewvc/pkg-mailman/trunk/debian/patches/91_utf8.patch?view=markup&pathrev=788> patch in your installation, but somehow you didn't have Debian's recoded utf-8 templates.
I've now downloaded the most recent Debian Mailman package (with the UTF-8), extracted all the language template directories and loaded them to the server (replacing all the ones that were there already). All the languages I've listed above that I was having trouble with are now all good (including Greek) - for both the web pages and the emails. https://packages.debian.org/stretch/mailman *****
On 01/18/2017 08:02 PM, Mark Dale wrote:
Thank you Mark. This has been an education.
Although I may be headed in the opposite direction of what you have been saying, my thinking is that Debian is seemingly geared for UTF-8, so with that in mind I've loaded the language templates and message files (UTF-8) from the latest Debian Mailman package (extracted from the .deb file) to the server.
That is the correct thing to do in your case.
Things now appear to be okay, with the webpages, emails, message archives - now displaying correctly for all the languages - including Greek and Japanese.
I read the threads that you pointed me too, in particular the one regarding the qrunner. My understanding from reading it is that the issue of bad unicode characters causing problems doesn't apply after version 2.1.21 due to the patch you committed. Did I get that right?
The patch that I committed to "fix" <https://bugs.launchpad.net/mailman/+bug/1462755> only addresses one thing. That one thing is shunted messages due to the list's description attribute being encoded in a character set incompatible with utf-8. It doesn't fix the fact that the list's description attribute is now incorrectly encoded, it just ignores it, so when making an email address of the form list description <list@example.com> for use in the List-Id: header and perhaps in a Reply-To: header if reply goes to the list or a Cc: header for fully personalized lists, the description is just not included. The issues caused by recoding all the templates and messages to utf-8 and changing Mailman's character set to utf-8 are that any strings in existing list attributes that are encoded in a prior, utf-8 incompatible character set will at best be garbled and at worst throw exceptions causing shunted messages. This is fixed by just re-entering the list attributes via config_list or the web UI, but this can be a big job of there are many such strings in multiple lists. There is a script at <https://www.msapiro.net/scripts/recode_list> (mirrored at <https://fog.ccsf.edu/~msapiro/scripts/recode_list> that can do this programmatically. The script will guess at what needs to be recoded and can pretty much guarantee that after use, no UnicodeError exceptions will be thrown, but it doesn't guarantee that the recoded strings are correct. Just read the beginning of the script or install it and run it with the --help option for usage.
I've now downloaded the most recent Debian Mailman package (with the UTF-8), extracted all the language template directories and loaded them to the server (replacing all the ones that were there already).
All the languages I've listed above that I was having trouble with are now all good (including Greek) - for both the web pages and the emails.
Good, but the real question is why when your Mailman package with Debian's universal utf-8 patch was installed were the message catalogs and templates not also replaced with the utf-8 encoded versions. -- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
participants (3)
-
Mark Dale -
Mark Sapiro -
Stefan Plewako