Ben Gertzfield escribió:

...Does anyone have any comments? I would like to go through the catalogs and change the HTML escapes back into the original characters, so that emails Mailman generates are correct again...

Please, don't change any entity. I will do that. My changes on spanish catalog will be finished next week and then I will send them to Barry. Cheers -- ___ / F \ [[[]]]] ( O O ) #----------------0000--(_)--0000---------------# | Juan Carlos Rey Anaya (jcrey@uma.es) | | Servicio Central de informática | | Universidad de Málaga - España | #----------------------------------------------#

...

...

...

...

"Juan" == Juan Carlos Rey Anaya writes:

Ben> ...Does anyone have any comments? I would like to go through Ben> the catalogs and change the HTML escapes back into the Ben> original characters, so that emails Mailman generates are Ben> correct again... Juan> Please, don't change any entity. I will do that. My changes Juan> on spanish catalog will be finished next week and then I Juan> will send them to Barry. Great! Thanks very much, Juan. Ben -- Brought to you by the letters Z and J and the number 4. "You should be glad you don't have diaper rash. Mah Jongg." Debian GNU/Linux maintainer of Gimp and Nethack -- http://www.debian.org/

Ben Gertzfield writes:

Actually, to be precise, HTML 4.01's native encoding is Unicode, which Latin-1 happens to be a (very small) subset of.

To be really precise, HTML 4.01's "document character set" is the "Universal Character Set" (as defined in ISO 10646), see http://www.w3.org/TR/html4/charset.html What the character encoding is is a different matter (Unicode is not a character encoding); that is transmitted as part of the HTTP response. As the document above points out, the default encoding, if none is specified, is Latin-1 (they also point out that it is bad to rely on that).

Unfortunately, as much as I'd like, we can't make *everything* Unicode, because a lot of older browsers still don't support it.

That is completely irrelevant; Unicode is *not* a character encoding. In this context, it is a Python internal datatype. When producing HTML document, strings of that type need to be encoded in the target document encoding (which definitely will *not* be Unicode, but perhaps a Unicode encoding, such as UTF-8, or some other encoding).

Which East Asian ones are missing? Mailman CVS works beautifully for me with Japanese, and the screenshot I sent earlier today shows Chinese (both simplified and traditional) working in email.

Python does not currently include codecs for iso-2022-jp, gb2312, big5, euc-jp, shift-jis. Since mailman leaves all strings as-is, and never mixes encodings, it can let them pass through unmodified. There are a number of pitfalls, though: - On mailing lists, people may use different encodings; some of the common combinations might be: European languages: ISO-8859-1, ISO-8859-15 (for the Euro), UTF-8 Japanese: ISO-2022-JP, eucJP, shift-jis, UTF-8 Chinese: gb2312, big5 This is probably an archive problem only; however, if mailman adds a footer, it will produce garbage if the footer encoding differs from the message body encoding. - To analyse the subject, Mailman needs to strip off the subject_prefix from the incoming message. If the message uses a MIME-encoded header, it may be that the subject prefix is base64 encoded. Currently, mailman fails to strip the prefix in this case. There is a patch on SF that tries to decode the subject. If the encoding is not known to Python, this will still fail. - To produce HTML pages, mailman needs to quote markup characters. For some encodings (e.g. iso-2022-jp), HTML markup character such as '<' may also occur as part of the multi-byte encoding. For these encodings, mailman currently performs no quoting at all. This is incorrect if an iso-2022-jp message contains a true '<' character, which would need to be converted to '<'.

The Japanese codec is in a good state and will be easy enough to ship; the Chinese ones are only available in CVS that I know of, so we will need to make a proper distribution.

I'd encourage you to have a look at the iconv codec also. If the system iconv is powerful enough (e.g. on Linux glibc), all encodings of the world would be supported with that single codec. Regards, Martin

Ben Gertzfield writes:

As for the footer.. hm. Needs more thought. I doubt anyone wants to add an attachment for the footer; I think the best thing to do would be to look up the body's charset in a table and attach a properly localized footer if it's found. If it's not found, no footer is attached. If the charset is not specified, assume us-ascii.

What do you think?

Sounds good. If the necessary codecs are available, one might try to recode the footer in the charset of the message, if no properly encoded footer is available.

Martin> I'd encourage you to have a look at the iconv codec Martin> also. If the system iconv is powerful enough (e.g. on Martin> Linux glibc), all encodings of the world would be Martin> supported with that single codec.

Ah, if only all systems had such an iconv codec.

Should that stop you from using iconv where available? On a Debian system, you'll know it is present :-)

I'm surprised iconv is so powerful on Linux glibc, yet gettext does not support iso-2022-jp directly.

I'm not surprised. The traditional encoding on Unix is eucJP; and gettext/libc can transparently recode the message to any target format. So the catalog could be in eucJP, or utf-8, and you still could produce messages in iso-2022-jp. I just tried it on a catalog; msgfmt complained about two problems: for one thing, it complains that iso-2022-jp is not a portable character set name, i.e. that many systems apparently don't recognize this character set name. The other problem is that it complains about illegal escape sequences. I don't know where this comes from; it might be a gettext mbcs bug, or a glibc bug, or an error in my data. If you have real data which are not properly processed by msgfmt, please report that bug to Bruno Haible. I can't see any reason why gettext(3) would have any difficulties with iso-2022-jp. Regards Martin

Hi,

I'm not surprised. The traditional encoding on Unix is eucJP; and gettext/libc can transparently recode the message to any target format. So the catalog could be in eucJP, or utf-8, and you still could produce messages in iso-2022-jp.

I just tried it on a catalog; msgfmt complained about two problems: for one thing, it complains that iso-2022-jp is not a portable character set name, i.e. that many systems apparently don't recognize this character set name. The other problem is that it complains about illegal escape sequences. I don't know where this comes from; it might be a gettext mbcs bug, or a glibc bug, or an error in my data. If you have real data which are not properly processed by msgfmt, please report that bug to Bruno Haible.

I can't see any reason why gettext(3) would have any difficulties with iso-2022-jp.

iso-2022-jp conatins " and \ in the second byte. Big5 also. I think iso-2022-jp is only for message transportation and not good for text manipulation like within Mailman. I still recommend all the messages and templates should be encoded in EUC-JP within Mailman and Web Interfaces, and converting them into iso-2022-jp when mail is in and out. -- Tokio Kikuchi, tkikuchi@ is.kochi-u.ac.jp http://weather.is.kochi-u.ac.jp/

Ben Gertzfield wrote:

...

...

...

...

...

"Tokio" == Tokio Kikuchi writes:

Tokio> I still recommend all the messages and templates should be Tokio> encoded in EUC-JP within Mailman and Web Interfaces, and Tokio> converting them into iso-2022-jp when mail is in and out.

This is exactly what the patch I sent to the list yesterday does.

Well then, why do you need iso-2022-jp in gettext messages ? -- Tokio Kikuchi

Ben Gertzfield writes:

Martin> I can't see any reason why gettext(3) would have any Martin> difficulties with iso-2022-jp.

It's possible that " could exist within a iso-2022-jp string, I believe; this would probably throw off gettext's parsing when you get a string like:

That would be a msgfmt(1) problem, right? I doubt gettext(3) has any problems if you manage to compile the catalog. As for the msgfmt problem, I think msgfmt uses the mbcs routines of the C library, which should be capable of distinguishing between a byte that is part of a multi-byte character, and an individual ASCII byte. So I think this should be implementable without any change to the syntax; if it still fails, you might want to submit a bug report. Regards, Martin

Quoting Tokio Kikuchi :

iso-2022-jp conatins " and \ in the second byte. Big5 also.

I think iso-2022-jp is only for message transportation and not good for text manipulation like within Mailman.

I still recommend all the messages and templates should be encoded in EUC-JP within Mailman and Web Interfaces, and converting them into iso-2022-jp when mail is in and out.

The GNU gettext utilities should be able to handle these charsets without problem (provided you have gettext >= 0.10.36). There were a number of changes between 0.10.35 and 36, so that it takes the encoding into account when processing the file (so that the second byte of a multi-byte character doesn't need to be escaped; in fact, if you did escape bytes like that, the file won't work with newer gettext). Whether the python tools do the same when processing a message catalog is another matter. James. -- James Henstridge