...

...

...

...

"Juan" == Juan Carlos Rey Anaya writes:

Ben> ...Does anyone have any comments? I would like to go through Ben> the catalogs and change the HTML escapes back into the Ben> original characters, so that emails Mailman generates are Ben> correct again... Juan> Please, don't change any entity. I will do that. My changes Juan> on spanish catalog will be finished next week and then I Juan> will send them to Barry. Great! Thanks very much, Juan. Ben -- Brought to you by the letters Z and J and the number 4. "You should be glad you don't have diaper rash. Mah Jongg." Debian GNU/Linux maintainer of Gimp and Nethack -- http://www.debian.org/

...

...

...

...

"Martin" == Martin von Loewis writes:

Martin> - for this to work, Mailman needs to properly declare the Martin> encoding of each generated HTML page, and the declaration Martin> needs to match the actual content. For Latin-1, this is Martin> not strictly necessary, since that is the default encoding Martin> of HTML, anyway, but there may be plans to move to XHTML Martin> some day, at which time even this assumption breaks. Actually, to be precise, HTML 4.01's native encoding is Unicode, which Latin-1 happens to be a (very small) subset of. Martin> - Problems will arise if Mailman inserts strings from Martin> various sources into the same template, especially if Martin> these use different encodings. If that can ever happen, Martin> you need to recode all strings to the same encoding. If Martin> that fails (e.g. because the encoding is unknown, or Martin> because the string cannot be represented in the encoding), Right now, I don't think Mailman does that anywhere. If it does, I think the best thing to do is to convert to Unicode. Unfortunately, as much as I'd like, we can't make *everything* Unicode, because a lot of older browsers still don't support it. Martin> This document is encoded in ISO-8859-9 (for Turkish); Martin> but it still contains French accepts. Using entities is Martin> the only choice here, short of using UTF-8 for the entire Martin> page. Yes. This kind of issue will come up only in two places in Mailman: 1) on the admin request page (for bounce handling, etc) 2) in the archives (a pipermail issue) Martin> Unfortunately, not all encodings in mailman are supported Martin> (the East Asians ones are missing). In general, I'd Martin> encourage usage of Unicode throughout in mailman, even if Martin> this means that additional codecs must be bundled with the Martin> distribution. Which East Asian ones are missing? Mailman CVS works beautifully for me with Japanese, and the screenshot I sent earlier today shows Chinese (both simplified and traditional) working in email. Barry and I have talked a lot about bundling codecs with Mailman, and he's agreed with me that we need to do it. The Japanese codec is in a good state and will be easy enough to ship; the Chinese ones are only available in CVS that I know of, so we will need to make a proper distribution. Ben -- Brought to you by the letters T and N and the number 12. "Hoosh is a kind of soup." Debian GNU/Linux maintainer of Gimp and Nethack -- http://www.debian.org/

...

...

...

...

"Martin" == Martin von Loewis writes:

Martin> To be really precise, HTML 4.01's "document character set" Martin> is the "Universal Character Set" (as defined in ISO Martin> 10646), see Yes, many thanks. Martin> Python does not currently include codecs for iso-2022-jp, Martin> gb2312, big5, euc-jp, shift-jis. Since mailman leaves all Martin> strings as-is, and never mixes encodings, it can let them Martin> pass through unmodified. There are a number of pitfalls, Martin> though: I have been working actively on these problems. Hopefully we can ship these codecs with Mailman 2.1. Martin> This is probably an archive problem only; however, if Martin> mailman adds a footer, it will produce garbage if the Martin> footer encoding differs from the message body encoding. Martin> - To analyse the subject, Mailman needs to strip off the Martin> subject_prefix from the incoming message. The subject and footer issue is a good one, and needs some work. We basically need a map of charset -> localized "Re:" prefixes; with the new email module's i18n support, it's trivial to decode headers and make sure we don't add a [PREFIX] to a message with Re: [PREFIX] in the local language. I know German uses AW: -- does anyone have a list of commonly-used response prefixes in other languages? Japanese uses Re:, as far as I know. As for the footer.. hm. Needs more thought. I doubt anyone wants to add an attachment for the footer; I think the best thing to do would be to look up the body's charset in a table and attach a properly localized footer if it's found. If it's not found, no footer is attached. If the charset is not specified, assume us-ascii. What do you think? Martin> - To produce HTML pages, mailman needs to quote markup Martin> characters. For some encodings (e.g. iso-2022-jp), HTML Martin> markup character such as '<' may also occur as part of the Martin> multi-byte encoding. For these encodings, mailman Martin> currently performs no quoting at all. This is incorrect if Martin> an iso-2022-jp message contains a true '<' character, Martin> which would need to be converted to '<'. I have written a Python module that deals with this problem directly for iso-2022-jp; it would also be possible by converting to Unicode, doing HTML escaping, then converting to the output format. http://nausicaa.interq.or.jp/mailman/JisEscape.py Martin> I'd encourage you to have a look at the iconv codec Martin> also. If the system iconv is powerful enough (e.g. on Martin> Linux glibc), all encodings of the world would be Martin> supported with that single codec. Ah, if only all systems had such an iconv codec. I'm surprised iconv is so powerful on Linux glibc, yet gettext does not support iso-2022-jp directly. Ben -- Brought to you by the letters M and W and the number 4. "Ohhhh, Mentos Boy!" Debian GNU/Linux maintainer of Gimp and Nethack -- http://www.debian.org/

...

...

...

...

"Martin" == Martin von Loewis writes:

Martin> I can't see any reason why gettext(3) would have any Martin> difficulties with iso-2022-jp. It's possible that " could exist within a iso-2022-jp string, I believe; this would probably throw off gettext's parsing when you get a string like: "\eB(stuff stuff stuff " more stuff stuff\eb" It's too bad that gettext doesn't seem to have an alternate quoting mechanism. Ben -- Brought to you by the letters G and V and the number 9. "What's different, Pete, about the 69 that makes it so exciting to you?" Debian GNU/Linux maintainer of Gimp and Nethack -- http://www.debian.org/

Hi,

I'm not surprised. The traditional encoding on Unix is eucJP; and gettext/libc can transparently recode the message to any target format. So the catalog could be in eucJP, or utf-8, and you still could produce messages in iso-2022-jp.

I just tried it on a catalog; msgfmt complained about two problems: for one thing, it complains that iso-2022-jp is not a portable character set name, i.e. that many systems apparently don't recognize this character set name. The other problem is that it complains about illegal escape sequences. I don't know where this comes from; it might be a gettext mbcs bug, or a glibc bug, or an error in my data. If you have real data which are not properly processed by msgfmt, please report that bug to Bruno Haible.

I can't see any reason why gettext(3) would have any difficulties with iso-2022-jp.

iso-2022-jp conatins " and \ in the second byte. Big5 also. I think iso-2022-jp is only for message transportation and not good for text manipulation like within Mailman. I still recommend all the messages and templates should be encoded in EUC-JP within Mailman and Web Interfaces, and converting them into iso-2022-jp when mail is in and out. -- Tokio Kikuchi, tkikuchi@ is.kochi-u.ac.jp http://weather.is.kochi-u.ac.jp/

Ben Gertzfield wrote:

...

...

...

...

...

"Tokio" == Tokio Kikuchi writes:

Tokio> I still recommend all the messages and templates should be Tokio> encoded in EUC-JP within Mailman and Web Interfaces, and Tokio> converting them into iso-2022-jp when mail is in and out.

This is exactly what the patch I sent to the list yesterday does.

Well then, why do you need iso-2022-jp in gettext messages ? -- Tokio Kikuchi

Tokio Kikuchi writes:

iso-2022-jp contains " and \ in the second byte. Big5 also.

That should not be a problem: the MBCS routines of the C library (or, rather, iconv) should be able to take care of that (if the encoding is known); so the problem can be solved. I know it isn't, but anybody interested in solving it could either report a bug or maybe even design a patch. Regards, Martin