Barry A. Warsaw wrote:

...

...

...

...

...

"BG" == Ben Gertzfield writes:

BG> When submitting an HTML form, the character set used for the BG> submitted data is the same as the one specified in the HTML or BG> header of the original form's page.

If they were subscribed via email, we'd already have the encoded form of their real name.

What's left are the command line and mass subscribe page (both the text box and the file upload). In these cases should we simply reject addresses with non-ascii real names? That'd mean they'd have to be encoded prior to being subscribed.

As far as the command-line goes, we should probably reject non-ASCII real names, yes. (It MIGHT be possible to parse the various LANG/LC_CHARSET environment variables and guess the character set, but that's a pain.) The mass subscribe page case should be the same as any other HTML form, right? Whatever character set the original form's page used is what all the real names' character sets get set to. Ben

...

...

...

...

"BG" == Ben Gertzfield writes:

BG> When submitting an HTML form, the character set used for the BG> submitted data is the same as the one specified in the HTML or BG> header of the original form's page. I must be dense because I'm not quite seeing how this will work. I visit the mass subscribe page and in the text box, I enter a funny name, e.g. barry@python.org (Barry Wârsaw) My list is conducted in English. Now when I look at all the data submitted by the form, I don't see anything immediately useful in either the cgi environment or in the form data. Here are some excerpts: CONTENT_TYPE: multipart/form-data; boundary=---------------------------527473093431726113359136092 Hmm, nothing there. HTTP_ACCEPT_CHARSET: ISO-8859-1, utf-8;q=0.66, *;q=0.66 That doesn't really help us does it? That's telling us what charsets the browser will accept, right? Not the same thing. Now for the form data, I'll see a section like: -----------------------------527473093431726113359136092 Content-Disposition: form-data; name="subscribees" warsaw@wooz.org (Barry Wârsaw) -----------------------------527473093431726113359136092 This doesn't tell me enough either does it? So I'm at a loss as to where to such the information out of the form data or the environment to figure out what charset the form was posted in. -Barry

barry@zope.com (Barry A. Warsaw) writes:

I must be dense because I'm not quite seeing how this will work. [...] This doesn't tell me enough either does it?

You are running into one of the most awful oddities of HTTP and i18n. In short, the encoding of the page that contained the form was used to encoding the form contents :-( The RFC says the browser SHOULD declare the encoding for each field in the per-field MIME header of multipart/form-data message. None of the browsers does that. I filed bug reports for all of them, and Mozilla people responded that they can't do that because many CGI scripts break when they get a charset= (it won't fit their regexp). The RFC says, as a fall-back, the browser should use the encoding of the HTML page which contained the form. Mailman doesn't declare a charset in the administrative pages, but it should. It may happen that the user enters a character which cannot be represented in the charset of the page. In this case, Mozilla sends a '?' (question mark), so you can only tell that there was a character, but not which one. Internet Exploder sends a HTML entity, which gives you more information, but is undistinguishable from the case where the user entered an ampersand-digits sequence. For Mailman, this gives two options: 1. Each administrative page should be encoded in the list's "native" charset. This will allow to add names in that charset. 2. Each page should be encoded in UTF-8. This will allow to enter arbitrary names, but will require recoding to the list's charset later (or using UTF-8 in the To: fields as well). Actually, it appears that mailman already does 1, in the HTTP header. Barry, what is the charset of your admin pages? Regards, Martin

To follow up, I believe I have this working now. Here's how it works. First, the only change to the MemberAdaptor API is that real names can now be Unicode strings as well as 8-bit strings. If they're 8-bit then they'll contain only ascii characters. When a real name is entered into a web form, we'll first attempt to convert it to us-ascii. If that succeeds, we know the real name is ascii only and we'll store it in the membership database as an 8-bit ascii-only-containing string. If the conversion fails, we'll convert the real name to Unicode using the charset of the context's language (i.e. list preferred if we're looking at an admin page, user preferred if we're looking at an options page, and form value if we're looking at the subscribe page -- all with appropriate fallbacks to Something Sensible). We'll also do html entity replacement (e.g. #&246; -> ö). We'll store this Unicode string as the member's real name in the membership database, but we don't store the charset because... ...when we need to get a printable version of the member name, we yank out either the ascii string or the Unicode string. If it's ascii, we're done. If it's Unicode, then we try to encode it to the charset of the web page we're printing (for cgi), or to the charset of the outgoing email message. For output web pages, if the encoding fails, we'll convert chars > 127 to html entities (e.g. ö -> #&246;) so in most cases we'll still see the name with the proper characters. For this case, think about a user who selects Spanish, enters a ñ in their name, and then switches their preferred language to English (us-ascii). You'd like their name to still show up correctly. For email, if the name has non-ascii characters in it, we'll use the email.Header.Header class to convert the To string to an RFC-compliant format. If that fails we fall back on encoding to us-ascii replacing non-ascii characters with `?'s. This seems to work fairly well (with some ugly changes also necessary to the logging system), with one minor kludge. I want to allow non-ASCII characters in real names for English lists. I'm nervous about changing the default charset for English from us-ascii because I'm superstitious about unintended side-effects. So I'm making a couple of special cases for us-ascii. When decoding a string from a web form, if the default charset would be us-ascii, I'll use iso-8859-1 instead. Then when encoding a name in an email header, if the charset is us-ascii, again, I'll use iso-8859-1. This seems like a practical compromise, if a bit ugly. Feedback is welcome. I'm about to check all this stuff in. Testing will be /greatly/ appreciated! -Barry

...

...

...

...

"BG" == Ben Gertzfield writes:

>> To follow up, I believe I have this working now. Here's how it >> works. BG> Thanks for the excellent explanation and implementation, BG> Barry. Took me two days. I still say Unicode is something everyone wants until they get it. :) BG> I'll test this when it's checked in. Some comments below.. Excellent! >> First, the only change to the MemberAdaptor API is that real >> names can now be Unicode strings as well as 8-bit strings. If >> they're 8-bit then they'll contain only ascii characters. BG> ASCII is by definition 7-bit, Barry. Did you mean ISO-8859-1 BG> here? Sorry, I meant "normal" Python strings (sometimes called "8-bit strings") but which contain only 7-bit ascii characters. Those beasties I don't convert to Python unicode strings. >> When a real name is entered into a web form, we'll first >> attempt to convert it to us-ascii. If that succeeds, we know >> the real name is ascii only and we'll store it in the >> membership database as an 8-bit ascii-only-containing string. BG> Again, I assume you mean ISO-8859-1 instead of ascii here. Same thing here. We do name.encode('us-ascii') and catch any UnicodeError that might occur. If no error occurs, we know we have a string with 7-bit ascii characters in it, so we store that as an 8-bit Python string, not as a unicode Python string. >> If the conversion fails, we'll convert the real name to Unicode >> using the charset of the context's language (i.e. list >> preferred if we're looking at an admin page, user preferred if >> we're looking at an options page, and form value if we're >> looking at the subscribe page -- all with appropriate fallbacks >> to Something Sensible). We'll also do html entity replacement >> (e.g. #&246; -> ö). We'll store this Unicode string as the >> member's real name in the membership database, but we don't >> store the charset because... BG> This is a good thing. Note that some browsers might (I BG> haven't checked this) incorrectly send the entity &246; for BG> whatever character is at position 246 in the user's default BG> character set, not character 246 in Unicode. This might be BG> something to look out for, but I don't know if it's important. I don't know what else to do. Note that you could literally type ö into the web form and it would have the same effect. This is probably an 80/20 solution. BG> Everything else looks good. The kludge to assume iso-8859-1 BG> on us-ascii pages is unfortunately a generally good one, as BG> that will make the most people happy. I hate to do it, BG> though! Me too! It means that names in other charsets will be screwed on English lists, but again, I think this is best we can do for a practical 80/20 solution. Thanks for the feedback. -Barry

...

...

...

...

"TK" == Tokio Kikuchi writes:

TK> I had to add "import japanese" in Utils.py. Otherwise, I got TK> unknown charset error when I entered japanese in fullname. Do you have a traceback? Can you send me a valid Japanese name string so that I can try it myself? I would have thought that the codecs package would have imported the Japanese codecs automatically. If not, we have to figure out the right way to hook it up so it works. If adding the import is the only way to do it, then we'll go that route, but I'd like to try to reproduce the problem here. TK> And, "korean" ? Where should I put them ? We probably have to handle this the same way we handle Japanese, but I'm not sure what the right way to do that is at the moment. Thanks, -Barry

Thanks for the feedback. I think I've got the Japanese support working in cvs now. But then, I'm pretty tired, so please double check! -Barry

Tokio Kikuchi wrote:

Good night.

Barry A. Warsaw wrote:

...

Thanks for the feedback. I think I've got the Japanese support working in cvs now. But then, I'm pretty tired, so please double check!

Well, things are not so easy. I get errors here and there. It's 5 in Japan and I have to feed my kid and puppy. later, -- Tokio Kikuchi, tkikuchi@ is.kochi-u.ac.jp http://weather.is.kochi-u.ac.jp/

Barry A. Warsaw wrote:

I have one nit that I'm not sure how to address though. I use VM in a MULE-ified XEmacs to read my mail and it has pretty good header decoding routines. If the charset is provided, it'll show me the real characters in a presentation buffer. However, it needs help for utf-8.

I used MULE and xemacs (with Gnus) for years to read my email, but I was never able to find a solution for UTF-8 headers. If you do end up finding one, please let the list know ;) The real problem is that MULE has a very hard-coded base on ISO-2022, and just grafting on UTF-8 is very, very hard. There is a MULE-UCS package for Unicode support, but the last time I tried to use it it barfed hard. If you can get it installed, it seems all you need is: (if (locate-library "un-define") (require 'un-define)) in your ~/.xemacs/init.el. See: http://lists.suse.com/archive/m17n/2002-May/0016.html Ben

barry@python.org (Barry A. Warsaw) writes:

Well, I spent a little time playing with un-define, and googling around, but wasn't able to come up with the magic incantations. Maybe this XEmacs FAQ entry sheds light that we have a while to wait yet...

In GNU Emacs, the situation is slightly different: It recognizes a few Unicode subsets, like mule-unicode-0100-24ff, mule-unicode-2500-33ff, and mule-unicode-e000-ffff. Notice that this excludes the CJK ideographs. The real problem is the ISO-2022 inheritance of Mule: While a buffer can represent multiple charsets, it can rarely equate characters across charset borders. So if two originate from two different charsets, they are considered different (unless there is a conversion procedure for this specific pair of charsets). In CVS Emacs, the "Latin unification" is considered a big step forward, ie. the fact that Emacs can now treat the overlapping character from iso-8859-{1,2,15,16} as equivalent. Without unification, Emacs has difficulties displaying the characters: Even if there is a font that has the necessary character, Emacs cannot find out that this font could be used if the font encoding is different from the buffer encoding. These problems would go away had Emacs used Unicode as an internal encoding, but there is some hostility from the Mule authors towards Unicode, in particular because of the need for Unihan disambiguation. Regards, Martin

barry@python.org (Barry A. Warsaw) writes:

BG> Again, I assume you mean ISO-8859-1 instead of ascii here.

Same thing here. We do name.encode('us-ascii') and catch any UnicodeError that might occur.

That should be name.decode, or unicode(name, "us-ascii"). name.encode unfortunately works as well, but it first decodes using the system default encoding, then encodes using us-ascii - very close to what you want, but not the precise same thing. Regards, Martin

barry@python.org (Barry A. Warsaw) writes:

Martin, sometimes this Unicode stuff makes my head hurt. ;)

In an application that deals with multiple charsets on a regular basis (such as mailman), I recommend not to mix byte strings and Unicode strings. This can be achieved by - converting all byte strings that represent text data to Unicode at the earliest possible point in processing, - converting all Unicode strings back to byte strings just before output. If most data is likely ASCII, it is tempting to use byte strings for pure-ASCII, and Unicode for everything else. Try to resist this temptation. If you follow this strategy, you find that processing becomes much simpler.

So it seems like name.encode('us-ascii') is my only choice. What am I missing?

If you are following the above strategy, you will know whether name is Unicode or byte string. If it is Unicode, .encode is fine. If it is a byte string, unicode(name,'ascii') will work. I admit that the strategy has two problems: 1. In some cases, it might be impossible to generate a Unicode string for text data. In MIME, the encoding may not be specified, or it may be unknown to mailman, or the data may fail to convert. In these cases, it may be acceptable to "force" the data to Unicode: If there is no encoding, guess latin-1. If the string fails to convert, convert it with "replace". If the encoding is unknown, replace all non-printable characters with question marks. Whether this is acceptable depends on how frequent the problem occurs and whose fault that is (e.g. an unknown encoding should be added to Mailman). 2. When converting an application that used to be byte-oriented to Unicode, adding conversions at all required places might be too much effort, or breakage because of incorrect data might be unacceptable. In these cases, I recommend to add type tests at strategic places, and taper over any incorrect data. E.g. in this case, you could write a function def unicode_is_pure_ascii(text): if type(text) is types.UnicodeType: try: text.encode("ascii") return 1 except UnicodeError: return 0 if DEBUG: raise DebugError, "string not unicode:"+repr(text) try: unicode(text,"ascii") return 1 except UnicodeError: return 0 If you expect name to be a byte string, the function would be bytes_are_ascii, of course. Regards, Martin

Barry A. Warsaw dijo:

BG> Everything else looks good. The kludge to assume iso-8859-1 BG> on us-ascii pages is unfortunately a generally good one, as BG> that will make the most people happy. I hate to do it, BG> though!

Me too! It means that names in other charsets will be screwed on English lists, but again, I think this is best we can do for a practical 80/20 solution.

from my ignorance: what about using iso-8859-15 instead of -1? it seems to add some forgotten french, finnish, estonian and czech letters. cheers, quique -- Torres más altas han caído.

[Martin v. Löwis]

I'm personally quite unhappy with iso-8859-15: it was invented at a time when Unicode was already there, and the world didn't really need any more character sets.

And there is also these MES charsets, too. I did not monitor them closely, but they are surely going to reach us one of these days... While many people see new 8-bit charsets as unwelcome, they fill a need. Unicode bears with it a few myths (simplicity, orthogonality, universality) which attract people at first, and which are also fed by incomplete or naive implementations, and fanatic proponents. But deeper one dives, deeper one realises that Unicode is quite complex after all, and the origin of a new lucrative industry where only Unicode specialists have a privileged niche. In real practice, many see that Unicode is not so democratic. The handling of strings is fundamental in computer science, nations invent new _simple_ character sets as a way of not loosing control of their own national computer industries over foreign specialists for artificial complexity. My usual explanations are fantasies. Supposing UTF-8 is not coincident with ASCII for U0000-U007F, imagine how Americans would react is asked to give up using ASCII. Another simple fantasy: supposing Microsoft did not push for Unicode, try figuring a place on this planet where Unicode would be prominent or prevailing today. It does not take a lot of Zen meditation to see that Unicode might be perceived in some foreign countries as an American threat to the local autonomy, as far as computers as concerned. Do you feel like fantasising further? Imagine that Americans (this holds for French, German, Vietnamese, and all those who got a special treat in Unicode with pre-combined characters) were asked that _some_ letters in their alphabet be only representable by combinable sequences, so Americans are now stuck all over with a variable number of bits per English character. Do you think Unicode would stand _any_ chance of ever being accepted in America? (or France, Germany, etc.) Such thoughts might help us understand why 8-bits characters are still going to multiply on this planet. -- François Pinard http://www.iro.umontreal.ca/~pinard