Martin v. Loewis wrote:

...

it is real. I won't repeat the arguments one more time; please read the W3C character model note and the python-dev archives, and read up on the unicode support in Tcl and Perl.

I did read all that, so there really is no point in repeating the arguments - yet I'm still not convinced. One of the causes may be that all your commentary either

- discusses an alternative solution to the existing one, merely pointing out the difference, without any strong selling point - explains small examples that work counter-intuitively

umm. I could have sworn that getting rid of counter-intuitive behaviour was rather important in python. maybe we're using the language in radically different ways?

I'd like to know whether you have an example of a real-world big-application problem that could not be conveniently implemented using the new Unicode API. For all the examples I can think where Unicode would matter (XML processing, CORBA wstring mapping, internationalized messages and GUIs), it would work just fine.

of course I can kludge my way around the flaws in MAL's design, but why should I have to do that? it's broken. fixing it is easy.

Perhaps my problem is that I'm not a perfectionist :-)

perfectionist or not, I only want Python's Unicode support to be as intuitive as anything else in Python. as it stands right now, Perl and Tcl's Unicode support is intuitive. Python's not. (it also backs us into a corner -- once you mess this one up, you cannot fix it in Py3K without breaking lots of code. that's really bad). in contrast, Guido's compromise proposal allows us to do this the right way in 1.7/Py3K (i.e. teach python about source code encodings, system api encodings, and stream i/o encodings). btw, I thought we'd all agreed on GvR's solution for 1.6? what did I miss?

So while it may not be perfect, I think it is good enough.

so tell me, if "good enough" is what we're aiming at, why isn't my counter-proposal good enough? if not else, it's much easier to document... </F>

Fredrik Lundh writes:

perfectionist or not, I only want Python's Unicode support to be as intuitive as anything else in Python. as it stands right now, Perl and Tcl's Unicode support is intuitive. Python's not.

I don't know about Tcl, but Perl 5.6's Unicode support is still considered experimental. Consider the following excerpts, for example. (And Fredrik's right; we shouldn't release a 1.6 with broken support, or we'll pay for it for *years*... But if GvR's ASCII proposal is considered OK, then great!) ======================== http://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/2000-04/msg00084.html:

Ah, yes. Unicode. But after two years of work, the one thing that users will want to do - open and read Unicode data - is still not there. Who cares if stuff's now represented internally in Unicode if they can't read the files they need to.

This is a "big" (as in "huge") disappointment for me as well. I hope we'll do better next time. ======================== http://www.egroups.com/message/perl5-porters/67906: But given that interpretation, I'm amazed at how many operators seem to be broken with UTF8. It certainly supports Ilya's contention of "pre-alpha". Here's another example: DB<1> x (256.255.254 . 257.258.259) eq (256.255.254.257.258.259) 0 '' DB<2> Rummaging with Devel::Peek shows that in this case, it's the fault of the . operator. And eq is broken as well: DB<11> x "\x{100}" eq "\xc4\x80" 0 1 DB<12> Aaaaargh! ======================== http://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/2000-03/msg00971.html: A couple problems here...passage through a hash key removes the UTF8 flag (as might be expected). Even if keys were to attempt to restore the UTF8 flag (ala Convert::UTF::decode_utf8) or hash keys were real SVs, what then do you do with $h{"\304\254"} and the like? Suggestions: 1. Leave things as they are, but document UTF8 hash keys as experimental and subject to change. or 2. When under use bytes, leave things as they are. Otherwise, have keys turn on the utf8 flag if appropriate. Also give a warning when using a hash key like "\304\254" since keys will in effect return a different string that just happens to have the same interal encoding. ========================

Martin v. Loewis wrote:

...

perfectionist or not, I only want Python's Unicode support to be as intuitive as anything else in Python. as it stands right now, Perl and Tcl's Unicode support is intuitive. Python's not.

I haven't much experience with Perl, but I don't think Tcl is intuitive in this area. I really think that they got it all wrong.

"all wrong"? Tcl works hard to maintain the characters are characters model (implementation level 2), just like Perl. the length of a string is always the number of characters, slicing works as it should, the internal representation is as efficient as you can make it. but yes, they have a somewhat dubious autoconversion mechanism in there. if something isn't valid UTF-8, it's assumed to be Latin-1. scary, huh? not really, if you step back and look at how UTF-8 was designed. quoting from RFC 2279: "UTF-8 strings can be fairly reliably recognized as such by a simple algorithm, i.e. the probability that a string of characters in any other encoding appears as valid UTF-8 is low, diminishing with increasing string length." besides, their design is based on the plan 9 rune stuff. that code was written by the inventors of UTF-8, who has this to say: "There is little a rune-oriented program can do when given bad data except exit, which is unreasonable, or carry on. Originally the conversion routines, described below, returned errors when given invalid UTF, but we found ourselves repeatedly checking for errors and ignoring them. We therefore decided to convert a bad sequence to a valid rune and continue processing. "This technique does have the unfortunate property that con- verting invalid UTF byte strings in and out of runes does not preserve the input, but this circumstance only occurs when non-textual input is given to a textual program." so let's see: they aimed for a high level of unicode support (layer 2, stream encodings, and system api encodings, etc), they've based their design on work by the inventors of UTF-8, they have several years of experience using their implementation in real life, and you seriously claim that they got it "all wrong"? that's weird.

AFAICT, all it does is to change the default encoding from UTF-8 to Latin-1.

now you're using "all" in that strange way again... check the archives for the full story (hint: a conceptual design model isn't the same thing as a C implementation)

I can't follow why this should be *better*, but it would be certainly as good... In comparison, restricting the "character" interpretation of the string type (in terms of your proposal) to 7-bit characters has the advantage that it is less error-prone, as Guido points out.

the main reason for that is that Python 1.6 doesn't have any way to specify source encodings. add that, so you no longer have to guess what a string *literal* really is, and that problem goes away. but that's something for 1.7. </F>

Fred L. Drake wrote:

On Wed, 17 May 2000, Fredrik Lundh wrote:

...

the main reason for that is that Python 1.6 doesn't have any way to specify source encodings. add that, so you no longer have to guess what a string *literal* really is, and that problem goes away. but

You seem to be familiar with the Tcl work, so I'll ask you this question: Does Tcl have a way to specify source encoding?

Tcl has a system encoding (which is used when passing strings through system APIs), and file/channel-specific encodings. (for info on how they initialize the system encoding, see earlier posts). unfortunately, they're using the system encoding also for source code. for portable code, they recommend sticking to ASCII or using "bootstrap scripts", e.g: set fd [open "app.tcl" r] fconfigure $fd -encoding euc-jp set jpscript [read $fd] close $fd eval $jpscript we can surely do better in 1.7... </F>

Martin v. Loewis wrote:

To my knowledge, no. Tcl (at least 8.3) supports the \u notation for Unicode escapes, and treats all other source code as Latin-1. encoding(n) says

# However, because the source command always reads files using the # ISO8859-1 encoding, Tcl will treat each byte in the file as a # separate character that maps to the 00 page in Unicode.

as far as I can tell from digging through the sources, the "source" command uses the system encoding. and from the look of it, it's not always iso-latin-1... </F>

Fredrik Lundh wrote:

of course I can kludge my way around the flaws in MAL's design, but why should I have to do that? it's broken. fixing it is easy.

Look Fredrik, it's not *my* design. All this was discussed in public and in several rounds late last year. If someone made a mistake and "broke" anything, then we all did... I still don't think so, but that's my personal opinion. -- Now to get back to some non-flammable content: Has anyone played around with the latest sys.set_string_encoding() patches ? I would really like to know what you think. The idea behind it is that you can define what the Unicode implementaion is to expect as encoding when it sees an 8-bit string. The encoding is used for coercion, str(unicode) and printing. It is currently *not* used for the "s" parser marker and hash values (mainly due to internal issues). See my patch comments for details. -- Marc-Andre Lemburg ______________________________________________________________________ Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/

"Martin v. Loewis" wrote:

...

I'd like to know whether you have an example of a real-world big-application problem that could not be conveniently implemented using the new Unicode API. For all the examples I can think where Unicode would matter (XML processing, CORBA wstring mapping, internationalized messages and GUIs), it would work just fine.

Of course an implicit behavior can never get in the way of big-application building. The question is about principle of least surprise, and simplicity of explanation and understanding. I'm-told-that-even-Perl-and-C++-can-be-used-for-big-apps -ly yrs -- Paul Prescod - ISOGEN Consulting Engineer speaking for himself "Hardly anything more unwelcome can befall a scientific writer than having the foundations of his edifice shaken after the work is finished. I have been placed in this position by a letter from Mr. Bertrand Russell..." - Frege, Appendix of Basic Laws of Arithmetic (of Russell's Paradox)