Re: [XML-SIG] Re: [I18n-sig] Re: [Python-Dev] Unicode debate
Guido van Rossum wrote, about how to represent strings:
Paul, we're both just saying the same thing over and over without convincing each other. I'll wait till someone who wasn't in this debate before chimes in.
I'm with Paul and Federick on this one - at least about characters being the atoms of a string. We **have** to be able to refer to **characters** in a string, and without guessing. Otherwise, how could you ever construct a test, like theString[3]==[a particular japanese ideograph]? If we do it by having a "string" datatype, which is really a byte list, and a "unicodeString" datatype which is a list of abstract characters, I'd say everyone could get used to working with them. We'd have to supply conversion functions, of course. This route might be the easiest to understand for users. We'd have to be very clear about what file.read() would return, for example, and all those similar read and write functions. And we'd have to work out how real 8-bit calls (like writing to a socket?) would play with the new types. For extra clarity, we could leave string the way it is, introduce stringU (unicode string) **and** string8 (Latin-1 or byte list, whichever seems to be the best equivalent to the current string). Then we would deprecate string in favor of string8. Then if tcl and perl go to unicode strings we pass them a stringU, and if they go some other way, we pass them something else. COme to think of it, we need some some data type that will continue to work with c and c++. Would that be string8 or would we keep string for that purpose? Clarity and ease of use for the user should be primary, fast implementations next. If we didn't care about ease of use and clarity, we could all use Scheme or c, don't use sight of it. I'd suggest we could create some use cases or scenarios for this area - needs input from those who know encodings and low level Python stuff better than I. Then we could examine more systematically how well various approaches would work out. Regards, Tom Passin
Guido van Rossum wrote, about how to represent strings:
Paul, we're both just saying the same thing over and over without convincing each other. I'll wait till someone who wasn't in this debate before chimes in.
Ive chimed in a little, but Ill chime in again :-)
I'm with Paul and Federick on this one - at least about characters being the atoms of a string. We **have** to be able to refer to **characters** in a string, and without guessing. Otherwise, how could you
I see the point, and agree 100% with the intent. However, reality does bite. As far as I can see, the following are immuatable: * There will be 2 types - a string type and a Unicode type. * History dicates that the string type may hold binary data. Thus, it is clear that Python simply can not treat characters as the smallest atoms of strings. If I understand things correctly, this is key to Guido's point, and a bit of a communication block. The issue, to my mind, is how we handle these facts to produce "the principal of least surprise". We simply need to accept that Python 1.x will never be able to treat string objects as sequences of "characters" - only bytes. However, with my limited understanding of the full issues, it does appear that the proposal championed by Fredrik, Just and Paul is the best solution - not because it magically causes Python to treat strings as characters in all cases, but because it offers the prinipcal of least surprise. As I said, I dont really have a deep enough understanding of the issues, so this is probably (hopefully!?) my last word on the matter - but that doesnt mean I dont share the concerns raised here... Mark.
Tom Passin:
I'm with Paul and Federick on this one - at least about characters being the atoms of a string. We **have** to be able to refer to **characters** in a string, and without guessing. Otherwise, how could you ever construct a test, like theString[3]==[a particular japanese ideograph]? If we do it by having a "string" datatype, which is really a byte list, and a "unicodeString" datatype which is a list of abstract characters, I'd say everyone could get used to working with them. We'd have to supply conversion functions, of course.
You seem unfamiliar with the details of the implementation we're proposing? We already have two datatypes, 8-bit string (call it byte array) and Unicode string. There are conversions between them: explicit conversions such as u.encode("utf-8") or unicode(s, "latin-1") and implicit conversions used in situations like u+s or u==s. The whole discussion is *only* about what the default conversion in the latter cases should be -- the rest of the implementation is rock solid and works well. Users can accomplish what you are proposing by simply ensuring that theString is a Unicode string.
This route might be the easiest to understand for users. We'd have to be very clear about what file.read() would return, for example, and all those similar read and write functions. And we'd have to work out how real 8-bit calls (like writing to a socket?) would play with the new types.
These are all well defined -- they all deal in 8-bit strings internally, and all use the default conversions when given Unicode strings. Programs that only deal in 8-bit strings don't need to change. Programs that want to deal with Unicode and sockets, for example, must know what encoding to use on the socket, and if it's not the default encoding, must use explicit conversions.
For extra clarity, we could leave string the way it is, introduce stringU (unicode string) **and** string8 (Latin-1 or byte list, whichever seems to be the best equivalent to the current string). Then we would deprecate string in favor of string8. Then if tcl and perl go to unicode strings we pass them a stringU, and if they go some other way, we pass them something else. COme to think of it, we need some some data type that will continue to work with c and c++. Would that be string8 or would we keep string for that purpose?
What would be the difference between string and string8?
Clarity and ease of use for the user should be primary, fast implementations next. If we didn't care about ease of use and clarity, we could all use Scheme or c, don't use sight of it.
I'd suggest we could create some use cases or scenarios for this area - needs input from those who know encodings and low level Python stuff better than I. Then we could examine more systematically how well various approaches would work out.
Very good. Here's one usage scenario. A Japanese user is reading lines from a file encoded in ISO-2022-JP. The readline() method returns 8-bit strings in that encoding (the file object doesn't do any decoding). She realizes that she wants to do some character-level processing on the file so she decides to convert the strings to Unicode. I believe that whether the default encoding is UTF-8 or Latin-1 doesn't matter for here -- both are wrong, she needs to write explicit unicode(line, "iso-2022-jp") code anyway. I would argue that UTF-8 is "better", because interpreting ISO-2022-JP data as UTF-8 will most likely give an exception (when a \300 range byte isn't followed by a \200 range byte) -- while interpreting it as Latin-1 will silently do the wrong thing. (An explicit error is always better than silent failure.) I'd love to discuss other scenarios. --Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum said <snip/>
What would be the difference between string and string8?
Probably none, except to alert people that string8 might have different behavior than the present-day string, perhaps when interacting with unicode - probably its behavior would be specified more tightly (i.e., is it strictly a list of bytes or does it have some assumption about encoding?) or changed in some way from what we have now. Or if it turned out that a lot of programmers in other languages (perl, tcl, perhaps?) expected "string" to behave in particular ways, the use of a term like "string8" might reduce confusion. Possibly none of these apply - no need for "string8" then.
Clarity and ease of use for the user should be primary, fast
implementations
next. If we didn't care about ease of use and clarity, we could all use Scheme or c, don't use sight of it.
I'd suggest we could create some use cases or scenarios for this area - needs input from those who know encodings and low level Python stuff better than I. Then we could examine more systematically how well various approaches would work out.
Very good.
<snip/> Tom Passin
At 11:31 PM -0400 01-05-2000, Guido van Rossum wrote:
Here's one usage scenario.
A Japanese user is reading lines from a file encoded in ISO-2022-JP. The readline() method returns 8-bit strings in that encoding (the file object doesn't do any decoding). She realizes that she wants to do some character-level processing on the file so she decides to convert the strings to Unicode.
I believe that whether the default encoding is UTF-8 or Latin-1 doesn't matter for here -- both are wrong, she needs to write explicit unicode(line, "iso-2022-jp") code anyway. I would argue that UTF-8 is "better", because interpreting ISO-2022-JP data as UTF-8 will most likely give an exception (when a \300 range byte isn't followed by a \200 range byte) -- while interpreting it as Latin-1 will silently do the wrong thing. (An explicit error is always better than silent failure.)
But then it's even better to *always* raise an exception, since it's entirely possible a string contains valid utf-8 while not *being* utf-8. I really think the exception argument is moot, since there can *always* be situations that will pass silently. Encoding issues are silent by nature -- eg. there's no way any system can tell that interpreting MacRoman data as Latin-1 is wrong, maybe even fatal -- the user will just have to deal with it. You can argue what you want, but *any* multi-byte encoding stored in an 8-bit string is a buffer, not a string, for all the reasons Fredrik and Paul have thrown at you, and right they are. Choosing such an encoding as a default conversion to Unicode makes no sense at all. Recap of the main arguments: pro UTF-8: always reversible when going from Unicode to 8-bit con UTF-8: not a string: confusing semantics pro Latin-1: simpler semantics con Latin-1: non-reversible, western-centric Given the fact that very often *both* will be wrong, I'd go for the simpler semantics. Just
I'm dropping in a bit late in this thread but can the current problem be summarised in an example as "how is 'literal' interpreted here"? s = aUnicodeStringFromSomewhere DoSomething(s + "<literal>") The two options being that literal is either assumed to be encoded in Latin-1 or UTF-8. I can see some arguments for both sides. Latin-1: more current code was written in a European locale with an implicit assumption that all string handling was Latin-1. Current editors are more likely to be displaying literal as it is meant to be interpreted. UTF-8: all languages can be written in UTF-8 and more recent editors can display this correctly. Thus people using non-Roman alphabets can write code which is interpreted as is seen with no need to remember to call conversion functions. Neil
Neil Hodgson <nhodgson@bigpond.net.au> wrote:
I'm dropping in a bit late in this thread but can the current problem be summarised in an example as "how is 'literal' interpreted here"?
s = aUnicodeStringFromSomewhere DoSomething(s + "<literal>")
nope. the whole discussion centers around what happens if you type: # example 1 u = aUnicodeStringFromSomewhere s = an8bitStringFromSomewhere DoSomething(s + u) and # example 2 u = aUnicodeStringFromSomewhere s = an8bitStringFromSomewhere if len(u) + len(s) == len(u + s): print "true" else: print "not true" in Guido's design, the first example may or may not result in an "UTF-8 decoding error: UTF-8 decoding error: unexpected code byte" exception. the second example may result in a similar error, print "true", or print "not true", depending on the contents of the 8-bit string. (under the counter proposal, the first example will never raise an exception, and the second will always print "true") ... the string literal issue is a slightly different problem.
The two options being that literal is either assumed to be encoded in Latin-1 or UTF-8. I can see some arguments for both sides.
better make that "two options", not "the two options" ;-) a more flexible scheme would be to borrow the design from XML (see http://www.w3.org/TR/1998/REC-xml-19980210). for those who haven't looked closer at XML, it basically treats the source file as an encoded unicode character stream, and does all pro- cessing on the decoded side. replace "entity" with "script file" in the following excerpts, and you get close: section 2.2: A parsed entity contains text, a sequence of characters, which may represent markup or character data. A character is an atomic unit of text as specified by ISO/IEC 10646. section 4.3.3: Each external parsed entity in an XML document may use a different encoding for its characters. All XML processors must be able to read entities in either UTF-8 or UTF-16. Entities encoded in UTF-16 must begin with the Byte Order Mark /.../ XML processors must be able to use this character to differentiate between UTF-8 and UTF-16 encoded documents. Parsed entities which are stored in an encoding other than UTF-8 or UTF-16 must begin with a text declaration containing an encoding declaration. (also see appendix F: Autodetection of Character Encodings) I propose that we adopt a similar scheme for Python -- but not in 1.6. the current "dunno, so we just copy the characters" is good enough for now... </F>
u = aUnicodeStringFromSomewhere s = an8bitStringFromSomewhere
DoSomething(s + u)
in Guido's design, the first example may or may not result in an "UTF-8 decoding error: UTF-8 decoding error: unexpected code byte" exception.
I would say it is less surprising for most people for this to follow the silent-widening of each byte - the Fredrik-Paul position. With the current scarcity of UTF-8 code, very few people will expect an automatic UTF-8 to UTF-16 conversion. While complete prohibition of automatic conversion has some appeal, it will just be more noise to many.
u = aUnicodeStringFromSomewhere s = an8bitStringFromSomewhere
if len(u) + len(s) == len(u + s): print "true" else: print "not true"
the second example may result in a similar error, print "true", or print "not true", depending on the contents of the 8-bit string.
I don't see this as important as its trying to take the Unicode strings are equivalent to 8 bit strings too far. How much further before you have to break? I always thought of len measuring the number of bytes rather than characters when applied to strings. The same as strlen in C when you have a DBCS string. I should correct some of the stuff Mark wrote about me. At Fujitsu we did a lot more DBCS work than Unicode because that's what Japanese code uses. Even with Java most storage is still DBCS. I was more involved with Unicode architecture at Reuters 6 or so years ago. Neil
Neil, I sincerely appreciate your informed input. I want to emphasize one ideological difference though. :) Neil Hodgson wrote:
...
The two options being that literal is either assumed to be encoded in Latin-1 or UTF-8.
I reject that characterization. I claim that both strings contain Unicode characters but one can contain Unicode charactes with higher digits. UTF-8 versus latin-1 does not enter into it. Python strings should not be documented in terms of encodings any more than Python ints are documented in terms of their two's complement representation. Then we could describe the default conversion from integers to floats in terms of their bit-representation. Ugh! I accept that the effect is similar to calling Latin-1 the "default" that's a side effect of the simple logical model that we are proposing. -- Paul Prescod - ISOGEN Consulting Engineer speaking for himself It's difficult to extract sense from strings, but they're the only communication coin we can count on. - http://www.cs.yale.edu/~perlis-alan/quotes.html
Just a small note on the subject of a character being atomic which seems to have been forgotten by the discussing parties: Unicode itself can be understood as multi-word character encoding, just like UTF-8. The reason is that Unicode entities can be combined to produce single display characters (e.g. u"e"+u"\u0301" will print "é" in a Unicode aware renderer). Slicing such a combined Unicode string will have the same effect as slicing UTF-8 data. It seems that most Latin-1 proponents seem to have single display characters in mind. While the same is true for many Unicode entities, there are quite a few cases of combining characters in Unicode 3.0 and the Unicode nomarization algorithm uses these as basis for its work. So in the end the "UTF-8 doesn't slice" argument holds for Unicode itself too, just as it also does for many Asian multi-byte variable length character encodings, image formats, audio formats, database formats, etc. You can't really expect slicing to always "just work" without some knowledge about the data you are slicing. -- Marc-Andre Lemburg ______________________________________________________________________ Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/
M.-A. Lemburg <mal@lemburg.com> wrote:
Just a small note on the subject of a character being atomic which seems to have been forgotten by the discussing parties:
Unicode itself can be understood as multi-word character encoding, just like UTF-8. The reason is that Unicode entities can be combined to produce single display characters (e.g. u"e"+u"\u0301" will print "é" in a Unicode aware renderer). Slicing such a combined Unicode string will have the same effect as slicing UTF-8 data.
really? does it result in a decoder error? or does it just result in a rendering error, just as if you slice off any trailing character without looking...
It seems that most Latin-1 proponents seem to have single display characters in mind. While the same is true for many Unicode entities, there are quite a few cases of combining characters in Unicode 3.0 and the Unicode nomarization algorithm uses these as basis for its work.
do we supported automatic normalization in 1.6? </F>
Fredrik Lundh wrote:
M.-A. Lemburg <mal@lemburg.com> wrote:
Just a small note on the subject of a character being atomic which seems to have been forgotten by the discussing parties:
Unicode itself can be understood as multi-word character encoding, just like UTF-8. The reason is that Unicode entities can be combined to produce single display characters (e.g. u"e"+u"\u0301" will print "é" in a Unicode aware renderer). Slicing such a combined Unicode string will have the same effect as slicing UTF-8 data.
really? does it result in a decoder error? or does it just result in a rendering error, just as if you slice off any trailing character without looking...
In the example, if you cut off the u"\u0301", the "e" would appear without the acute accent, cutting off the u"e" would probably result in a rendering error or worse put the accent over the next character to the left. UTF-8 is better in this respect: it warns you about the error by raising an exception when being converted to Unicode.
It seems that most Latin-1 proponents seem to have single display characters in mind. While the same is true for many Unicode entities, there are quite a few cases of combining characters in Unicode 3.0 and the Unicode normalization algorithm uses these as basis for its work.
do we supported automatic normalization in 1.6?
No, but it is likely to appear in 1.7... not sure about the "automatic" though. FYI: Normalization is needed to make comparing Unicode strings robust, e.g. u"é" should compare equal to u"e\u0301". -- Marc-Andre Lemburg ______________________________________________________________________ Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/
[MAL]
Unicode itself can be understood as multi-word character encoding, just like UTF-8. The reason is that Unicode entities can be combined to produce single display characters (e.g. u"e"+u"\u0301" will print "é" in a Unicode aware renderer). Slicing such a combined Unicode string will have the same effect as slicing UTF-8 data. [/F] really? does it result in a decoder error? or does it just result in a rendering error, just as if you slice off any trailing character without looking... [MAL] In the example, if you cut off the u"\u0301", the "e" would appear without the acute accent, cutting off the u"e" would probably result in a rendering error or worse put the accent over the next character to the left.
UTF-8 is better in this respect: it warns you about the error by raising an exception when being converted to Unicode.
I think /F's point was that the Unicode standard prescribes different behavior here: for UTF-8, a missing or lone continuation byte is an error; for Unicode, accents are separate characters that may be inserted and deleted in a string but whose display is undefined under certain conditions. (I just noticed that this doesn't work in Tkinter but it does work in wish. Strange.)
FYI: Normalization is needed to make comparing Unicode strings robust, e.g. u"é" should compare equal to u"e\u0301".
Aha, then we'll see u == v even though type(u) is type(v) and len(u) != len(v). /F's world will collapse. :-) --Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum <guido@python.org> wrote:
FYI: Normalization is needed to make comparing Unicode strings robust, e.g. u"Ú" should compare equal to u"e\u0301".
Aha, then we'll see u == v even though type(u) is type(v) and len(u) != len(v). /F's world will collapse. :-)
you're gonna do automatic normalization? that's interesting. will this make Python the first language to defines strings as a "sequence of graphemes"? or was this just the cheap shot it appeared to be? </F>
At 8:30 AM -0400 02-05-2000, Guido van Rossum wrote:
I think /F's point was that the Unicode standard prescribes different behavior here: for UTF-8, a missing or lone continuation byte is an error; for Unicode, accents are separate characters that may be inserted and deleted in a string but whose display is undefined under certain conditions.
(I just noticed that this doesn't work in Tkinter but it does work in wish. Strange.)
FYI: Normalization is needed to make comparing Unicode strings robust, e.g. u"È" should compare equal to u"e\u0301".
Aha, then we'll see u == v even though type(u) is type(v) and len(u) != len(v). /F's world will collapse. :-)
Does the Unicode spec *really* specifies u should compare equal to v? This behavior would be the responsibility of a layout engine, a role which is way beyond the scope of Unicode support in Python, as it is language- and script-dependent. Just
Just van Rossum wrote:
At 8:30 AM -0400 02-05-2000, Guido van Rossum wrote:
I think /F's point was that the Unicode standard prescribes different behavior here: for UTF-8, a missing or lone continuation byte is an error; for Unicode, accents are separate characters that may be inserted and deleted in a string but whose display is undefined under certain conditions.
(I just noticed that this doesn't work in Tkinter but it does work in wish. Strange.)
FYI: Normalization is needed to make comparing Unicode strings robust, e.g. u"È" should compare equal to u"e\u0301".
^ | Here's a good example of what encoding errors can do: the above character was an "e" with acute accent (u"é"). Looks like some mailer converted this to some other code page and yet another back to Latin-1 again and this even though the message header for Content-Type clearly states that the document uses ISO-8859-1.
Aha, then we'll see u == v even though type(u) is type(v) and len(u) != len(v). /F's world will collapse. :-)
Does the Unicode spec *really* specifies u should compare equal to v?
The behaviour is needed in order to implement sorting Unicode. See the www.unicode.org site for more information and the tech reports describing this. Note that I haven't mentioned anything about "automatic" normalization. This should be a method on Unicode strings and could then be used in sorting compare callbacks. -- Marc-Andre Lemburg ______________________________________________________________________ Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/
Guido van Rossum wrote:
Aha, then we'll see u == v even though type(u) is type(v) and len(u) != len(v). /F's world will collapse. :-)
There are many levels of equality that are interesting. I don't think we would move to grapheme equivalence until "the rest of the world" (XML, Java, W3C, SQL) did. If we were going to move to grapheme equivalence (some day), the right way would be to normalize characters in the construction of the Unicode string. This is known as "Early normalization": http://www.w3.org/TR/charmod/#NormalizationApplication -- Paul Prescod - ISOGEN Consulting Engineer speaking for himself It's difficult to extract sense from strings, but they're the only communication coin we can count on. - http://www.cs.yale.edu/~perlis-alan/quotes.html
At 10:36 AM +0200 02-05-2000, M.-A. Lemburg wrote:
Just a small note on the subject of a character being atomic which seems to have been forgotten by the discussing parties:
Unicode itself can be understood as multi-word character encoding, just like UTF-8. The reason is that Unicode entities can be combined to produce single display characters (e.g. u"e"+u"\u0301" will print "é" in a Unicode aware renderer).
Erm, are you sure Unicode prescribes this behavior, for this example? I know similar behaviors are specified for certain languages/scripts, but I didn't know it did that for latin.
Slicing such a combined Unicode string will have the same effect as slicing UTF-8 data.
Not true. As Fredrik noted: no exception will be raised. [ Speaking of exceptions, after I sent off my previous post I realized Guido's non-utf8-strings-interpreted-as-utf8-will-often-raise-an-exception argument can easily be turned around, backfiring at utf-8: Defaulting to utf-8 when going from Unicode to 8-bit and back only gives the *illusion* things "just work", since it will *silently* "work", even if utf-8 is *not* the desired 8-bit encoding -- as shown by Fredrik's excellent "fun with Unicode, part 1" example. Defaulting to Latin-1 will warn the user *much* earlier, since it'll barf when converting a Unicode string that contains any character code > 255. So there. ]
It seems that most Latin-1 proponents seem to have single display characters in mind. While the same is true for many Unicode entities, there are quite a few cases of combining characters in Unicode 3.0 and the Unicode nomarization algorithm uses these as basis for its work.
Still, two combining characters are still two input characters for the renderer! They may result in one *glyph*, but trust me, that's an entirly different can of worms. However, if you'd be talking about Unicode surrogates, you'd definitely have a point. How do Java/Perl/Tcl deal with surrogates? Just
Just van Rossum wrote:
At 10:36 AM +0200 02-05-2000, M.-A. Lemburg wrote:
Just a small note on the subject of a character being atomic which seems to have been forgotten by the discussing parties:
Unicode itself can be understood as multi-word character encoding, just like UTF-8. The reason is that Unicode entities can be combined to produce single display characters (e.g. u"e"+u"\u0301" will print "é" in a Unicode aware renderer).
Erm, are you sure Unicode prescribes this behavior, for this example? I know similar behaviors are specified for certain languages/scripts, but I didn't know it did that for latin.
The details are on the www.unicode.org web-site burried in some of the tech reports on normalization and collation.
Slicing such a combined Unicode string will have the same effect as slicing UTF-8 data.
Not true. As Fredrik noted: no exception will be raised.
Huh ? You will always get an exception when you convert a broken UTF-8 sequence to Unicode. This is per design of UTF-8 itself which uses the top bit to identify multi-byte character encodings. Or can you give an example (perhaps you've found a bug that needs fixing) ?
[ Speaking of exceptions,
after I sent off my previous post I realized Guido's non-utf8-strings-interpreted-as-utf8-will-often-raise-an-exception argument can easily be turned around, backfiring at utf-8:
Defaulting to utf-8 when going from Unicode to 8-bit and back only gives the *illusion* things "just work", since it will *silently* "work", even if utf-8 is *not* the desired 8-bit encoding -- as shown by Fredrik's excellent "fun with Unicode, part 1" example. Defaulting to Latin-1 will warn the user *much* earlier, since it'll barf when converting a Unicode string that contains any character code > 255. So there. ]
It seems that most Latin-1 proponents seem to have single display characters in mind. While the same is true for many Unicode entities, there are quite a few cases of combining characters in Unicode 3.0 and the Unicode nomarization algorithm uses these as basis for its work.
Still, two combining characters are still two input characters for the renderer! They may result in one *glyph*, but trust me, that's an entirly different can of worms.
No. Please see my other post on the subject...
However, if you'd be talking about Unicode surrogates, you'd definitely have a point. How do Java/Perl/Tcl deal with surrogates?
Good question... anybody know the answers ? -- Marc-Andre Lemburg ______________________________________________________________________ Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/
M.-A. Lemburg writes:
The details are on the www.unicode.org web-site burried in some of the tech reports on normalization and collation.
This is described in the Unicode standard itself, and in UTR #15 and UTR #10. Normalization is an issue with wider imlications than just handling glyph variants: indeed, it's irrelevant. The question is this: should U+00DC LATIN CAPITAL LETTER U WITH DIAERESIS compare equal to U+0055 LATIN CAPITAL LETTER U U+0308 COMBINING DIAERESIS or not? It depends on the application. Certainly in a database system I would want these to compare equal. Perhaps normalization form needs to be an option of the string comparator? -tree -- Tom Emerson Basis Technology Corp. Language Hacker http://www.basistech.com "Beware the lollipop of mediocrity: lick it once and you suck forever"
At 5:24 PM +0200 02-05-2000, M.-A. Lemburg wrote:
Still, two combining characters are still two input characters for the renderer! They may result in one *glyph*, but trust me, that's an entirly different can of worms.
No. Please see my other post on the subject...
It would help if you'd post some actual doco. Just
Combining characters are a whole 'nother level of complexity. Charater sets are hard. I don't accept that the argument that "Unicode itself has complexities so that gives us license to introduce even more complexities at the character representation level."
FYI: Normalization is needed to make comparing Unicode strings robust, e.g. u"é" should compare equal to u"e\u0301".
That's a whole 'nother debate at a whole 'nother level of abstraction. I think we need to get the bytes/characters level right and then we can worry about display-equivalent characters (or leave that to the Python programmer to figure out...). -- Paul Prescod - ISOGEN Consulting Engineer speaking for himself It's difficult to extract sense from strings, but they're the only communication coin we can count on. - http://www.cs.yale.edu/~perlis-alan/quotes.html
Paul Prescod wrote:
Combining characters are a whole 'nother level of complexity. Charater sets are hard. I don't accept that the argument that "Unicode itself has complexities so that gives us license to introduce even more complexities at the character representation level."
FYI: Normalization is needed to make comparing Unicode strings robust, e.g. u"é" should compare equal to u"e\u0301".
That's a whole 'nother debate at a whole 'nother level of abstraction. I think we need to get the bytes/characters level right and then we can worry about display-equivalent characters (or leave that to the Python programmer to figure out...).
I just wanted to point out that the argument "slicing doesn't work with UTF-8" is moot. I do see a point against UTF-8 auto-conversion given the example that Guido mailed me: """ s = 'ab\341\210\264def' # == str(u"ab\u1234def") s.find(u"def") This prints 3 -- the wrong result since "def" is found at s[5:8], not at s[3:6]. """ -- Marc-Andre Lemburg ______________________________________________________________________ Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/
[MAL vs. PP]
FYI: Normalization is needed to make comparing Unicode strings robust, e.g. u"é" should compare equal to u"e\u0301".
That's a whole 'nother debate at a whole 'nother level of abstraction. I think we need to get the bytes/characters level right and then we can worry about display-equivalent characters (or leave that to the Python programmer to figure out...).
I just wanted to point out that the argument "slicing doesn't work with UTF-8" is moot.
And failed... I asked two Unicode guru's I happen to know about the normalization issue (which is indeed not relevant to the current discussion, but it's fascinating nevertheless!). (Sorry about the possibly wrong email encoding... "è" is u"\350", "ö" is u"\366") John Jenkins replied: """ Well, I'm not sure you want to hear the answer -- but it really depends on what the language is attempting to do. By and large, Unicode takes the position that "e`" should always be treated the same as "è". This is a *semantic* equivalence -- that is, they *mean* the same thing -- and doesn't depend on the display engine to be true. Unicode also provides a default collation algorithm (http://www.unicode.org/unicode/reports/tr10/). At the same time, the standard acknowledges that in real life, string comparison and collation are complicated, language-specific problems requiring a lot of work and interaction with the user to do right.
From the perspective of a programming language, it would best be served IMHO by implementing the contents of TR10 for string comparison and collation. That would make "e`" and "è" come out as equivalent. """
Dave Opstad replied: """ Unicode talks about "canonical decomposition" in order to make it easier to answer questions like yours. Specifically, in the Unicode 3.0 standard, rule D24 in section 3.6 (page 44) states that: "Two character sequences are said to be canonical equivalents if their full canonical decompositions are identical. For example, the sequences <o, combining-diaeresis> and <ö> are canonical equivalents. Canonical equivalence is a Unicode propert. It should not be confused with language-specific collation or matching, which may add additional equivalencies." So they still have language-specific differences, even if Unicode sees them as canonically equivalent. You might want to check this out: http://www.unicode.org/unicode/reports/tr15/tr15-18.html It's the latest technical report on these issues, which may help clarify things further. """ It's very deep stuff, which seems more appropriate for an extension than for builtin comparisons to me. Just
[MAL]
I just wanted to point out that the argument "slicing doesn't work with UTF-8" is moot.
[Just]
And failed...
He succeeded for me. Blind slicing doesn't always "work right" no matter what encoding you use, because "work right" depends on semantics beyond the level of encoding. UTF-8 is no worse than anything else in this respect.
[MAL]
I just wanted to point out that the argument "slicing doesn't work with UTF-8" is moot.
[Just]
And failed...
[Tim]
He succeeded for me. Blind slicing doesn't always "work right" no matter what encoding you use, because "work right" depends on semantics beyond the level of encoding. UTF-8 is no worse than anything else in this respect.
But the discussion *was* at the level of encoding! Still it is worse, since an arbitrary utf-8 slice may result in two illegal strings -- slicing "e`" results in two perfectly legal strings, at the encoding level. Had he used surrogates as an example, he would've been right... (But even that is an encoding issue.) Just
Just van Rossum wrote:
[MAL vs. PP]
FYI: Normalization is needed to make comparing Unicode strings robust, e.g. u"é" should compare equal to u"e\u0301".
That's a whole 'nother debate at a whole 'nother level of abstraction. I think we need to get the bytes/characters level right and then we can worry about display-equivalent characters (or leave that to the Python programmer to figure out...).
I just wanted to point out that the argument "slicing doesn't work with UTF-8" is moot.
And failed...
Huh ? The pure fact that you can have two (or more) Unicode characters to represent a single character makes Unicode itself have the same problems as e.g. UTF-8.
[Refs about collation and decomposition]
It's very deep stuff, which seems more appropriate for an extension than for builtin comparisons to me.
That's what I think too; I never argued for making this builtin and automatic (don't know where people got this idea from). -- Marc-Andre Lemburg ______________________________________________________________________ Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/
At 10:15 AM +0200 03-05-2000, M.-A. Lemburg wrote:
Huh ? The pure fact that you can have two (or more) Unicode characters to represent a single character makes Unicode itself have the same problems as e.g. UTF-8.
It's the different level of abstraction that makes it different. Even if "e`" is _equivalent_ to the combined character, that doesn't mean that it _is_ the combined character, on the level of abstraction we are talking about: it's still 2 characters, and those can be sliced apart without a problem. Slicing utf-8 doesn't work because it yields invalid strings, slicing "e`" does work since both halves are valid strings. The fact that "e`" is semantically equivalent to the combined character doesn't change that. Just
I'll warn you that i'm not much experienced or well-informed, but i suppose i might as well toss in my naive opinion. At 11:31 PM -0400 01-05-2000, Guido van Rossum wrote:
I believe that whether the default encoding is UTF-8 or Latin-1 doesn't matter for here -- both are wrong, she needs to write explicit unicode(line, "iso-2022-jp") code anyway. I would argue that UTF-8 is "better", because [this] will most likely give an exception...
On Tue, 2 May 2000, Just van Rossum wrote:
But then it's even better to *always* raise an exception, since it's entirely possible a string contains valid utf-8 while not *being* utf-8.
I believe it is time for me to make a truly radical proposal: No automatic conversions between 8-bit "strings" and Unicode strings. If you want to turn UTF-8 into a Unicode string, say so. If you want to turn Latin-1 into a Unicode string, say so. If you want to turn ISO-2022-JP into a Unicode string, say so. Adding a Unicode string and an 8-bit "string" gives an exception. I know this sounds tedious, but at least it stands the least possible chance of confusing anyone -- and given all i've seen here and in other i18n and l10n discussions, there's plenty enough confusion to go around already. If it turns out automatic conversions *are* absolutely necessary, then i vote in favour of the simple, direct method promoted by Paul and Fredrik: just copy the numerical values of the bytes. The fact that this happens to correspond to Latin-1 is not really the point; the main reason is that it satisfies the Principle of Least Surprise. Okay. Feel free to yell at me now. -- ?!ng P. S. The scare-quotes when i talk about 8-bit "strings" expose my sense of them as byte-buffers -- since that *is* all you get when you read in some bytes from a file. If you manipulate an 8-bit "string" as a character string, you are implicitly making the assumption that the byte values correspond to the character encoding of the character repertoire you want to work with, and that's your responsibility. P. P. S. If always having to specify encodings is really too much, i'd probably be willing to consider a default-encoding state on the Unicode class, but it would have to be a stack of values, not a single value.
No automatic conversions between 8-bit "strings" and Unicode strings.
If you want to turn UTF-8 into a Unicode string, say so. If you want to turn Latin-1 into a Unicode string, say so. If you want to turn ISO-2022-JP into a Unicode string, say so. Adding a Unicode string and an 8-bit "string" gives an exception.
I'd accept this, with one change: mixing Unicode and 8-bit strings is okay when the 8-bit strings contain only ASCII (byte values 0 through 127). That does the right thing when the program is combining ASCII data (e.g. literals or data files) with Unicode and warns you when you are using characters for which the encoding matters. I believe that this is important because much existing code dealing with strings can in fact deal with Unicode just fine under these assumptions. (E.g. I needed only 4 changes to htmllib/sgmllib to make it deal with Unicode strings -- those changes were all getattr() and setattr() calls.) When *comparing* 8-bit and Unicode strings, the presence of non-ASCII bytes in either should make the comparison fail; when ordering is important, we can make an arbitrary choice e.g. "\377" < u"\200". Why not Latin-1? Because it gives us Western-alphabet users a false sense that our code works, where in fact it is broken as soon as you change the encoding.
P. S. The scare-quotes when i talk about 8-bit "strings" expose my sense of them as byte-buffers -- since that *is* all you get when you read in some bytes from a file. If you manipulate an 8-bit "string" as a character string, you are implicitly making the assumption that the byte values correspond to the character encoding of the character repertoire you want to work with, and that's your responsibility.
This is how I think of them too.
P. P. S. If always having to specify encodings is really too much, i'd probably be willing to consider a default-encoding state on the Unicode class, but it would have to be a stack of values, not a single value.
Please elaborate? --Guido van Rossum (home page: http://www.python.org/~guido/)
On Tue, 02 May 2000 08:31:55 -0400, Guido van Rossum <guido@python.org> wrote:
No automatic conversions between 8-bit "strings" and Unicode strings.
If you want to turn UTF-8 into a Unicode string, say so. If you want to turn Latin-1 into a Unicode string, say so. If you want to turn ISO-2022-JP into a Unicode string, say so. Adding a Unicode string and an 8-bit "string" gives an exception.
I'd accept this, with one change: mixing Unicode and 8-bit strings is okay when the 8-bit strings contain only ASCII (byte values 0 through 127). That does the right thing when the program is combining ASCII data (e.g. literals or data files) with Unicode and warns you when you are using characters for which the encoding matters. I believe that this is important because much existing code dealing with strings can in fact deal with Unicode just fine under these assumptions. (E.g. I needed only 4 changes to htmllib/sgmllib to make it deal with Unicode strings -- those changes were all getattr() and setattr() calls.)
When *comparing* 8-bit and Unicode strings, the presence of non-ASCII bytes in either should make the comparison fail; when ordering is important, we can make an arbitrary choice e.g. "\377" < u"\200".
I assume 'fail' means 'non-equal', rather than 'raises an exception'? Toby Dickenson tdickenson@geminidataloggers.com
[me]
When *comparing* 8-bit and Unicode strings, the presence of non-ASCII bytes in either should make the comparison fail; when ordering is important, we can make an arbitrary choice e.g. "\377" < u"\200".
[Toby]
I assume 'fail' means 'non-equal', rather than 'raises an exception'?
Yes, sorry for the ambiguity. --Guido van Rossum (home page: http://www.python.org/~guido/)
At 10:00 AM -0400 02-05-2000, Guido van Rossum wrote:
[me]
When *comparing* 8-bit and Unicode strings, the presence of non-ASCII bytes in either should make the comparison fail; when ordering is important, we can make an arbitrary choice e.g. "\377" < u"\200".
[Toby]
I assume 'fail' means 'non-equal', rather than 'raises an exception'?
Yes, sorry for the ambiguity.
You're going to have a hard time explaining that "\377" != u"\377". Again, if you define that "all strings are unicode" and that 8-bit strings contain Unicode characters up to 255, you're all set. Clear semantics, few surprises, simple implementation, etc. etc. Just
[Just]
You're going to have a hard time explaining that "\377" != u"\377".
I agree. You are an example of how hard it is to explain: you still don't understand that for a person using CJK encodings this is in fact the truth.
Again, if you define that "all strings are unicode" and that 8-bit strings contain Unicode characters up to 255, you're all set. Clear semantics, few surprises, simple implementation, etc. etc.
But not all 8-bit strings occurring in programs are Unicode. Ask Moshe. --Guido van Rossum (home page: http://www.python.org/~guido/)
[Just]
You're going to have a hard time explaining that "\377" != u"\377".
[GvR]
I agree. You are an example of how hard it is to explain: you still don't understand that for a person using CJK encodings this is in fact the truth.
That depends on the definition of truth: it you document that 8-bit strings are Latin-1, the above is the truth. Conceptually classify all other 8-bit encodings as binary goop makes the semantics chrystal clear.
Again, if you define that "all strings are unicode" and that 8-bit strings contain Unicode characters up to 255, you're all set. Clear semantics, few surprises, simple implementation, etc. etc.
But not all 8-bit strings occurring in programs are Unicode. Ask Moshe.
I know. They can be anything, even binary goop. But that's *only* an artifact of the fact that 8-bit strings need to double as buffer objects. Just
Guido van Rossum wrote:
...
But not all 8-bit strings occurring in programs are Unicode. Ask Moshe.
Where are we going? What's our long-range vision? Three years from now where will we be? 1. How will we handle characters? 2. How will we handle bytes? 3. What will unadorned literal strings "do"? 4. Will literal strings be the same type as byte arrays? I don't see how we can make decisions today without a vision for the future. I think that this is the central point in our disagreement. Some of us are aiming for as much compatibility with where we think we should be going and others are aiming for as much compatibility as possible with where we came from. -- Paul Prescod - ISOGEN Consulting Engineer speaking for himself It's difficult to extract sense from strings, but they're the only communication coin we can count on. - http://www.cs.yale.edu/~perlis-alan/quotes.html
Paul Prescod <paul@prescod.net>:
Where are we going? What's our long-range vision?
Three years from now where will we be?
1. How will we handle characters? 2. How will we handle bytes? 3. What will unadorned literal strings "do"? 4. Will literal strings be the same type as byte arrays?
I don't see how we can make decisions today without a vision for the future. I think that this is the central point in our disagreement. Some of us are aiming for as much compatibility with where we think we should be going and others are aiming for as much compatibility as possible with where we came from.
And *that* is the most insightful statement I have seen in this entire foofaraw (which I have carefully been staying right the hell out of). Everybody meditate on the above, please. Then declare your objectives *at this level* so our Fearless Leader can make an informed decision *at this level*. Only then will it make sense to argue encoding theology... -- <a href="http://www.tuxedo.org/~esr">Eric S. Raymond</a> "Extremism in the defense of liberty is no vice; moderation in the pursuit of justice is no virtue." -- Barry Goldwater (actually written by Karl Hess)
[Guido]
When *comparing* 8-bit and Unicode strings, the presence of non-ASCII bytes in either should make the comparison fail; when ordering is important, we can make an arbitrary choice e.g. "\377" < u"\200".
[Toby]
I assume 'fail' means 'non-equal', rather than 'raises an exception'?
[Guido]
Yes, sorry for the ambiguity.
Huh! You sure about that? If we're setting up a case where meaningful comparison is impossible, isn't an exception more appropriate? The current
83479278 < "42" 1
probably traps more people than it helps.
On Wed, 3 May 2000, Tim Peters wrote:
[Toby]
I assume 'fail' means 'non-equal', rather than 'raises an exception'?
[Guido]
Yes, sorry for the ambiguity.
Huh! You sure about that? If we're setting up a case where meaningful comparison is impossible, isn't an exception more appropriate? The current
83479278 < "42" 1
probably traps more people than it helps.
Yeah, when i said No automatic conversions between Unicode strings and 8-bit "strings". i was about to say Raise an exception on any operation attempting to combine or compare Unicode strings and 8-bit "strings". ...and then i thought, oh crap, but everything in Python is supposed to be comparable. What happens when you have some lists with arbitrary objects in them and you want to sort them for printing, or to canonicalize them so you can compare? It might be too troublesome for list.sort() to throw an exception because e.g. strings and ints were incomparable, or 8-bit "strings" and Unicode strings were incomparable... So -- what's the philosophy, Guido? Are we committed to "everything is comparable" (well, "all built-in types are comparable") or not? -- ?!ng
Ka-Ping Yee <ping@lfw.org> wrote:
So -- what's the philosophy, Guido? Are we committed to "everything is comparable" (well, "all built-in types are comparable") or not?
in 1.6a2, obviously not:
aUnicodeString < an8bitString Traceback (most recent call last): File "<stdin>", line 1, in ? UnicodeError: UTF-8 decoding error: unexpected code byte
in 1.6a3, maybe. </F>
[Guido]
When *comparing* 8-bit and Unicode strings, the presence of non-ASCII bytes in either should make the comparison fail; when ordering is important, we can make an arbitrary choice e.g. "\377" < u"\200".
[Toby]
I assume 'fail' means 'non-equal', rather than 'raises an exception'?
[Guido]
Yes, sorry for the ambiguity.
[Tim]
Huh! You sure about that? If we're setting up a case where meaningful comparison is impossible, isn't an exception more appropriate? The current
83479278 < "42" 1
probably traps more people than it helps.
Agreed, but that's the rule we all currently live by, and changing it is something for Python 3000. I'm not real strong on this though -- I was willing to live with exceptions from the UTF-8-to-Unicode conversion. If we all agree that it's better for u"\377" == "\377" to raise an precedent-setting exception than to return false, that's fine with me too. I do want u"a" == "a" to be true though (and I believe we all already agree on that one). Note that it's not the first precedent -- you can already define classes whose instances can raise exceptions during comparisons. --Guido van Rossum (home page: http://www.python.org/~guido/)
At 8:31 AM -0400 02-05-2000, Guido van Rossum wrote:
When *comparing* 8-bit and Unicode strings, the presence of non-ASCII bytes in either should make the comparison fail; when ordering is important, we can make an arbitrary choice e.g. "\377" < u"\200".
Blech. Just document 8-bit strings *are* Latin-1 unless converted explicitly, and you're done. It's really much simpler this way. For you as well as the users.
Why not Latin-1? Because it gives us Western-alphabet users a false sense that our code works, where in fact it is broken as soon as you change the encoding.
Yeah, and? It least it'll *show* it's broken instead of *silently* doing the wrong thing with utf-8. It's like using Python ints all over the place, and suddenly a user of the application enters data that causes an integer overflow. Boom. Program needs to be fixed. What's the big deal? Just
[Guido going ASCII] Do you mean going ASCII all the way (using it for all aspects where Unicode gets converted to a string and cases where strings get converted to Unicode), or just for some aspect of conversion, e.g. just for the silent conversions from strings to Unicode ? [BTW, I'm pretty sure that the Latin-1 folks won't like ASCII for the same reason they don't like UTF-8: it's simply an inconvenient way to write strings in their favorite encoding directly in Python source code. My feeling in this whole discussion is that it's more about convenience than anything else. Still, it's very amusing ;-) ] FYI, here's the conversion table of (potentially) all conversions done by the implementation: Python: ------- string + unicode: unicode(string,'utf-8') + unicode string.method(unicode): unicode(string,'utf-8').method(unicode) print unicode: print unicode.encode('utf-8'); with stdout redirection this can be changed to any other encoding str(unicode): unicode.encode('utf-8') repr(unicode): repr(unicode.encode('unicode-escape')) C (PyArg_ParserTuple): ---------------------- "s" + unicode: same as "s" + unicode.encode('utf-8') "s#" + unicode: same as "s#" + unicode.encode('unicode-internal') "t" + unicode: same as "t" + unicode.encode('utf-8') "t#" + unicode: same as "t#" + unicode.encode('utf-8') This effects all C modules and builtins. In case a C module wants to receive a certain predefined encoding, it can use the new "es" and "es#" parser markers. Ways to enter Unicode: ---------------------- u'' + string same as unicode(string,'utf-8') unicode(string,encname) any supported encoding u'...unicode-escape...' unicode-escape currently accepts Latin-1 chars as single-char input; using escape sequences any Unicode char can be entered (*) codecs.open(filename,mode,encname) opens an encoded file for reading and writing Unicode directly raw_input() + stdin redirection (see one of my earlier posts for code) returns UTF-8 strings based on the input encoding IO: --- open(file,'w').write(unicode) same as open(file,'w').write(unicode.encode('utf-8')) open(file,'wb').write(unicode) same as open(file,'wb').write(unicode.encode('unicode-internal')) codecs.open(file,'wb',encname).write(unicode) same as open(file,'wb').write(unicode.encode(encname)) codecs.open(file,'rb',encname).read() same as unicode(open(file,'rb').read(),encname) stdin + stdout can be redirected using StreamRecoders to handle any of the supported encodings -- Marc-Andre Lemburg ______________________________________________________________________ Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/
At 5:55 PM +0200 02-05-2000, M.-A. Lemburg wrote:
[BTW, I'm pretty sure that the Latin-1 folks won't like ASCII for the same reason they don't like UTF-8: it's simply an inconvenient way to write strings in their favorite encoding directly in Python source code. My feeling in this whole discussion is that it's more about convenience than anything else. Still, it's very amusing ;-) ]
For the record, I don't want Latin-1 because it's my favorite encoding. It isn't. Guido's right: I can't even *use* it derictly on my platform. I want it *only* because it's the most logical 8-bit subset of Unicode -- as we have stated over and opver and over and over again. What's so hard to understand about this? Just
Guido van Rossum wrote:
No automatic conversions between 8-bit "strings" and Unicode strings.
If you want to turn UTF-8 into a Unicode string, say so. If you want to turn Latin-1 into a Unicode string, say so. If you want to turn ISO-2022-JP into a Unicode string, say so. Adding a Unicode string and an 8-bit "string" gives an exception.
I'd accept this, with one change: mixing Unicode and 8-bit strings is okay when the 8-bit strings contain only ASCII (byte values 0 through 127).
I could live with this compromise as long as we document that a future version may use the "character is a character" model. I just don't want people to start depending on a catchable exception being thrown because that would stop us from ever unifying unmarked literal strings and Unicode strings. -- Are there any steps we could take to make a future divorce of strings and byte arrays easier? What if we added a binary_read() function that returns some form of byte array. The byte array type could be just like today's string type except that its type object would be distinct, it wouldn't have as many string-ish methods and it wouldn't have any auto-conversion to Unicode at all. People could start to transition code that reads non-ASCII data to the new function. We could put big warning labels on read() to state that it might not always be able to read data that is not in some small set of recognized encodings (probably UTF-8 and UTF-16). Or perhaps binary_open(). Or perhaps both. I do not suggest just using the text/binary flag on the existing open function because we cannot immediately change its behavior without breaking code. -- Paul Prescod - ISOGEN Consulting Engineer speaking for himself It's difficult to extract sense from strings, but they're the only communication coin we can count on. - http://www.cs.yale.edu/~perlis-alan/quotes.html
I could live with this compromise as long as we document that a future version may use the "character is a character" model. I just don't want people to start depending on a catchable exception being thrown because that would stop us from ever unifying unmarked literal strings and Unicode strings.
Agreed (as I've said before).
--
Are there any steps we could take to make a future divorce of strings and byte arrays easier? What if we added a
binary_read()
function that returns some form of byte array. The byte array type could be just like today's string type except that its type object would be distinct, it wouldn't have as many string-ish methods and it wouldn't have any auto-conversion to Unicode at all.
You can do this now with the array module, although clumsily:
import array f = open("/core", "rb") a = array.array('B', [0]) * 1000 f.readinto(a) 1000
Or if you wanted to read raw Unicode (UTF-16):
a = array.array('H', [0]) * 1000 f.readinto(a) 2000 u = unicode(a, "utf-16")
There are some performance issues, e.g. you have to initialize the buffer somehow and that seems a bit wasteful.
People could start to transition code that reads non-ASCII data to the new function. We could put big warning labels on read() to state that it might not always be able to read data that is not in some small set of recognized encodings (probably UTF-8 and UTF-16).
Or perhaps binary_open(). Or perhaps both.
I do not suggest just using the text/binary flag on the existing open function because we cannot immediately change its behavior without breaking code.
A new method makes most sense -- there are definitely situations where you want to read in text mode for a while and then switch to binary mode (e.g. HTTP). I'd like to put this off until after Python 1.6 -- but it deserves attention. --Guido van Rossum (home page: http://www.python.org/~guido/)
On Tue, 2 May 2000, Guido van Rossum wrote:
P. P. S. If always having to specify encodings is really too much, i'd probably be willing to consider a default-encoding state on the Unicode class, but it would have to be a stack of values, not a single value.
Please elaborate?
On general principle, it seems bad to just have a "set" method that encourages people to set static state in a way that irretrievably loses the current state. For something like this, you want a "push" method and a "pop" method with which to bracket a series of operations, so that you can easily write code which politely leaves other code unaffected. For example: >>> x = unicode("d\351but") # assume Guido-ASCII wins UnicodeError: ASCII encoding error: value out of range >>> x = unicode("d\351but", "latin-1") >>> x u'd\351but' >>> print x.encode("latin-1") # on my xterm with Latin-1 fonts d�but >>> x.encode("utf-8") 'd\303\251but' Now: >>> u"".pushenc("latin-1") # need a better interface to this? >>> x = unicode("d\351but") # okay now >>> x u'd\351but' >>> u"".pushenc("utf-8") >>> x = unicode("d\351but") UnicodeError: UTF-8 decoding error: invalid data >>> x = unicode("d\303\251but") >>> print x.encode("latin-1") d�but >>> str(x) 'd\303\251\but' >>> u"".popenc() # back to the Latin-1 encoding >>> str(x) 'd\351but' . . . >>> u"".popenc() # back to the ASCII encoding Similarly, imagine: >>> x = u"<Japanese text...>" >>> file = open("foo.jis", "w") >>> file.pushenc("iso-2022-jp") >>> file.uniwrite(x) . . . >>> file.popenc() >>> import sys >>> sys.stdout.write(x) # bad! x contains chars > 127 UnicodeError: ASCII decoding error: value out of range >>> sys.stdout.pushenc("iso-2022-jp") >>> sys.stdout.write(x) # on a kterm with kanji fonts <Japanese text...> . . . >>> sys.stdout.popenc() The above examples incorporate the Guido-ASCII proposal, which makes a fair amount of sense to me now. How do they look to y'all? This illustrates the remaining wart: >>> sys.stdout.pushenc("iso-2022-jp") >>> print x # still bad! str is still doing ASCII UnicodeError: ASCII decoding error: value out of range >>> u"".pushenc("iso-2022-jp") >>> print x # on a kterm with kanji fonts <Japanese text...> Writing to files asks the file object to convert from Unicode to bytes, then write the bytes. Printing converts the Unicode to bytes first with str(), then hands the bytes to the file object to write. This wart is really a larger printing issue. If we want to solve it, files have to know what to do with objects, i.e. print x doesn't mean sys.stdout.write(str(x) + "\n") instead it means sys.stdout.printout(x) Hmm. I think this might deserve a separate subject line. -- ?!ng
The following is all stolen from E: see http://www.erights.org/. As i mentioned in the previous message, there are reasons that we might want to enable files to know what it means to print things on them. print x would mean sys.stdout.printout(x) where sys.stdout is defined something like def __init__(self): self.encs = ["ASCII"] def pushenc(self, enc): self.encs.append(enc) def popenc(self): self.encs.pop() if not self.encs: self.encs = ["ASCII"] def printout(self, x): if type(x) is type(u""): self.write(x.encode(self.encs[-1])) else: x.__print__(self) self.write("\n") and each object would have a __print__ method; for lists, e.g.: def __print__(self, file): file.write("[") if len(self): file.printout(self[0]) for item in self[1:]: file.write(", ") file.printout(item) file.write("]") for floats, e.g.: def __print__(self, file): if hasattr(file, "floatprec"): prec = file.floatprec else: prec = 17 file.write("%%.%df" % prec % self) The passing of control between the file and the objects to be printed enables us to make Tim happy: >>> l = [1/2, 1/3, 1/4] # I can dream, can't i? >>> print l [0.3, 0.33333333333333331, 0.25] >>> sys.stdout.floatprec = 6 >>> print l [0.5, 0.333333, 0.25] Fantasizing about other useful kinds of state beyond "encs" and "floatprec" ("listmax"? "ratprec"?) and managing this namespace is left as an exercise to the reader. -- ?!ng
On Wed, 3 May 2000, Ka-Ping Yee wrote:
Fantasizing about other useful kinds of state beyond "encs" and "floatprec" ("listmax"? "ratprec"?) and managing this namespace is left as an exercise to the reader.
Okay, i lied. Shortly after writing this i realized that it is probably advisable for all such bits of state to be stored in stacks, so an interface such as this might do: def push(self, key, value): if not self.state.has_key(key): self.state[key] = [] self.state[key].append(value) def pop(self, key): if self.state.has_key(key): if len(self.state[key]): self.state[key].pop() def get(self, key): if not self.state.has_key(key): stack = self.state[key][-1] if stack: return stack[-1] return None Thus: >>> print 1/3 0.33333333333333331 >>> sys.stdout.push("float.prec", 6) >>> print 1/3 0.333333 >>> sys.stdout.pop("float.prec") >>> print 1/3 0.33333333333333331 And once we allow arbitrary strings as keys to the bits of state, the period is a natural separator we can use for managing the namespace. Take the special case for Unicode out of the file object: def printout(self, x): x.__print__(self) self.write("\n") and have the Unicode string do the work: def __printon__(self, file): file.write(self.encode(file.get("unicode.enc"))) This behaves just right if an encoding of None means ASCII. If mucking with encodings is sufficiently common, you could imagine conveniences on file objects such as def __init__(self, filename, mode, encoding=None): ... if encoding: self.push("unicode.enc", encoding) def pushenc(self, encoding): self.push("unicode.enc", encoding) def popenc(self, encoding): self.pop("unicode.enc") -- ?!ng
At 1:42 AM -0700 02-05-2000, Ka-Ping Yee wrote:
If it turns out automatic conversions *are* absolutely necessary, then i vote in favour of the simple, direct method promoted by Paul and Fredrik: just copy the numerical values of the bytes. The fact that this happens to correspond to Latin-1 is not really the point; the main reason is that it satisfies the Principle of Least Surprise.
Exactly. I'm not sure if automatic conversions are absolutely necessary, but seeing 8-bit strings as Latin-1 encoded Unicode strings seems most natural to me. Heck, even 8-bit strings should have an s.encode() method, that would behave *just* like u.encode(), and unicode(blah) could even *return* an 8-bit string if it turns out the string has no character codes > 255! Conceptually, this gets *very* close to the ideal of "there is only one string type", and at the same times leaves room for 8-bit strings doubling as byte arrays for backward compatibility reasons. (Unicode strings and 8-bit strings could even be the same type, which only uses wide chars when neccesary!) Just
participants (13)
-
Eric S. Raymond -
Fredrik Lundh -
Guido van Rossum -
Just van Rossum -
Ka-Ping Yee -
M.-A. Lemburg -
Mark Hammond -
Neil Hodgson -
Paul Prescod -
Tim Peters -
Toby Dickenson -
Tom Emerson -
tpassin＠home.com