CPythons unicode compare function contains some code to compare surrogate characters in code-point order (I think). This is properly a very neat feature but is differs from java's way of comparing strings. Python 2.0b1 (#0, Jul 26 2000, 21:29:11) [MSC 32 bit (Intel)] on win32 Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam Copyright 1995-2000 Corporation for National Research Initiatives (CNRI)
print u'\ue000' < u'\ud800' 1 print ord(u'\ue000') < ord(u'\ud800') 0
Java (and JPython) compares the 16-bit characters numericly which result in: JPython 1.1+08 on java1.3.0 (JIT: null) Copyright (C) 1997-1999 Corporation for National Research Initiatives
print u'\ue000' < u'\ud800' 0 print ord(u'\ue000') < ord(u'\ud800') 0
I don't think I can come up with a solution that allow JPython to emulate CPython on this type of comparison. regards, finn
finn bock wrote:
Java (and JPython) compares the 16-bit characters numericly which result in:
JPython 1.1+08 on java1.3.0 (JIT: null) Copyright (C) 1997-1999 Corporation for National Research Initiatives
print u'\ue000' < u'\ud800' 0 print ord(u'\ue000') < ord(u'\ud800') 0
I don't think I can come up with a solution that allow JPython to emulate CPython on this type of comparison.
since we don't really support surrogates (or anything else related to 32-bit unicode) anywhere else, I'd rather change CPython to emulate JPython here. (just #ifdef out the neat stuff, and don't switch it on again before we've sorted all other 32-bit unicode issues...) </F>
Finn Bock wrote:
CPythons unicode compare function contains some code to compare surrogate characters in code-point order (I think). This is properly a very neat feature but is differs from java's way of comparing strings.
Python 2.0b1 (#0, Jul 26 2000, 21:29:11) [MSC 32 bit (Intel)] on win32 Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam Copyright 1995-2000 Corporation for National Research Initiatives (CNRI)
print u'\ue000' < u'\ud800' 1 print ord(u'\ue000') < ord(u'\ud800') 0
Java (and JPython) compares the 16-bit characters numericly which result in:
JPython 1.1+08 on java1.3.0 (JIT: null) Copyright (C) 1997-1999 Corporation for National Research Initiatives
print u'\ue000' < u'\ud800' 0 print ord(u'\ue000') < ord(u'\ud800') 0
I don't think I can come up with a solution that allow JPython to emulate CPython on this type of comparison.
The code originally worked the same way as what Java does here. Bill Tutt then added ideas from some IBM Java lib which turns the UTF-16 comparison into a true UCS-4 comparison. This really has nothing to do with being able to support surrogates or not (as Fredrik mentioned), it is the correct behaviour provided UTF-16 is used as encoding for UCS-4 values in Unicode literals which is what Python currently does. BTW, does Java support UCS-4 ? If not, then Java is wrong here ;-) Comparing Unicode strings is not as trivial as one might think: private point areas introduce a great many possibilities of getting things wrong and the fact that many characters can be expressed by combining other characters adds to the confusion. E.g. for sorting, we'd need full normalization support for Unicode and would have to come up with some smart strategy to handle private code point areas. All this is highly non-trivial and will probably not get implemented for a while (other issues are more important right now, e.g. getting the internals to use the default encoding instead of UTF-8). For now I'd suggest leaving Bill's code activated because it does the right thing for Python's Unicode implementation (which is built upon UTF-16). -- Marc-Andre Lemburg ______________________________________________________________________ Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/
mal wrote:
This really has nothing to do with being able to support surrogates or not (as Fredrik mentioned), it is the correct behaviour provided UTF-16 is used as encoding for UCS-4 values in Unicode literals which is what Python currently does.
Really? I could have sworn that most parts of Python use UCS-2, not UTF-16. Built-ins like ord, unichr, len; slicing; string methods; regular expressions, etc. all clearly assume that a Py_UNICODE is a unicode code point. My point is that we shouldn't pretend we're supporting UTF-16 if we don't do that throughout. As far as I can tell, cmp() is the *only* unicode function that thinks the internal storage is UTF-16. Everything else assumes UCS-2. And for Python 2.0, it's surely easier to fix cmp() than to fix everything else. (this is the same problem as 8-bit/unicode comparisions, where the current consensus is that it's better to raise an exception if it looks like the programmer doesn't know what he was doing, rather than pretend it's another encoding). ::: To summarize, here's the "character encoding guidelines" for Python 2.0: In Unicode context, 8-bit strings contain ASCII. Characters in the 0x80-0xFF range are invalid. 16-bit strings contain UCS-2. Characters in the 0xD800-0xDFFF range are invalid. If you want to use any other encoding, use the codecs pro- vided by the Unicode subsystem. If you need to use Unicode characters that cannot be represented as UCS-2, you cannot use Python 2.0's Unicode subsystem. Anything else is just a hack. </F>
Fredrik Lundh wrote:
mal wrote:
This really has nothing to do with being able to support surrogates or not (as Fredrik mentioned), it is the correct behaviour provided UTF-16 is used as encoding for UCS-4 values in Unicode literals which is what Python currently does.
Really? I could have sworn that most parts of Python use UCS-2, not UTF-16.
The design specifies that Py_UNICODE refers to UTF-16. To make life easier, the implementation currently assumes UCS-2 in many parts, but this is should only be considered a temporary situation. Since supporting UTF-16 poses some real challenges (being a variable length encoding), full support for surrogates was postponed to some future implementation.
Built-ins like ord, unichr, len; slicing; string methods; regular expressions, etc. all clearly assume that a Py_UNICODE is a unicode code point.
My point is that we shouldn't pretend we're supporting UTF-16 if we don't do that throughout.
We should keep that design detail in mind though.
As far as I can tell, cmp() is the *only* unicode function that thinks the internal storage is UTF-16.
Everything else assumes UCS-2.
True.
And for Python 2.0, it's surely easier to fix cmp() than to fix everything else.
Also true :-)
(this is the same problem as 8-bit/unicode comparisions, where the current consensus is that it's better to raise an exception if it looks like the programmer doesn't know what he was doing, rather than pretend it's another encoding).
Perhaps you are right and we should #if 0 the comparison sections related to UTF-16 for now. I'm not sure why Bill needed the cmp() function to support surrogates... Bill ? Still, it will have to be reenabled sometime in the future when full surrogate support is added to Python.
:::
To summarize, here's the "character encoding guidelines" for Python 2.0:
In Unicode context, 8-bit strings contain ASCII. Characters in the 0x80-0xFF range are invalid. 16-bit strings contain UCS-2. Characters in the 0xD800-0xDFFF range are invalid.
The latter is not true. In fact, thanks to Bill, the UTF-8 codec supports processing surrogates already and will output correct UTF-8 code even for Unicode strings containing surrogates.
If you want to use any other encoding, use the codecs pro- vided by the Unicode subsystem. If you need to use Unicode characters that cannot be represented as UCS-2, you cannot use Python 2.0's Unicode subsystem.
See above. -- Marc-Andre Lemburg ______________________________________________________________________ Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/
As far as I can tell, cmp() is the *only* unicode function that thinks the internal storage is UTF-16.
Everything else assumes UCS-2.
And for Python 2.0, it's surely easier to fix cmp() than to fix everything else.
Agreed (I think).
(this is the same problem as 8-bit/unicode comparisions, where the current consensus is that it's better to raise an exception if it looks like the programmer doesn't know what he was doing, rather than pretend it's another encoding).
:::
To summarize, here's the "character encoding guidelines" for Python 2.0:
In Unicode context, 8-bit strings contain ASCII. Characters in the 0x80-0xFF range are invalid. 16-bit strings contain UCS-2. Characters in the 0xD800-0xDFFF range are invalid.
If you want to use any other encoding, use the codecs pro- vided by the Unicode subsystem. If you need to use Unicode characters that cannot be represented as UCS-2, you cannot use Python 2.0's Unicode subsystem.
Anything else is just a hack.
I wouldn't go so far as raising an exception when a comparison involves 0xD800-0xDFFF; after all we don't raise an exception when an ASCII string contains 0x80-0xFF either (except when converting to Unicode). The invalidity of 0xD800-0xDFFF means that these aren't valid Unicode code points; it doesn't mean that we should trap all attempts to use these values. That ways, apps that need UTF-16 awareness can code it themselves. Why? Because I don't want to proliferate code that explicitly traps 0xD800-0xDFFF throughout the code. --Guido van Rossum (home page: http://www.pythonlabs.com/~guido/)
[me]
To summarize, here's the "character encoding guidelines" for Python 2.0:
In Unicode context, 8-bit strings contain ASCII. Characters in the 0x80-0xFF range are invalid. 16-bit strings contain UCS-2. Characters in the 0xD800-0xDFFF range are invalid.
If you want to use any other encoding, use the codecs pro- vided by the Unicode subsystem. If you need to use Unicode characters that cannot be represented as UCS-2, you cannot use Python 2.0's Unicode subsystem.
Anything else is just a hack.
[guido]
I wouldn't go so far as raising an exception when a comparison involves 0xD800-0xDFFF; after all we don't raise an exception when an ASCII string contains 0x80-0xFF either (except when converting to Unicode).
that's what the "unicode context" qualifier means: 8-bit strings can contain anything, unless you're using them as unicode strings.
The invalidity of 0xD800-0xDFFF means that these aren't valid Unicode code points; it doesn't mean that we should trap all attempts to use these values. That ways, apps that need UTF-16 awareness can code it themselves.
Why? Because I don't want to proliferate code that explicitly traps 0xD800-0xDFFF throughout the code.
you only need to check "on the way in", and leave it to the decoders to make sure they generate UCS-2 only. the original unicode implementation did just that, but Bill and Marc-Andre recently lowered the shields: the UTF-8 decoder now generates UTF-16 encoded data. (so does \N{}, but that's a non-issue: (oddly enough, the UTF-16 decoder won't accept anything that isn't UCS-2 ;-) my proposal is to tighten this up in 2.0: ifdef out the UTF-16 code in the UTF-8 decoder, and ifdef out the UTF-16 stuff in the compare function. let's wait until 2.1 before we support the full unicode character set (and I'm pretty sure "the right way" is UCS-4 storage and a unified string implementation, but let's discuss that later). patch coming. </F>
my proposal is to tighten this up in 2.0: ifdef out the UTF-16 code in the UTF-8 decoder, and ifdef out the UTF-16 stuff in the compare function.
patch coming.
http://sourceforge.net/patch/index.php?func=detailpatch&patch_id=100990&group_id=5470 </F>
[M.-A. Lemburg]
BTW, does Java support UCS-4 ? If not, then Java is wrong here ;-)
Java claims to use unicode 2.1 [*]. I couldn't locate anything describing if this is UCS-2 or UTF-16. I think unicode 2.1 includes UCS-4. The actual level of support for UCS-4 is properly debatable. - The builtin char is 16bit wide and can obviously not support UCS-4. - The Character class can report if a character is a surrogate: >>> from java.lang import Character >>> Character.getType("\ud800") == Character.SURROGATE 1 - As reported, direct string comparison ignore surrogates. - The BreakIterator does not handle surrogates. It does handle combining characters and it seems a natural place to put support for surrogates. - The Collator class offers different levels of normalization before comparing string but does not seem to support surrogates. This class seems a natural place for javasoft to put support for surrogates during string comparison. These findings are gleaned from the sources of JDK1.3 [*] http://java.sun.com/docs/books/vmspec/2nd-edition/html/Concepts.doc.html#253... regards, finn
Finn Bock wrote:
[M.-A. Lemburg]
BTW, does Java support UCS-4 ? If not, then Java is wrong here ;-)
Java claims to use unicode 2.1 [*]. I couldn't locate anything describing if this is UCS-2 or UTF-16. I think unicode 2.1 includes UCS-4. The actual level of support for UCS-4 is properly debatable.
- The builtin char is 16bit wide and can obviously not support UCS-4. - The Character class can report if a character is a surrogate: >>> from java.lang import Character >>> Character.getType("\ud800") == Character.SURROGATE 1
unicodedata.category(u'\ud800') 'Cs'
... which means the same thing only in Unicode3 standards notation. Make me think: perhaps we should add the Java constants to unicodedata base. Is there a list of those available somewhere ?
- As reported, direct string comparison ignore surrogates.
I would guess that this'll have to change as soon as JavaSoft folks realize that they need to handle UTF-16 and not only UCS-2.
- The BreakIterator does not handle surrogates. It does handle combining characters and it seems a natural place to put support for surrogates.
What is a BreakIterator ? An iterator to scan line/word breaks ?
- The Collator class offers different levels of normalization before comparing string but does not seem to support surrogates. This class seems a natural place for javasoft to put support for surrogates during string comparison.
We'll need something like this for 2.1 too: first some standard APIs for normalization and then a few unicmp() APIs to use for sorting. We might even have to add collation sequences somewhere because this is a locale issue as well... sometimes it's even worse with different strategies used for different tasks within one locale, e.g. in Germany we sometimes sort the Umlaut ä as "ae" and at other times as extra character.
These findings are gleaned from the sources of JDK1.3
[*] http://java.sun.com/docs/books/vmspec/2nd-edition/html/Concepts.doc.html#253...
Thanks for the infos, -- Marc-Andre Lemburg ______________________________________________________________________ Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/
[M.-A. Lemburg]
Finn Bock wrote:
[M.-A. Lemburg]
BTW, does Java support UCS-4 ? If not, then Java is wrong here ;-)
Java claims to use unicode 2.1 [*]. I couldn't locate anything describing if this is UCS-2 or UTF-16. I think unicode 2.1 includes UCS-4. The actual level of support for UCS-4 is properly debatable.
- The builtin char is 16bit wide and can obviously not support UCS-4. - The Character class can report if a character is a surrogate: >>> from java.lang import Character >>> Character.getType("\ud800") == Character.SURROGATE 1
unicodedata.category(u'\ud800') 'Cs'
... which means the same thing only in Unicode3 standards notation.
Make me think: perhaps we should add the Java constants to unicodedata base. Is there a list of those available somewhere ?
UNASSIGNED = 0 UPPERCASE_LETTER LOWERCASE_LETTER TITLECASE_LETTER MODIFIER_LETTER OTHER_LETTER NON_SPACING_MARK ENCLOSING_MARK COMBINING_SPACING_MARK DECIMAL_DIGIT_NUMBER LETTER_NUMBER OTHER_NUMBER SPACE_SEPARATOR LINE_SEPARATOR PARAGRAPH_SEPARATOR CONTROL FORMAT PRIVATE_USE SURROGATE DASH_PUNCTUATION START_PUNCTUATION END_PUNCTUATION CONNECTOR_PUNCTUATION OTHER_PUNCTUATION MATH_SYMBOL CURRENCY_SYMBOL MODIFIER_SYMBOL OTHER_SYMBOL
- As reported, direct string comparison ignore surrogates.
I would guess that this'll have to change as soon as JavaSoft folks realize that they need to handle UTF-16 and not only UCS-2.
Predicting the future can be difficult, but here is my take: javasoft will never change the way String.compareTo works. String.compareTo is documented as: """ Compares two strings lexicographically. The comparison is based on the Unicode value of each character in the strings. ... """ Instead they will mark it as a very naive string comparison and suggest users to use the Collator classes for anything but the simplest cases.
- The BreakIterator does not handle surrogates. It does handle combining characters and it seems a natural place to put support for surrogates.
What is a BreakIterator ? An iterator to scan line/word breaks ?
Yes, as well as character breaks. It already contains the framework for marking two chars next to each other as one.
- The Collator class offers different levels of normalization before comparing string but does not seem to support surrogates. This class seems a natural place for javasoft to put support for surrogates during string comparison.
We'll need something like this for 2.1 too: first some standard APIs for normalization and then a few unicmp() APIs to use for sorting.
We might even have to add collation sequences somewhere because this is a locale issue as well... sometimes it's even worse with different strategies used for different tasks within one locale, e.g. in Germany we sometimes sort the Umlaut ä as "ae" and at other times as extra character.
Info: The java Collator class is configured with - a locale and - a strengh parameter IDENTICAL; all difference are significant. PRIMARY (a vs b) SECONDARY (a vs ä) TERTIARY (a vs A) - a decomposition (http://www.unicode.org/unicode/reports/tr15/) NO_DECOMPOSITION CANONICAL_DECOMPOSITION FULL_DECOMPOSITION regards, finn
Predicting the future can be difficult, but here is my take: javasoft will never change the way String.compareTo works. String.compareTo is documented as: """ Compares two strings lexicographically. The comparison is based on the Unicode value of each character in the strings. ... """
(Noting that their definition of "character" is probably "a 16-bit value of type char", and has only fleeting resemblance to what is or is not defined as a character by the Unicode standard.)
Instead they will mark it as a very naive string comparison and suggest users to use the Collator classes for anything but the simplest cases.
Without having digested the entire discussion, this sounds like a good solution for Python too. The "==" operator should compare strings based on a simple-minded representation-oriented definition, and all the other stuff gets pushed into separate methods or classes. --Guido van Rossum (home page: http://www.pythonlabs.com/~guido/)
Guido van Rossum wrote:
Predicting the future can be difficult, but here is my take: javasoft will never change the way String.compareTo works. String.compareTo is documented as: """ Compares two strings lexicographically. The comparison is based on the Unicode value of each character in the strings. ... """
(Noting that their definition of "character" is probably "a 16-bit value of type char", and has only fleeting resemblance to what is or is not defined as a character by the Unicode standard.)
Instead they will mark it as a very naive string comparison and suggest users to use the Collator classes for anything but the simplest cases.
Without having digested the entire discussion, this sounds like a good solution for Python too. The "==" operator should compare strings based on a simple-minded representation-oriented definition, and all the other stuff gets pushed into separate methods or classes.
This would probably be the best way to go: we'll need collation routines sooner or later anyway. Bill's "true UCS-4" compare could then become part of that lib. Should I #if 0 the current implementation of the UCS-4 compare in CVS ? -- Marc-Andre Lemburg ______________________________________________________________________ Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/
participants (4)
-
bckfnn＠worldonline.dk -
Fredrik Lundh -
Guido van Rossum -
M.-A. Lemburg