Mailman 3 July 2000 - Python-Dev

RE: [Python-Dev] UTF-16 code point comparison
by Bill Tutt July 27, 2000

July 27, 2000

-----Original Message----- > From: Guido van Rossum [mailto:guido@beopen.com] > The invalidity of 0xD800-0xDFFF means that these aren't valid Unicode > code points; it doesn't mean that we should trap all attempts to use > these values. That ways, apps that need UTF-16 awareness can code it > themselves. 0xD800-0xDFFF are valid code points. They just aren't characters that you normally want to render all on their own. :) > Why? Because I don't want to proliferate code that explicitly traps > 0xD800-0xDFFF throughout the code. Err... I don't think you have much choice in the long term. Its not like the code is all that difficult, and certain cases might be able to use helper functions to help centralize that particular part of code. Bill

4 3

Re: [Python-checkins] CVS: python/nondist/peps pep-0160.txt,1.3,1.4
by Fredrik Lundh July 27, 2000

July 27, 2000

fred wrote: > Remove the ugly form-feed characters some people use with Emacs. note that those "ugly" form-feeds become nice <hr> and <h3> in the HTML versions... </F>

3 2

RE: [Python-Dev] UTF-16 code point comparison
by Bill Tutt July 27, 2000

July 27, 2000

> From: bckfnn(a)worldonline.dk [mailto:bckfnn@worldonline.dk] > - The BreakIterator does not handle surrogates. It does handle > combining characters and it seems a natural place to put support > for surrogates. > - The Collator class offers different levels of normalization before > comparing string but does not seem to support surrogates. This class > seems a natural place for javasoft to put support for surrogates > during string comparison. Both of these aren't surprising given that there aren't any officially allocated characters in surrogate land (but they're coming fast), and the most likely first allocations are to the combined CJK glyph space which doesn't really have anything to collate. :) Bill

1 0

RE: [Python-Dev] UTF-16 code point comparison
by Bill Tutt July 27, 2000

July 27, 2000

> umm. the Java docs I have access to doesn't mention surrogates > at all (they do point out that a character is 16-bit, and they don't > provide an \U escape). When Java does support surrogates, it seems likely that for backward compatibility reasons that they'll start paying attention to surrogates. Altering to a 32-bit int would break too much of their users code. > on the other hand, MSDN says: > Windows 2000 provides support for basic input, output, and > simple sorting of surrogates. However, not all Windows 2000 > system components are surrogate compatible. Also, surrogates > are not supported in Windows 95/98 or in Windows NT 4.0. > and then mentions all the usual problems with variable-width > encodings... Which means it supports UTF-16, and the support can only get better. > > > after all, if variable-width internal storage had been easy to deal > > > with, we could have used UTF-8 from the start... (and just like > > > the Tcl folks, we would have ended up rewriting the whole thing > > > in the next release ;-) > >=20 > > Oh please, UTF-16 is substantially simpler to deal with than UTF-8. > in what way? as in "one or two words" is simpler than "one, two, > three, four, five, or six bytes"? > or as in "nobody will notice anyway..." ;-) As in it's very easy to determine arbitrarily which byte of the surrogate you're dealing with based on its 16-bit value. You can't say that about UTF-8. > if UCS-2/BMP was good enough for NT 4.0, Unicode 1.1, and Java 1.0, > it's surely good enough for Python 2.0 ;-) > > (and if I understand things correctly, 2.1 isn't that far away...) They were, since UTF-16 didn't exist at the time. :) I think we both want Python's unicode support to eventually support surrogate range characters. Let me see if I can reiterate what we're both trying to say. What you're saying: * We might switch to UCS-4 when we support this extra stuff because variable length encodings are annoying. My counter point: Switching to UCS-4 isn't backward compatible. If you want to avoid variable length encoding then start with UCS-4 from the beginning. BUT, keeping in mind that Windows, and Java are either right now, or likely to be UTF-16 systems. What I'm saying: * Just use UTF-16 and be done with it. Windows is using it, and Java if it isn't already is definitely likely to do so. Your counter point: Supporting UTF-16 is complicated, and we don't want to do all of this for Python 2.0. (Especially anything that involves writing a Unicode string object that hides the surrogate as two 16-bit entities) My response: This is true. I've never suggested we do all of the UTF-16 support for Python 2.0. The UTF-16 code point order comparision patch I submitted was just something I noticed online, and submitted the patch more for feedback and comments rather then wanting, or needing the patch to get in. However, that doesn't mean bad things will happen if we allow the UTF-8/16 (en/de)coders handle surrogate characters. The Python code can worry about this variable length encoding on its own. The core should still be able to UTF-8 the unicode string so that it can be pickled however. Did I get what you're saying right, and do you understand what I'm getting at? Bill -----Original Message----- From: Fredrik Lundh [mailto:effbot@telia.com] Sent: Thursday, July 27, 2000 8:35 AM To: Bill Tutt Cc: python-dev(a)python.org Subject: Re: [Python-Dev] UTF-16 code point comparison This message uses a character set that is not supported by the Internet Service. To view the original message content, open the attached message. If the text doesn't display correctly, save the attachment to disk, and then open it using a viewer that can display the original character set. << File: message.txt >>

1 0

RE: [Python-Dev] UTF-16 code point comparison
by Bill Tutt July 27, 2000

July 27, 2000

Fredrik: > sorry, but you're being silly. using variable-width encoding for > interal storage is difficult, slow, and just plain stupid on modern > hardware. So use UCS-4 internal storage now. UTF-16 just seems like a handy internal storage mechanism to pick since Win32 and Java use it for their native string processing. > (image processing people stopped doing stupid things like that > ages ago, and trust me -- a typical image contains many more > pixels than a typical text ;-) > after all, if variable-width internal storage had been easy to deal > with, we could have used UTF-8 from the start... (and just like > the Tcl folks, we would have ended up rewriting the whole thing > in the next release ;-) Oh please, UTF-16 is substantially simpler to deal with than UTF-8. I would go nuts if our internal storage mechanism was UTF-8. Bill

2 1

was Re: Type inference now simplicity (fwd)
by Moshe Zadka July 27, 2000

July 27, 2000

I think most of the people here aren't reading c.l.py, so here are some feelings from "the community" -- Moshe Zadka <moshez(a)math.huji.ac.il> There is no IGLU cabal. http://advogato.org/person/moshez ---------- Forwarded message ---------- Date: Wed, 26 Jul 2000 21:14:34 GMT From: Shae Erisson <shapr(a)uab.edu> To: python-list(a)python.org Newsgroups: comp.lang.python Subject: was Re: Type inference now simplicity Neil Hodgson wrote: > I found this article quite depressing. One of the effects of .NET appears > to be the homogenisation of programming languages. When the platform > emphasises compilation, compiler helpful warts like variable declaration and > "option fast" start appearing. There is a place for languages with optional > typing but there is also a place for a language that tries to stay very > simple. > > VB has been on a long journey of complexification which has been great > for many but has left some people behind. One of my mates works as a > manager/bureaucrat - programming isn't a significant part of his job but > over the years he's been able to write bits of dBase, Excel macros and HTML > with embedded JavaScript. Excel moved from a simple macro language to a > simplified VB variant (OK, so far) to using full VB (getting worse) and then > VB starts gaining C++ features (bad). So this bloke has stopped programming > in Excel - just too hard now. Adding 'programming in the large' features > appears good to help the more advanced users but it introduces unwanted > complexity for others. JavaScript was the last refuge of simplicity but now > Microsoft (and Netscape) are moving it into the same zone as Python, VB and > C#. I expect a new simple language will appear, gain some users and then > embark on the same journey to complexity. I agree with this. With Python, I'm opposed to the idea of adding autoincrement operators like += etc. I'm opposed to adding the extra for loop stuff like "for [x,y,z] in [a,b,c]." And I'm opposed to it because it seems like sugar to me. I've always loved Python's simplicity, and I hope it stays that way. Adding new operators and new syntax will make Python harder to read, and harder to learn. I don't want that. Simplicity always seems to improve things. Usually, I can improve by removing rather than adding. Concepts like WikiWiki, Refactoring, etc. appeal to me tremendously. -- Shae Matijs Erisson - http://www.webwitches.com/~shae/ VirtualPairProgramming Wanted - Linux/Emacs/Python/Speak Freely .fi: rakastan ohjelmointia - python kengitt�� aasia -- http://www.python.org/mailman/listinfo/python-list

2 3

RE: [Python-Dev] UTF-16 code point comparison
by Bill Tutt July 27, 2000

July 27, 2000

> From: M.-A. Lemburg [mailto:mal@lemburg.com] > Fredrik Lundh wrote: > > > > mal wrote: > > > This really has nothing to do with being able to support > > > surrogates or not (as Fredrik mentioned), it is the correct > > > behaviour provided UTF-16 is used as encoding for UCS-4 values > > > in Unicode literals which is what Python currently does. > > > > Really? I could have sworn that most parts of Python use > > UCS-2, not UTF-16. > The design specifies that Py_UNICODE refers to UTF-16. To make > life easier, the implementation currently assumes UCS-2 in > many parts, but this is should only be considered a > temporary situation. Since supporting UTF-16 poses some > real challenges (being a variable length encoding), full > support for surrogates was postponed to some future > implementation. Heh. Now you're being silly. Supporting UTF-16 isn't that difficult. You always know whether the character is a low surrogate or a high surrogate. The interesting question is whether or not you have your builtin Unicode object expose each 16-bit character as is, or you support iterating over Py_UCS4 characters, or you want to have a wrapping object that does the right thing here. This might be the way to go. > > Built-ins like ord, unichr, len; slicing; > > string methods; regular expressions, etc. all clearly assume > > that a Py_UNICODE is a unicode code point. > > > > My point is that we shouldn't pretend we're supporting > > UTF-16 if we don't do that throughout. > We should keep that design detail in mind though. > > As far as I can tell, cmp() is the *only* unicode function > > that thinks the internal storage is UTF-16. > > > Everything else assumes UCS-2. No, its UTF-16, it just doesn't yet handle surrogates in all of the appropriate places. :) The unicodename stuff also needs to support future named surrogate characters now. > > And for Python 2.0, it's surely easier to fix cmp() than to > > fix everything else. > Also true :-) Everything but the regular expressions would be fairly simple to add UTF-16 support to. I'd imagine that adding support for \u10FFFF in the regular expression syntax wouldn't be that hard either. > > (this is the same problem as 8-bit/unicode comparisions, where > > the current consensus is that it's better to raise an exception > > if it looks like the programmer doesn't know what he was doing, > > rather than pretend it's another encoding). > Perhaps you are right and we should #if 0 the comparison > sections related to UTF-16 for now. I'm not sure why Bill > needed the cmp() function to support surrogates... Bill ? I didn't need it to. I happened upon the code on the IBM website, so I figured I'd point it out and see what people thought of sticking it into the Python Unicode stuff. :) (Wishing Python 2.0 would ship with Unicode collation support) See the earlier comment about creating a wrapping class that handles UTF-16 issues better. > Still, it will have to be reenabled sometime in the > future when full surrogate support is added to Python. Bill

2 2

RE: [Python-Dev] UTF-16 code point comparison
by Bill Tutt July 27, 2000

July 27, 2000

Fredik wrote: > the original unicode implementation did just that, but Bill and > Marc-Andre recently lowered the shields: the UTF-8 decoder > now generates UTF-16 encoded data. (so does \N{}, but > that's a non-issue:=20 > my proposal is to tighten this up in 2.0: ifdef out the UTF-16 > code in the UTF-8 decoder, and ifdef out the UTF-16 stuff in > the compare function. Commenting the UTF-16 stuff out in the compare function is a valid point, given our current Unicode string object. I disagree strongly with disabling the surrogate support in UTF-8, and we should fix the UTF-16 decoder. Since the decoder/encoder support doesn't hurt any other piece of code by emitting surrogate pairs, I don't see why you want to disable the code. > (oddly enough, the UTF-16 decoder won't accept anything > that isn't UCS-2 ;-) Well that's an easy bug to fix. > let's wait until 2.1 before we support the full unicode character > set (and I'm pretty sure "the right way" is UCS-4 storage and a > unified string implementation, but let's discuss that later). I've mentioned this before, but why discuss this later? Indeed why would we want to fix it for 2.1? Esp. if we move to UCS-4 storage in a minor release. Why not just get the Unicode support correct this time around. Save the poor users of the Python Unicode support from going nuts when we make these additional confusing changes later. If you think you want to move to UCS-4 later, don't wait, do it know. Add support for special surrogate handling later if we must, but please oh please don't change the storage mechanism in memory in a later relase. Java and Win32 are all UTF-16 based, and those extra 16-bits are actually wasted for nearly every common Unicode data you'd hope to handle. I think using UTF-16 as an internal storage mechanism actually makes sense. Whether or not you want to have your character type expose an array of 16-bit values, or the appearance of an array of UCS-4 characters is a separate issue. An issue I think should be answered now, instead of fixing it later. Bill

1 0

RE: [Python-Dev] UTF-16 code point comparison
by Bill Tutt July 27, 2000

July 27, 2000

Fredrik wrote: > since we don't really support surrogates (or anything else > related to 32-bit unicode) anywhere else, I'd rather change > CPython to emulate JPython here. That's not entirely true. UTF-8 encoding supports surrogates. No reason that we couldn't put the code into a unicode helper function module. If I get a chance to finish my port of the simple Unicode compression thing that'll support surrogates as well. We're eventually going to need one of these anyway for additional Unicode related algorithms. Collation, etc... Bill

1 0

is this obselete?: [Patch #100888] Fix UCNs machine with >= 32bit longs
by Trent Mick July 27, 2000

July 27, 2000

I was assigned this patch a while back to try on some 64-bit systems. Fine, I can do that but I have since seen a number of message on python-dev regarding it: - Fredrik had a new, better replacement? Fredrik? - The name changed to unicodedata? MAL? - Bill, you said that the patch is no longer necessary? Anyway, can someone familiar with the patch tell me if I should check it on 64-bit systems (Win64, Linux64) and then who I should pass the patch on to? OR should I just mark it Out of Date or Rejected? Thanks, Trent -- Trent Mick TrentM(a)ActiveState.com

4 3