[AMK]

The proposal seems reasonable to me.

Thanks. I really hope that this time we can move forward united...

...

(Together with a new Unicode regex engine by /F.)

This is good news! Would it be a from-scratch regex implementation, or would it be an adaptation of an existing engine? Would it involve modifications to the existing re module, or a completely new unicodere module? (If, unlike re.py, it has POSIX longest-match semantics, that would pretty much settle the question.)

It's from scratch, and I believe it's got Perl style, not POSIX style semantics -- per Tim Peters' recommendations. Do we need to open the discussion again? It involves a redone re module (supporting Unicode as well as 8-bit), but its API could be unchanged. /F does the parsing and compilation in Python, only the matching engine is in C -- not sure how that impacts performance, but I imagine with aggressive caching it would be okay. --Guido van Rossum (home page: http://www.python.org/~guido/)

On Tue, 9 Nov 1999, Andrew M. Kuchling wrote:

Guido van Rossum writes:

...

It's from scratch, and I believe it's got Perl style, not POSIX style semantics -- per Tim Peters' recommendations. Do we need to open the discussion again?

No, no; I'm actually happier with Perl-style, because it's far better documented and familiar to people. Worse *is* better, after all.

I would concur with the preference for Perl-style semantics. Aside from the issue of consistency with other scripting languages, i think it's easier to predict the behaviour of these semantics. You can run the algorithm in your head, and try the backtracking yourself. It's good for the algorithm to be predictable and well understood.

Doing the compilation in Python is a good idea, and will make it possible to implement alternative syntaxes.

Also agree. I still have some vague wishes for a simpler, more readable (more Pythonian?) way to express patterns -- perhaps not as powerful as full regular expressions, but useful for many simpler cases (an 80-20 solution). -- ?!ng

Barry A. Warsaw writes: (in relation to support for Unicode regexes)

We need to be careful not to worsen performance for 8bit strings. I think we're already on the edge of acceptability w.r.t. P*** and hopefully we can /improve/ performance here.

I don't think that will be a problem, given that the Unicode engine would be a separate C implementation. A bit of 'if type(strg) == UnicodeType' in re.py isn't going to cost very much speed. (Speeding up PCRE -- that's another question. I'm often tempted to rewrite pcre_compile to generate an easier-to-analyse parse tree, instead of its current complicated-but-memory-parsimonious compiler, but I'm very reluctant to introduce a fork like that.) -- A.M. Kuchling http://starship.python.net/crew/amk/ The world does so well without me, that I am moved to wish that I could do equally well without the world. -- Robertson Davies, _The Diary of Samuel Marchbanks_

[Cc'ed to the String-SIG; sheesh, what's the point of having SIGs otherwise?] Fredrik Lundh writes:

any special pattern constructs that are in need of per- formance improvements? (compared to Perl, that is).

In the 1.5 source tree, I think one major slowdown is coming from the malloc'ed failure stack. This was introduced in order to prevent an expression like (x)* from filling the stack when applied to a string contained 50,000 'x' characters (hence 50,000 recursive function calls). I'd like to get rid of this stack because it's slow and requires much tedious patching of the upstream PCRE.

or maybe anyone has an extensive performance test suite for perlish regular expressions? (preferrably based on how real people use regular expressions, not only on things that are known to be slow if not optimized)

Friedl's book describes several optimizations which aren't implemented in PCRE. The problem is that PCRE never builds a parse tree, and parse trees are easy to analyse recursively. Instead, PCRE's functions actually look at the compiled byte codes (for example, look at find_firstchar or is_anchored in pypcre.c), but this makes analysis functions hard to write, and rearranging the code near-impossible. -- A.M. Kuchling http://starship.python.net/crew/amk/ I didn't say it was my fault. I said it was my responsibility. I know the difference. -- Rose Walker, in SANDMAN #60: "The Kindly Ones:4"

Tim Peters wrote:

...

The problem is that PCRE never builds a parse tree, and parse trees are easy to analyse recursively. Instead, PCRE's functions actually look at the compiled byte codes (for example, look at find_firstchar or is_anchored in pypcre.c), but this makes analysis functions hard to write, and rearranging the code near-impossible.

This is wonderfully & ironically Pythonic. That is, the Python compiler itself goes straight to byte code, and the optimization that's done works at the latter low level.

yeah, but by some reason, people (including GvR) expect a regular expression machinery to be more optimized than the language interpreter ;-) </F>

[Guido, on "a new Unicode regex engine by /F"]

It's from scratch, and I believe it's got Perl style, not POSIX style semantics -- per Tim Peters' recommendations. Do we need to open the discussion again?

No, but I get to whine just a little <wink>: I didn't recommend either approach. I asked many futile questions about HP's requirements, and sketched implications either way. If HP *has* a requirement wrt POSIX-vs-Perl, it would be good to find that out before it's too late. I personally prefer POSIX semantics -- but, as Andrew so eloquently said, worse is better here; all else being equal it's best to follow JPython's Perl-compatible re lead. last-time-i-ever-say-what-i-really-think<wink>-ly y'rs - tim

Mark Hammond wrote:

...

I think his proposal will go a long way towards your toolkit. I hope to hear soon from anybody who disagrees with Marc-Andre's proposal,

No disagreement as such, but a small hole:

...

From the proposal:

Internal Argument Parsing: -------------------------- ... 's': For Unicode objects: auto convert them to the <default encoding> and return a pointer to the object's <defencbuf> buffer.

-- Excellent - if someone passes a Unicode object, it can be auto-converted to a string. This will allow "open()" to accept Unicode strings.

Well almost... it depends on the current value of <default encoding>. If it's UTF8 and you only use normal ASCII characters the above is indeed true, but UTF8 can go far beyond ASCII and have up to 3 bytes per character (for UCS2, even more for UCS4). With <default encoding> set to other exotic encodings this is likely to fail though.

However, there doesnt appear to be a reverse. Eg, if my extension module interfaces to a library that uses Unicode natively, how can I get a Unicode object when the user passes a string? If I had to explicitely check for a string, then check for a Unicode on failure it would get messy pretty quickly... Is it not possible to have "U" also do a conversion?

"U" is meant to simplify checks for Unicode objects, much like "S". It returns a reference to the object. Auto-conversions are not possible due to this, because they would create new objects which don't get properly garbage collected later on. Another problem is that Unicode types differ between platforms (MS VCLIB uses 16-bit wchar_t, while GLIBC2 uses 32-bit wchar_t). Depending on the internal format of Unicode objects this could mean calling different conversion APIs. BTW, I'm still not too sure about the underlying internal format. The problem here is that Unicode started out as 2-byte fixed length representation (UCS2) but then shifted towards a 4-byte fixed length reprensetation known as UCS4. Since having 4 bytes per character is hard sell to customers, UTF16 was created to stuff the UCS4 code points (this is how character entities are called in Unicode) into 2 bytes... with a variable length encoding. Some platforms that started early into the Unicode business such as the MS ones use UCS2 as wchar_t, while more recent ones (e.g. the glibc2 on Linux) use UCS4 for wchar_t. I haven't yet checked in what ways the two are compatible (I would suspect the top bytes in UCS4 being 0 for UCS2 codes), but would like to hear whether it wouldn't be a better idea to use UTF16 as internal format. The latter works in 2 bytes for most characters and conversion to UCS2|4 should be fast. Still, conversion to UCS2 could fail. The downside of using UTF16: it is a variable length format, so iterations over it will be slower than for UCS4. Simply sticking to UCS2 is probably out of the question, since Unicode 3.0 requires UCS4 and we are targetting Unicode 3.0. -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 51 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/

Greg Stein wrote: [MAL:]

...

The downside of using UTF16: it is a variable length format, so iterations over it will be slower than for UCS4.

Bzzt. May as well go with UTF-8 as the internal format, much like Perl is doing (as I recall).

Ehm, pardon me for asking - what is the brief rationale for selecting UCS2/4, or whetever it ends up being, over UTF8? I couldn't find a discussion in the last months of the string SIG, was this decided upon and frozen long ago? I'm not trying to re-open a can of worms, just to understand. -- Jean-Claude

On Wed, 10 Nov 1999, Jean-Claude Wippler wrote:

Greg Stein wrote:

...

Bzzt. May as well go with UTF-8 as the internal format, much like Perl is doing (as I recall).

Ehm, pardon me for asking - what is the brief rationale for selecting UCS2/4, or whetever it ends up being, over UTF8?

I couldn't find a discussion in the last months of the string SIG, was this decided upon and frozen long ago?

Try sometime last year :-) ... something like July thru September as I recall. Things will be a lot faster if we have a fixed-size character. Variable length formats like UTF-8 are a lot harder to slice, search, etc. Also, (IMO) a big reason for this new type is for interaction with the underlying OS/platform. I don't know of any platforms right now that really use UTF-8 as their Unicode string representation (meaning we'd have to convert back/forth from our UTF-8 representation to talk to the OS). Cheers, -g -- Greg Stein, http://www.lyra.org/

Greg Stein wrote:

Have you ever noticed how Python modules, packages, tools, etc, never define an import hook?

hey, didn't MAL use one in one of his mx kits? ;-)

I say axe it and say "UTF-8" is the fixed, default encoding. If you want something else, then do that explicitly.

exactly. modes are evil. python is not perl. etc.

Are we digging a hole for ourselves? Maybe. But there are two other big platforms that have the same hole to dig out of *IF* it ever comes to that. I posit that it won't be necessary; that the people needing UCS-4 can do so entirely in Python.

last time I checked, there were no characters (even in the ISO standard) outside the 16-bit range. has that changed? </F>

Marc writes:

...

modes are evil. python is not perl. etc.

But a requirement by the customer... they want to be able to set the locale on a per thread basis. Not exactly my preference (I think all locale settings should be passed as parameters, not via globals).

Sure - that is what this customer wants, but we need to be clear about the "best thing" for Python generally versus what this particular client wants. For example, if we went with UTF-8 as the only default encoding, then HP may be forced to use a helper function to perform the conversion, rather than the built-in functions. This helper function can use TLS (in Python) to store the encoding. At least it is localized. I agree that having a default encoding that can be changed is a bad idea. It may make 3 line scripts that need to print something easier to work with, but at the cost of reliability in large systems. Kinda like the existing "locale" support, which is thread specific, and is well known to cause these sorts of problems. The end result is that in your app, you find _someone_ has changed the default encoding, and some code no longer works. So the solution is to change the default encoding back, so _your_ code works again. You just know that whoever it was that changed the default encoding in the first place is now going to break - but what else can you do? Having a fixed, default encoding may make life slightly more difficult when you want to work primarily in a different encoding, but at least your system is predictable and reliable. Mark.

...

...

Are we digging a hole for ourselves? Maybe. But there are two other big platforms that have the same hole to dig out of *IF* it ever comes to that. I posit that it won't be necessary; that the people needing UCS-4 can do so entirely in Python.

last time I checked, there were no characters (even in the ISO standard) outside the 16-bit range. has that changed?

No, but people are already thinking about it and there is a defined range in the >16-bit area for private encodings (F0000..FFFFD and 100000..10FFFD).

-- Marc-Andre Lemburg

______________________________________________________________________

Y2000: 51 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/

_______________________________________________ Python-Dev maillist - Python-Dev@python.org http://www.python.org/mailman/listinfo/python-dev

Tim Peters wrote:

[Mark Hammond]

...

Sure - that is what this customer wants, but we need to be clear about the "best thing" for Python generally versus what this particular client wants. ... Having a fixed, default encoding may make life slightly more difficult when you want to work primarily in a different encoding, but at least your system is predictable and reliable.

Well said, Mark! Me too. It's like HP is suffering from Windows envy <wink>.

See my other post on the subject... Note that if we make UTF-8 the standard encoding, nearly all special Latin-1 characters will produce UTF-8 errors on input and unreadable garbage on output. That will probably be unacceptable in Europe. To remedy this, one would *always* have to use u.encode('latin-1') to get readable output for Latin-1 strings repesented in Unicode. I'd rather see this happen the other way around: *always* explicitly state the encoding you want in case you rely on it, e.g. write file.write(u.encode('utf-8')) instead of file.write(u) # let's hope this goes out as UTF-8... Using the <default encoding> as site dependent setting is useful for convenience in those cases where the output format should be readable rather than parseable. -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 50 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/

Tim Peters wrote:

[MAL]

...

Note that if we make UTF-8 the standard encoding, nearly all special Latin-1 characters will produce UTF-8 errors on input and unreadable garbage on output. That will probably be unacceptable in Europe. To remedy this, one would *always* have to use u.encode('latin-1') to get readable output for Latin-1 strings repesented in Unicode.

I think it's time for the Europeans to pronounce on what's acceptable in Europe. To the limited extent that I can pretend I'm Eurpoean, I'm happy with Guido's rebind-stdin/stdout-in-PYTHONSTARTUP idea.

Agreed.

...

I'd rather see this happen the other way around: *always* explicitly state the encoding you want in case you rely on it, e.g. write

file.write(u.encode('utf-8'))

instead of

file.write(u) # let's hope this goes out as UTF-8...

By the same argument, those pesky Europeans who are relying on Latin-1 should write

file.write(u.encode('latin-1'))

instead of

file.write(u) # let's hope this goes out as Latin-1

Right.

...

Using the <default encoding> as site dependent setting is useful for convenience in those cases where the output format should be readable rather than parseable.

Well, "convenience" is always the argument advanced in favor of modes. Conflicts and nasty intermittent bugs are always the result. The latter will happen under Guido's idea too, as various careless modules rebind stdin & stdout to their own ideas of what "the proper" encoding should be. But at least the blame doesn't fall on the core language then <0.3 wink>.

Since there doesn't appear to be anything (either or good or bad) you can do (or avoid) by using Guido's scheme instead of magical core thread state, there's no *need* for the latter. That is, it can be done with a user-level API without involving the core.

Dito :-) I have nothing against telling people to take care about the problem in user space (meaning: not done by the core interpreter) and I'm pretty sure that HP will agree on this too, provided we give them the proper user space tools like file wrappers et al. -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 49 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/

Mark Hammond wrote:

...

Having a fixed, default encoding may make life slightly more difficult when you want to work primarily in a different encoding, but at least your system is predictable and reliable.

I think the discussion on this is getting a little too hot.

Really - I see it as moving to a rational consensus that doesnt support the proposal in this regard. I see no heat in it at all. Im sorry if you saw my post or any of the followups as "emotional", but I certainly not getting passionate about this. I dont see any of this as affecting me personally. I believe that I can replace my Unicode implementation with this either way we go. Just because a we are trying to get it right doesnt mean we are getting heated.

The point is simply that the option of changing the per-thread default encoding is there. You are not required to use it and if you do you are on your own when something breaks.

Hrm - Im having serious trouble following your logic here. If make _any_ assumptions about a default encoding, I am in danger of breaking. I may not choose to change the default, but as soon as _anyone_ does, unrelated code may break. I agree that I will be "on my own", but I wont necessarily have been the one that changed it :-( The only answer I can see is, as you suggest, to ignore the fact that there is _any_ default. Always specify the encoding. But obviously this is not good enough for HP:

Think of it as a HP specific feature... perhaps I should wrap the code in #ifdefs and leave it undocumented.

That would work - just ensure that no standard Python has those #ifdefs turned on :-) I would be sorely dissapointed if the fact that HP are throwing money for this means they get every whim implemented in the core language. Imagine the outcry if it were instead MS' money, and you were attempting to put an MS spin on all this. Are you writing a module for HP, or writing a module for Python that HP are assisting by providing some funding? Clear difference. IMO, it must also be seen that there is a clear difference. Maybe Im missing something. Can you explain why it is good enough everyone else to be required to assume there is no default encoding, but HP get their thread specific global? Are their requirements greater than anyone elses? Is everyone else not as important? What would you, as a consultant, recommend to people who arent HP, but have a similar requirement? It would seem obvious to me that HPs requirement can be met in "pure Python", thereby keeping this out of the core all together... Mark.

[Mark Hammond]

... Are you writing a module for HP, or writing a module for Python that HP are assisting by providing some funding? Clear difference. IMO, it must also be seen that there is a clear difference.

I can resolve this easily, but only with input from Guido. Guido, did HP's check clear yet? If so, we can ignore them <wink>.

On Fri, 12 Nov 1999, M.-A. Lemburg wrote:

<IMO> Since this is the first time a Python Consortium member is pushing development, I think we can learn a lot here. For one, it should be clear that money doesn't buy everything, OTOH, we cannot put the whole thing at risk just because of some minor disagreement that cannot be solved between the parties. The standard solution for the latter should be a customized Python interpreter. </IMO>

hehe... funny you mention this. Go read the Consortium docs. Last time that I read them, there are no "parties" to reach consensus. *Every* technical decision regarding the Python language falls to the Technical Director (Guido, of course). I looked. I found nothing that can override the T.D.'s decisions and no way to force a particular decision. Guido is still the Benevolent Dictator :-) Cheers, -g p.s. yes, there is always the caveat that "sure, Guido has final say" but "Al can fire him at will for being too stubborn" :-) ... but hey, Guido's title does have the word Benevolent in it, so things are cool... -- Greg Stein, http://www.lyra.org/

On Fri, 12 Nov 1999, M.-A. Lemburg wrote:

... Sure, but have you considered the option of a member simply bailing out ? HP could always stop funding Unicode integration. That wouldn't help us either...

I'm not that dumb... come on. That was my whole point about "Benevolent" below... Guido is a fair and reasonable Dictator... he wouldn't let that happen.

...

...

p.s. yes, there is always the caveat that "sure, Guido has final say" but "Al can fire him at will for being too stubborn" :-) ... but hey, Guido's title does have the word Benevolent in it, so things are cool...

Cheers, -g -- Greg Stein, http://www.lyra.org/

On Fri, 12 Nov 1999, M.-A. Lemburg wrote:

Sure there are some very subtile dangers in setting the default to anything other than the default ;-) For some this risk may be worthwhile taking, for others not. In fact, in large projects I would never take such a risk... I'm sure we can get this message across to them.

It's a lot easier to just never provide the rope (per-thread default encodings) in the first place. If the feature exists, then it will be used. Period. Try to get the message across until you're blue in the face, but it would be used. Anyhow... discussion is pretty moot until somebody can state that it is/isn't a "real requirement" and/or until The Guido takes a position. Cheers, -g -- Greg Stein, http://www.lyra.org/

On Thu, 11 Nov 1999, Greg Stein wrote:

Ha! I was getting ready to say exactly the same thing. Are building Python for a particular customer, or are we building it to Do The Right Thing?

I've been getting increasingly annoyed at "well, HP says this" or "HP wants that." I'm ecstatic that they are a Consortium member and are helping to fund the development of Python. However, if that means we are selling Python's soul to corporate wishes rather than programming and design ideals... well, it reduces my enthusiasm :-)

What about just explaining the rationale for the default-less point of view to whoever is in charge of this at HP and see why they came up with their rationale in the first place? They might have a good reason, or they might be willing to change said requirement. --david

[/F]

last time I checked, there were no characters (even in the ISO standard) outside the 16-bit range. has that changed?

[MAL]

No, but people are already thinking about it and there is a defined range in the >16-bit area for private encodings (F0000..FFFFD and 100000..10FFFD).

Over the decades I've developed a rule of thumb that has never wound up stuck in my ass <wink>: If I engineer code that I expect to be in use for N years, I make damn sure that every internal limit is at least 10x larger than the largest I can conceive of a user making reasonable use of at the end of those N years. The invariable result is that the N years pass, and fewer than half of the users have bumped into the limit <0.5 wink>. At the risk of offending everyone, I'll suggest that, qualitatively speaking, Unicode is as Eurocentric as ASCII is Anglocentric. We've just replaced "256 characters?! We'll *never* run out of those!" with 64K. But when Asian languages consume them 7K at a pop, 64K isn't even in my 10x comfort range for some individual languages. In just a few months, Unicode 3 will already have used up > 56K of the 64K slots. As I understand it, UTF-16 "only" adds 1M new code points. That's in my 10x zone, for about a decade. predicting-we'll-live-to-regret-it-either-way-ly y'rs - tim

[MAL]

If HP approves, I'd propose to use UTF-16 as if it were UCS-2 and signal failure of this assertion at Unicode object construction time via an exception. That way we are within the standard, can use reasonably fast code for Unicode manipulation and add those extra 1M character at a later stage.

I think this is reasonable. Using UTF-8 internally is also reasonable, and if it's being rejected on the grounds of supposed slowness, that deserves a closer look (it's an ingenious encoding scheme that works correctly with a surprising number of existing 8-bit string routines as-is). Indexing UTF-8 strings is greatly speeded by adding a simple finger (i.e., store along with the string an index+offset pair identifying the most recent position indexed to -- since string indexing is overwhelmingly sequential, this makes most indexing constant-time; and UTF-8 can be scanned either forward or backward from a random internal point because "the first byte" of each encoding is recognizable as such). I expect either would work well. It's at least curious that Perl and Tcl both went with UTF-8 -- does anyone think they know *why*? I don't. The people here saying UCS-2 is the obviously better choice are all from the Microsoft camp <wink>. It's not obvious to me, but then neither do I claim that UTF-8 is obviously better.

Besides, the Unicode object will have a buffer containing the <default encoding> representation of the object, which, if all goes well, will always hold the UTF-8 value.

<rant> over my dead body, that one... (fwiw, over the last 20 years, I've implemented about a dozen image processing libraries, supporting loads of pixel layouts and file formats. one important lesson from that is to stick to a single internal representation, and let the application programmers build their own layers if they need to speed things up -- yes, they're actually happier that way. and text strings are not that different from pixel buffers or sound streams or scientific data sets, after all...) (and sticks and modes will break your bones, but you know that...)

RE engines etc. can then directly work with this buffer.

sidebar: the RE engine that's being developed for this project can handle 8-bit, 16-bit, and (optionally) 32-bit text buffers. a single compiled expression can be used with any character size, and performance is about the same for all sizes (at least on any decent cpu).

...

I expect either would work well. It's at least curious that Perl and Tcl both went with UTF-8 -- does anyone think they know *why*? I don't. The people here saying UCS-2 is the obviously better choice are all from the Microsoft camp <wink>.

(hey, I'm not a microsofter. but I've been writing "i/o libraries" for various "object types" all my life, so I do have strong preferences on what works, and what doesn't... I use Python for good reasons, you know ;-) </rant> thanks. I feel better now. </F>

...

...

Besides, the Unicode object will have a buffer containing the <default encoding> representation of the object, which, if all goes well, will always hold the UTF-8 value.

<rant>

over my dead body, that one...

Such a buffer is needed to implement "s" and "s#" argument parsing. It's a simple requirement to support those two parsing markers -- there's not much to argue about, really...

why? I don't understand why "s" and "s#" has to deal with encoding issues at all...

unless, of course, you want to give up Unicode object support for all APIs using these parsers.

hmm. maybe that's exactly what I want... </F>

On Fri, 12 Nov 1999, M.-A. Lemburg wrote:

Fredrik Lundh wrote: ...

...

why? I don't understand why "s" and "s#" has to deal with encoding issues at all...

...

unless, of course, you want to give up Unicode object support for all APIs using these parsers.

hmm. maybe that's exactly what I want...

If we don't add that support, lot's of existing APIs won't accept Unicode object instead of strings. While it could be argued that automatic conversion to UTF-8 is not transparent enough for the user, the other solution of using str(u) everywhere would probably make writing Unicode-aware code a rather clumsy task and introduce other pitfalls, since str(obj) calls PyObject_Str() which also works on integers, floats, etc.

No no no... "s" and "s#" are NOT SUPPOSED TO return a UTF-8 encoding. They are supposed to return the raw bytes. If a caller wants 8-bit characters, then that caller will use "t#". If you want to argue for that separate, encoded buffer, then argue for it for support for the "t#" format. But do NOT say that it is needed for "s#" which simply means "give me some bytes." -g -- Greg Stein, http://www.lyra.org/

I think I have a reasonable grasp of the issues here, even though I still haven't read about 100 msgs in this thread. Note that t# and the charbuffer addition to the buffer API were added by Greg Stein with my support; I'll attempt to reconstruct our thinking at the time... [MAL]

Let me summarize a bit on the general ideas behind "s", "s#" and the extra buffer:

I think you left out t#.

First, we have a general design question here: should old code become Unicode compatible or not. As I recall the original idea about Unicode integration was to follow Perl's idea to have scripts become Unicode aware by simply adding a 'use utf8;'.

I've never heard of this idea before -- or am I taking it too literal? It smells of a mode to me :-) I'd rather live in a world where Unicode just works as long as you use u'...' literals or whatever convention we decide.

If this is still the case, then we'll have to come with a resonable approach for integrating classical string based APIs with the new type.

Since UTF-8 is a standard (some would probably prefer UTF-7,5 e.g. the Latin-1 folks) which has some very nice features (see http://czyborra.com/utf/ ) and which is a true extension of ASCII, this encoding seems best fit for the purpose.

Yes, especially if we fix the default encoding as UTF-8. (I'm expecting feedback from HP on this next week, hopefully when I see the details, it'll be clear that don't need a per-thread default encoding to solve their problems; that's quite a likely outcome. If not, we have a real-world argument for allowing a variable default encoding, without carnage.)

However, one should not forget that UTF-8 is in fact a variable length encoding of Unicode characters, that is up to 3 bytes form a *single* character. This is obviously not compatible with definitions that explicitly state data to be using a 8-bit single character encoding, e.g. indexing in UTF-8 doesn't work like it does in Latin-1 text.

Sure, but where in current Python are there such requirements?

So if we are to do the integration, we'll have to choose argument parser markers that allow for multi byte characters. "t#" does not fall into this category, "s#" certainly does, "s" is argueable.

I disagree. I grepped through the source for s# and t#. Here's a bit of background. Before t# was introduced, s# was being used for two distinct purposes: (1) to get an 8-bit text string plus its length, in situations where the length was needed; (2) to get binary data (e.g. GIF data read from a file in "rb" mode). Greg pointed out that if we ever introduced some form of Unicode support, these two had to be disambiguated. We found that the majority of uses was for (2)! Therefore we decided to change the definition of s# to mean only (2), and introduced t# to mean (1). Also, we introduced getcharbuffer corresponding to t#, while getreadbuffer was meant for s#. Note that the definition of the 's' format was left alone -- as before, it means you need an 8-bit text string not containing null bytes. Our expectation was that a Unicode string passed to an s# situation would give a pointer to the internal format plus a byte count (not a character count!) while t# would get a pointer to some kind of 8-bit translation/encoding plus a byte count, with the explicit requirement that the 8-bit translation would have the same lifetime as the original unicode object. We decided to leave it up to the next generation (i.e., Marc-Andre :-) to decide what kind of translation to use and what to do when there is no reasonable translation. Any of the following choices is acceptable (from the point of view of not breaking the intended t# semantics; we can now start deciding which we like best): - utf-8 - latin-1 - ascii - shift-jis - lower byte of unicode ordinal - some user- or os-specified multibyte encoding As far as t# is concerned, for encodings that don't encode all of Unicode, untranslatable characters could be dealt with in any number of ways (raise an exception, ignore, replace with '?', make best effort, etc.). Given the current context, it should probably be the same as the default encoding -- i.e., utf-8. If we end up making the default user-settable, we'll have to decide what to do with untranslatable characters -- but that will probably be decided by the user too (it would be a property of a specific translation specification). In any case, I feel that t# could receive a multi-byte encoding, s# should receive raw binary data, and they should correspond to getcharbuffer and getreadbuffer, respectively. (Aside: the symmetry between 's' and 's#' is now lost; 's' matches 't#', there's no match for 's#'.)

Also note that we have to watch out for embedded NULL bytes. UTF-16 has NULL bytes for every character from the Latin-1 domain. If "s" were to give back a pointer to the internal buffer which is encoded in UTF-16, you would loose data. UTF-8 doesn't have this problem, since only NULL bytes map to (single) NULL bytes.

This is a red herring given my explanation above.

Now Greg would chime in with the buffer interface and argue that it should make the underlying internal format accessible. This is a bad idea, IMHO, since you shouldn't really have to know what the internal data format is.

This is for C code. Quite likely it *does* know what the internal data format is!

Defining "s#" to return UTF-8 data does not only make "s" and "s#" return the same data format (which should always be the case, IMO),

That was before t# was introduced. No more, alas. If you replace s# with t#, I agree with you completely.

but also hides the internal format from the user and gives him a reliable cross-platform data representation of Unicode data (note that UTF-8 doesn't have the byte order problems of UTF-16).

If you are still with, let's look at what "s" and "s#"

(and t#, which is more relevant here)

do: they return pointers into data areas which have to be kept alive until the corresponding object dies.

The only way to support this feature is by allocating a buffer for just this purpose (on the fly and only if needed to prevent excessive memory load). The other options of adding new magic parser markers or switching to more generic one all have one downside: you need to change existing code which is in conflict with the idea we started out with.

Agreed. I think this was our thinking when Greg & I introduced t#. My own preference would be to allocate a whole string object, not just a buffer; this could then also be used for the .encode() method using the default encoding.

So, again, the question is: do we want this magical intergration or not ? Note that this is a design question, not one of memory consumption...

Yes, I want it. Note that this doesn't guarantee that all old extensions will work flawlessly when passed Unicode objects; but I think that it covers most cases where you could have a reasonable expectation that it works. (Hm, unfortunately many reasonable expectations seem to involve the current user's preferred encoding. :-( )

--

Ok, the above covered Unicode -> String conversion. Mark mentioned that he wanted the other way around to also work in the same fashion, ie. automatic String -> Unicode conversion.

This could also be done in the same way by interpreting the string as UTF-8 encoded Unicode... but we have the same problem: where to put the data without generating new intermediate objects. Since only newly written code will use this feature there is a way to do this though:

PyArg_ParseTuple(args,"s#",&utf8,&len);

No! That is supposed to give the native representation of the string object. I agree that Mark's problem requires a solution too, but it doesn't have to use existing formatting characters, since there's no backwards compatibility issue.

If your C API understands UTF-8 there's nothing more to do, if not, take Greg's option 3 approach:

PyArg_ParseTuple(args,"O",&obj); unicode = PyUnicode_FromObject(obj); ... Py_DECREF(unicode);

Here PyUnicode_FromObject() will return a new reference if obj is an Unicode object or create a new Unicode object by interpreting str(obj) as UTF-8 encoded string.

This might work. --Guido van Rossum (home page: http://www.python.org/~guido/)

On purpose -- according to my thinking. I see "t#" as an interface to bf_getcharbuf which I understand as 8-bit character buffer... UTF-8 is a multi byte encoding. It still is character data, but not necessarily 8 bits in length (up to 24 bits are used).

Anyway, I'm not really interested in having an argument about this. If you say, "t#" fits the purpose, then that's fine with me. Still, we should clearly define that "t#" returns text data and "s#" binary data. Encoding, bit length, etc. should explicitly remain left undefined.

Thanks for not picking an argument. Multibyte encodings typically have ASCII as a subset (in such a way that an ASCII string is represented as itself in bytes). This is the characteristic that's needed in my view.

...

...

First, we have a general design question here: should old code become Unicode compatible or not. As I recall the original idea about Unicode integration was to follow Perl's idea to have scripts become Unicode aware by simply adding a 'use utf8;'.

I've never heard of this idea before -- or am I taking it too literal? It smells of a mode to me :-) I'd rather live in a world where Unicode just works as long as you use u'...' literals or whatever convention we decide.

...

If this is still the case, then we'll have to come with a resonable approach for integrating classical string based APIs with the new type.

Since UTF-8 is a standard (some would probably prefer UTF-7,5 e.g. the Latin-1 folks) which has some very nice features (see http://czyborra.com/utf/ ) and which is a true extension of ASCII, this encoding seems best fit for the purpose.

Yes, especially if we fix the default encoding as UTF-8. (I'm expecting feedback from HP on this next week, hopefully when I see the details, it'll be clear that don't need a per-thread default encoding to solve their problems; that's quite a likely outcome. If not, we have a real-world argument for allowing a variable default encoding, without carnage.)

Fair enough :-)

...

...

However, one should not forget that UTF-8 is in fact a variable length encoding of Unicode characters, that is up to 3 bytes form a *single* character. This is obviously not compatible with definitions that explicitly state data to be using a 8-bit single character encoding, e.g. indexing in UTF-8 doesn't work like it does in Latin-1 text.

Sure, but where in current Python are there such requirements?

It was my understanding that "t#" refers to single byte character data. That's where the above arguments were aiming at...

t# refers to byte-encoded data. Multibyte encodings are explicitly designed to be passed cleanly through processing steps that handle single-byte character data, as long as they are 8-bit clean and don't do too much processing.

...

...

So if we are to do the integration, we'll have to choose argument parser markers that allow for multi byte characters. "t#" does not fall into this category, "s#" certainly does, "s" is argueable.

I disagree. I grepped through the source for s# and t#. Here's a bit of background. Before t# was introduced, s# was being used for two distinct purposes: (1) to get an 8-bit text string plus its length, in situations where the length was needed; (2) to get binary data (e.g. GIF data read from a file in "rb" mode). Greg pointed out that if we ever introduced some form of Unicode support, these two had to be disambiguated. We found that the majority of uses was for (2)! Therefore we decided to change the definition of s# to mean only (2), and introduced t# to mean (1). Also, we introduced getcharbuffer corresponding to t#, while getreadbuffer was meant for s#.

I know its too late now, but I can't really follow the arguments here: in what ways are (1) and (2) different from the implementations point of view ? If "t#" is to return UTF-8 then <length of the buffer> will not equal <text length>, so both parser markers return essentially the same information. The only difference would be on the semantic side: (1) means: give me text data, while (2) does not specify the data type.

Perhaps I'm missing something...

The idea is that (1)/s# disallows any translation of the data, while (2)/t# requires translation of the data to an ASCII superset (possibly multibyte, such as UTF-8 or shift-JIS). (2)/t# assumes that the data contains text and that if the text consists of only ASCII characters they are represented as themselves. (1)/s# makes no such assumption. In terms of implementation, Unicode objects should translate themselves to the default encoding for t# (if possible), but they should make the native representation available for s#. For example, take an encryption engine. While it is defined in terms of byte streams, there's no requirement that the bytes represent characters -- they could be the bytes of a GIF file, an MP3 file, or a gzipped tar file. If we pass Unicode to an encryption engine, we want Unicode to come out at the other end, not UTF-8. (If we had wanted to encrypt UTF-8, we should have fed it UTF-8.)

...

Note that the definition of the 's' format was left alone -- as before, it means you need an 8-bit text string not containing null bytes.

This definition should then be changed to "text string without null bytes" dropping the 8-bit reference.

Aha, I think there's a confusion about what "8-bit" means. For me, a multibyte encoding like UTF-8 is still 8-bit. Am I alone in this? (As far as I know, C uses char* to represent multibyte characters.) Maybe we should disambiguate it more explicitly?

...

Our expectation was that a Unicode string passed to an s# situation would give a pointer to the internal format plus a byte count (not a character count!) while t# would get a pointer to some kind of 8-bit translation/encoding plus a byte count, with the explicit requirement that the 8-bit translation would have the same lifetime as the original unicode object. We decided to leave it up to the next generation (i.e., Marc-Andre :-) to decide what kind of translation to use and what to do when there is no reasonable translation.

Hmm, I would strongly object to making "s#" return the internal format. file.write() would then default to writing UTF-16 data instead of UTF-8 data. This could result in strange errors due to the UTF-16 format being endian dependent.

But this was the whole design. file.write() needs to be changed to use s# when the file is open in binary mode and t# when the file is open in text mode.

It would also break the symmetry between file.write(u) and unicode(file.read()), since the default encoding is not used as internal format for other reasons (see proposal).

If the file is encoded using UTF-16 or UCS-2, you should open it in binary mode and use unicode(file.read(), 'utf-16'). (Or perhaps the app should read the first 2 bytes and check for a BOM and then decide to choose bewteen 'utf-16-be' and 'utf-16-le'.)

...

Any of the following choices is acceptable (from the point of view of not breaking the intended t# semantics; we can now start deciding which we like best):

I think we have already agreed on using UTF-8 for the default encoding. It has quite a few advantages. See

http://czyborra.com/utf/

for a good overview of the pros and cons.

Of course. I was just presenting the list as an argument that if we changed our mind about the default encoding, t# should follow the default encoding (and not pick an encoding by other means).

...

- utf-8 - latin-1 - ascii - shift-jis - lower byte of unicode ordinal - some user- or os-specified multibyte encoding

As far as t# is concerned, for encodings that don't encode all of Unicode, untranslatable characters could be dealt with in any number of ways (raise an exception, ignore, replace with '?', make best effort, etc.).

The usual Python way would be: raise an exception. This is what the proposal defines for Codecs in case an encoding/decoding mapping is not possible, BTW. (UTF-8 will always succeed on output.)

Did you read Andy Robinson's case study? He suggested that for certain encodings there may be other things you can do that are more user-friendly than raising an exception, depending on the application. I am proposing to leave this a detail of each specific translation. There may even be translations that do the same thing except they have a different behavior for untranslatable cases -- e.g. a strict version that raises an exception and a non-strict version that replaces bad characters with '?'. I think this is one of the powers of having an extensible set of encodings.

...

Given the current context, it should probably be the same as the default encoding -- i.e., utf-8. If we end up making the default user-settable, we'll have to decide what to do with untranslatable characters -- but that will probably be decided by the user too (it would be a property of a specific translation specification).

In any case, I feel that t# could receive a multi-byte encoding, s# should receive raw binary data, and they should correspond to getcharbuffer and getreadbuffer, respectively.

Why would you want to have "s#" return the raw binary data for Unicode objects ?

Because file.write() for a binary file, and other similar things (e.g. the encryption engine example I mentioned above) must have *some* way to get at the raw bits.

Note that it is not mentioned anywhere that "s#" and "t#" do have to necessarily return different things (binary being a superset of text). I'd opt for "s#" and "t#" both returning UTF-8 data. This can be implemented by delegating the buffer slots to the <defencstr> object (see below).

This would defeat the whole purpose of introducing t#. We might as well drop t# then altogether if we adopt this.

...

...

Now Greg would chime in with the buffer interface and argue that it should make the underlying internal format accessible. This is a bad idea, IMHO, since you shouldn't really have to know what the internal data format is.

This is for C code. Quite likely it *does* know what the internal data format is!

C code can use the PyUnicode_* APIs to access the data. I don't think that argument parsing is powerful enough to provide the C code with enough information about the data contents, e.g. it can only state the encoding length, not the string length.

Typically, all the C code does is pass multibyte encoded strings on to other library routines that know what to do to them, or simply give them back unchanged at a later time. It is essential to know the number of bytes, for memory allocation purposes. The number of characters is totally immaterial (and multibyte-handling code knows how to calculate the number of characters anyway).

...

...

Defining "s#" to return UTF-8 data does not only make "s" and "s#" return the same data format (which should always be the case, IMO),

That was before t# was introduced. No more, alas. If you replace s# with t#, I agree with you completely.

Done :-)

...

...

but also hides the internal format from the user and gives him a reliable cross-platform data representation of Unicode data (note that UTF-8 doesn't have the byte order problems of UTF-16).

If you are still with, let's look at what "s" and "s#"

(and t#, which is more relevant here)

...

do: they return pointers into data areas which have to be kept alive until the corresponding object dies.

The only way to support this feature is by allocating a buffer for just this purpose (on the fly and only if needed to prevent excessive memory load). The other options of adding new magic parser markers or switching to more generic one all have one downside: you need to change existing code which is in conflict with the idea we started out with.

Agreed. I think this was our thinking when Greg & I introduced t#. My own preference would be to allocate a whole string object, not just a buffer; this could then also be used for the .encode() method using the default encoding.

Good point. I'll change <defencbuf> to <defencstr>, a Python string object created on request.

...

...

So, again, the question is: do we want this magical intergration or not ? Note that this is a design question, not one of memory consumption...

Yes, I want it.

Note that this doesn't guarantee that all old extensions will work flawlessly when passed Unicode objects; but I think that it covers most cases where you could have a reasonable expectation that it works.

(Hm, unfortunately many reasonable expectations seem to involve the current user's preferred encoding. :-( )

-- Marc-Andre Lemburg

--Guido van Rossum (home page: http://www.python.org/~guido/)

On Mon, 15 Nov 1999, M.-A. Lemburg wrote:

Guido van Rossum wrote: ...

...

t# refers to byte-encoded data. Multibyte encodings are explicitly designed to be passed cleanly through processing steps that handle single-byte character data, as long as they are 8-bit clean and don't do too much processing.

Ah, ok. I interpreted 8-bit to mean: 8 bits in length, not "8-bit clean" as you obviously did.

Hrm. That might be dangerous. Many of the functions that use "t#" assume that each character is 8-bits long. i.e. the returned length == the number of characters. I'm not sure what the implications would be if you interpret the semantics of "t#" as multi-byte characters.

...

...

For example, take an encryption engine. While it is defined in terms of byte streams, there's no requirement that the bytes represent characters -- they could be the bytes of a GIF file, an MP3 file, or a gzipped tar file. If we pass Unicode to an encryption engine, we want Unicode to come out at the other end, not UTF-8. (If we had wanted to encrypt UTF-8, we should have fed it UTF-8.)

Heck. I just want to quickly throw the data onto my disk. I'll write a BOM, following by the raw data. Done. It's even portable.

...

...

Aha, I think there's a confusion about what "8-bit" means. For me, a multibyte encoding like UTF-8 is still 8-bit. Am I alone in this?

Maybe. I don't see multi-byte characters as 8-bit (in the sense of the "t" format).

...

(As far as I know, C uses char* to represent multibyte characters.) Maybe we should disambiguate it more explicitly?

We can disambiguate with a new format character, or we can clarify the semantics of "t" to mean single- *or* multi- byte characters. Again, I think there may be trouble if the semantics of "t" are defined to allow multibyte characters.

There should be some definition for the two markers and the ideas behind them in the API guide, I guess.

Certainly. [ man, I'm bad... I've got doc updates there and for the buffer stuff :-( ]

...

...

Hmm, I would strongly object to making "s#" return the internal format. file.write() would then default to writing UTF-16 data instead of UTF-8 data. This could result in strange errors due to the UTF-16 format being endian dependent.

But this was the whole design. file.write() needs to be changed to use s# when the file is open in binary mode and t# when the file is open in text mode.

Interesting idea, but that presumes that "t" will be defined for the Unicode object (i.e. it implements the getcharbuffer type slot). Because of the multi-byte problem, I don't think it will. [ not to mention, that I don't think the Unicode object should implicitly do a UTF-8 conversion and hold a ref to the resulting string ]

... I still don't feel very comfortable about the fact that all existing APIs using "s#" will suddenly receive UTF-16 data if being passed Unicode objects: this probably won't get us the "magical" Unicode integration we invision, since "t#" usage is not very wide spread and character handling code will probably not work well with UTF-16 encoded strings.

I'm not sure that we should definitely go for "magical." Perl has magic in it, and that is one of its worst faults. Go for clean and predictable, and leave as much logic to the Python level as possible. The interpreter should provide a minimum of functionality, rather than second-guessing and trying to be neat and sneaky with its operation.

...

...

Because file.write() for a binary file, and other similar things (e.g. the encryption engine example I mentioned above) must have *some* way to get at the raw bits.

What for ?

How about: "because I'm the application developer, and I say that I want the raw bytes in the file."

Any lossless encoding should do the trick... UTF-8 is just as good as UTF-16 for binary files; plus it's more compact for ASCII data. I don't really see a need to get explicitly at the internal data representation because both encodings are in fact "internal" w/r to Unicode objects.

The only argument I can come up with is that using UTF-16 for binary files could (possibly) eliminate the UTF-8 conversion step which is otherwise always needed.

The argument that I come up with is "don't tell me how to design my storage format, and don't make Python force me into one." If I want to write Unicode text to a file, the most natural thing to do is: open('file', 'w').write(u) If you do a conversion on me, then I'm not writing Unicode. I've got to go and do some nasty conversion which just monkeys up my program. If I have a Unicode object, but I *want* to write UTF-8 to the file, then the cleanest thing is: open('file', 'w').write(encode(u, 'utf-8')) This is clear that I've got a Unicode object input, but I'm writing UTF-8. I have a second argument, too: See my first argument. :-) Really... this is kind of what Fredrik was trying to say: don't get in the way of the application programmer. Give them tools, but avoid policy and gimmicks and other "magic". Cheers, -g -- Greg Stein, http://www.lyra.org/

FYI, the next version of the proposal now says "s#" gives you UTF-16 and "t#" returns UTF-8. File objects opened in text mode will use "t#" and binary ones use "s#".

Good.

I'll just use explicit u.encode('utf-8') calls if I want to write UTF-8 to binary files -- perhaps everyone else should too ;-)

You could write UTF-8 to files opened in text mode too; at least most actual systems will leave the UTF-8 escapes alone and just to LF -> CRLF translation, which should be fine. --Guido van Rossum (home page: http://www.python.org/~guido/)

On Wed, 17 Nov 1999, Tim Peters wrote:

[MAL]

...

FYI, the next version of the proposal ... File objects opened in text mode will use "t#" and binary ones use "s#".

Am I the only one who sees magical distinctions between text and binary mode as a Really Bad Idea? I wouldn't have guessed the Unix natives here would quietly acquiesce to importing a bit of Windows madness <wink>.

It's a seductive idea... yes, it feels wrong, but then... it seems kind of right, too... :-) Yes. It is a mode. Is it bad? Not sure. You've already told the system that you want to treat the file differently. Much like you're treating it differently when you specify 'r' vs. 'w'. The real annoying thing would be to assume that opening a file as 'r' means that I *meant* text mode and to start using "t#". In actuality, I typically open files that way since I do most of my coding on Linux. If I now have to pay attention to things and open it as 'rb', then I'll be pissed. And the change in behavior and bugs that interpreting 'r' as text would introduce? Ack! Cheers, -g -- Greg Stein, http://www.lyra.org/

On Wed, 17 Nov 1999, Tim Peters wrote:

... 'r' is already intepreted as text mode, but so far, on Unix-like systems, there's been no difference between text and binary modes. Introducing a distinction will certainly cause problems. I don't know what the compensating advantages are thought to be.

Wow. "compensating advantages" ... Excellent "power phrase" there. hehe... -g -- Greg Stein, http://www.lyra.org/

On Thu, 18 Nov 1999, M.-A. Lemburg wrote:

Tim Peters wrote:

...

[MAL]

...

File objects opened in text mode will use "t#" and binary ones use "s#".

[Greg Stein]

...

... The real annoying thing would be to assume that opening a file as 'r' means that I *meant* text mode and to start using "t#".

Isn't that exactly what MAL said would happen? Note that a "t" flag for "text mode" is an MS extension -- C doesn't define "t", and Python doesn't either; a lone "r" has always meant text mode.

Em, I think you've got something wrong here: "t#" refers to the parsing marker used for writing data to files opened in text mode.

Nope. We've got it right :-) Tim and I used 'r' and "t" to refer to file-open modes. I used "t#" to refer to the parse marker.

... I guess you won't notice any difference: strings define both interfaces ("s#" and "t#") to mean the same thing. Only other buffer compatible types may now fail to write to text files -- which is not so bad, because it forces the programmer to rethink what he really intended when opening the file in text mode.

It *is* bad if it breaks my existing programs in subtle ways that are a bitch to track down.

Besides, if you are writing portable scripts you should pay close attention to "r" vs. "rb" anyway.

I'm not writing portable scripts. I mentioned that once before. I don't want a difference between 'r' and 'rb' on my Linux box. It was never there before, I'm lazy, and I don't want to see it added :-). Honestly, I don't know offhand of any Python types that repond to "s#" and "t#" in different ways, such that changing file.write would end up writing something different (and thereby breaking existing code). I just don't like introduce text/binary to *nix platforms where it didn't exist before. Cheers, -g -- Greg Stein, http://www.lyra.org/

Greg Stein wrote:

On Thu, 18 Nov 1999, M.-A. Lemburg wrote:

...

Tim Peters wrote:

...

[MAL]

...

File objects opened in text mode will use "t#" and binary ones use "s#".

[Greg Stein]

...

... The real annoying thing would be to assume that opening a file as 'r' means that I *meant* text mode and to start using "t#".

Isn't that exactly what MAL said would happen? Note that a "t" flag for "text mode" is an MS extension -- C doesn't define "t", and Python doesn't either; a lone "r" has always meant text mode.

Em, I think you've got something wrong here: "t#" refers to the parsing marker used for writing data to files opened in text mode.

Nope. We've got it right :-)

Tim and I used 'r' and "t" to refer to file-open modes. I used "t#" to refer to the parse marker.

Ah, ok. But "t" as file opener is non-portable anyways, so I'll skip it here :-)

...

... I guess you won't notice any difference: strings define both interfaces ("s#" and "t#") to mean the same thing. Only other buffer compatible types may now fail to write to text files -- which is not so bad, because it forces the programmer to rethink what he really intended when opening the file in text mode.

It *is* bad if it breaks my existing programs in subtle ways that are a bitch to track down.

...

Besides, if you are writing portable scripts you should pay close attention to "r" vs. "rb" anyway.

I'm not writing portable scripts. I mentioned that once before. I don't want a difference between 'r' and 'rb' on my Linux box. It was never there before, I'm lazy, and I don't want to see it added :-).

Honestly, I don't know offhand of any Python types that repond to "s#" and "t#" in different ways, such that changing file.write would end up writing something different (and thereby breaking existing code).

I just don't like introduce text/binary to *nix platforms where it didn't exist before.

Please remember that up until now you were probably only using strings to write to files. Python strings don't differentiate between "t#" and "s#" so you wont see any change in function or find subtle errors being introduced. If you are already using the buffer feature for e.g. array which also implement "s#" but don't support "t#" for obvious reasons you'll run into trouble, but then: arrays are binary data, so changing from text mode to binary mode is well worth the effort even if you just consider it a nuisance. Since the buffer interface and its consequences haven't published yet, there are probably very few users out there who would actually run into any problems. And even if they do, its a good chance to catch subtle bugs which would only have shown up when trying to port to another platform. I'll leave the rest for Guido to answer, since it was his idea ;-) -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 43 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/

Mark Hammond wrote:

[MAL]

...

If you are already using the buffer feature for e.g. array which also implement "s#" but don't support "t#" for obvious reasons you'll run into trouble, but then: arrays are binary data, so changing from text mode to binary mode is well worth the effort even if you just consider it a nuisance.

Breaking existing code that works should be considered more than a nuisance.

Its an error that pretty easy to fix... that's what I was referring to with "nuisance". All you have to do is open the file in binary mode and you're done. BTW, the change will only effect platforms that don't differ between text and binary mode, e.g. Unix ones.

However, one answer would be to have "t#" _prefer_ to use the text buffer, but not insist on it. eg, the logic for processing "t#" could check if the text buffer is supported, and if not move back to the blob buffer.

I doubt that this is conform to what the buffer interface want's to reflect: if the getcharbuf slot is not implemented this means "I am not text". If you would write non-text to a text file, this may cause line breaks to be interpreted in ways that are incompatible with the binary data, i.e. when you read the data back in, it may fail to load because e.g. '\n' was converted to '\r\n'.

This should mean that all existing code still works, except for objects that support both buffers to mean different things. AFAIK there are no objects that qualify today, so it should work fine.

Well, even though the code would work, it might break badly someday for the above reasons. Better fix that now when there aren't too many possible cases around than at some later point where the user has to figure out the problem for himself due to the system not warning him about this.

Unix users _will_ need to revisit their thinking about "text mode" vs "binary mode" when writing these new objects (such as Unicode), but IMO that is more than reasonable - Unix users dont bother qualifying the open mode of their files, simply because it has no effect on their files. If for certain objects or requirements there _is_ a distinction, then new code can start to think these issues through. "Portable File IO" will simply be extended from simply "portable among all platforms" to "portable among all platforms and objects".

Right. -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 42 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/

>> FYI, the next version of the proposal ... File objects opened in >> text mode will use "t#" and binary ones use "s#".

Tim> Am I the only one who sees magical distinctions between text and Tim> binary mode as a Really Bad Idea?

No.

Tim> I wouldn't have guessed the Unix natives here would quietly Tim> acquiesce to importing a bit of Windows madness <wink>.

We figured you and Guido would come to our rescue... ;-)

Don't count on me. My brain is totally cross-platform these days, and writing "rb" or "wb" for files containing binary data is second nature for me. I actually *like* it. Anyway, the Unicode stuff ought to have a wrapper open(filename, mode, encoding) where the 'b' will be added to the mode if you don't give it and it's needed. --Guido van Rossum (home page: http://www.python.org/~guido/)

Greg Stein writes:

[ man, I'm bad... I've got doc updates there and for the buffer stuff :-( ]

And the sooner I receive them, the sooner they can be integrated! Any plans to get them to me? I'll probably want to do another release before the IPC8. -Fred -- Fred L. Drake, Jr. Corporation for National Research Initiatives

Fred L. Drake, Jr. wrote:

M.-A. Lemburg writes:

...

Such a buffer is needed to implement "s" and "s#" argument parsing. It's a simple requirement to support those two parsing markers -- there's not much to argue about, really... unless, of course, you want to give up Unicode object support for all APIs using these parsers.

Perhaps I missed the agreement that these should always receive UTF-8 from Unicode strings.

from unicode import * def getname(): # hidden in some database engine, or so... return unicode("Linköping", "iso-8859-1") ... name = getname() # emulate automatic conversion to utf-8 name = str(name) # print it in uppercase, in the usual way import string print string.upper(name) ## LINKöPING I don't know, but I think that I think that it perhaps should raise an exception instead... </F>

M.-A. Lemburg writes:

It's been in the proposal since version 0.1. The idea is to provide a decent way of making existing script Unicode aware.

Ok, so I haven't read closely enough.

This is what I intended to implement. The <defencbuf> buffer will be filled upon the first request to the UTF-8 encoding. "s" and "s#" are examples of such requests. The buffer will remain intact until the object is destroyed (since other code could store the pointer received via e.g. "s").

Right.

Note that Unicode object are completely different beast ;-) String object are not touched in any way by the proposal.

I wasn't suggesting the PyStringObject be changed, only that the PyUnicodeObject could maintain a reference. Consider: s = fp.read() u = unicode(s, 'utf-8') u would now hold a reference to s, and s/s# would return a pointer into s instead of re-building the UTF-8 form. I talked myself out of this because it would be too easy to keep a lot more string objects around than were actually needed. -Fred -- Fred L. Drake, Jr. Corporation for National Research Initiatives

[Fred L. Drake, Jr.]

... I wasn't suggesting the PyStringObject be changed, only that the PyUnicodeObject could maintain a reference. Consider:

s = fp.read() u = unicode(s, 'utf-8')

u would now hold a reference to s, and s/s# would return a pointer into s instead of re-building the UTF-8 form. I talked myself out of this because it would be too easy to keep a lot more string objects around than were actually needed.

Yet another use for a weak reference <0.5 wink>.

"Fred L. Drake, Jr." wrote:

Tim Peters writes:

...

Yet another use for a weak reference <0.5 wink>.

Those just keep popping up! I seem to recall Diane Hackborne actually implemented these under the name "vref" long ago; perhaps that's worth revisiting after all? (Not the implementation so much as the idea.) I think to make it general would cost one PyObject* in each object's structure, and some code in some constructors (maybe), and all destructors, but not much. Is this worth pursuing, or is it locked out of the core because of the added space for the PyObject*? (Note that the concept isn't necessarily useful for all object types -- numbers in particular -- but it only makes sense to bother if it works for everything, even if it's not very useful in some cases.)

FYI, there's mxProxy which implements a flavor of them. Look in the standard places for mx stuff ;-) -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 45 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/

[Fred L. Drake, Jr., pines for some flavor of weak refs; MAL reminds us of his work; & back to Fred]

Yes, but still not in the core. So we have two general examples (vrefs and mxProxy) and there's WeakDict (or something like that). I think there really needs to be a core facility for this.

This kind of thing certainly belongs in the core (for efficiency and smooth integration) -- if it belongs in the language at all. This was discussed at length here some months ago; that's what prompted MAL to "do something" about it. Guido hasn't shown visible interest, and nobody has been willing to fight him to the death over it. So it languishes. Buy him lunch tomorrow and get him excited <wink>.

[Fred L. Drake, Jr.]

Guido has asked me to pursue this topic [weak refs], so I'll be checking out available implementations and seeing if any are adoptable or if something different is needed to be fully general and well-integrated.

Just don't let "fully general" stop anything for its sake alone; e.g., if there's a slick trick that *could* exempt numbers, that's all to the good! Adding a pointer to every object is really unattractive, while adding a flag or two to type objects is dirt cheap. Note in passing that current Java addresses weak refs too (several flavors of 'em! -- very elaborate).

I am with Fredrik on that auxilliary buffer. You'll have two dead bodies to deal with :-)

I haven't made up my mind yet (due to a very successful Python-promoting visit to SD'99 east, I'm about 100 msgs behind in this thread alone) but let me warn you that I can deal with the carnage, if necessary. :-) --Guido van Rossum (home page: http://www.python.org/~guido/)

On Fri, 12 Nov 1999, Tim Peters wrote:

... Using UTF-8 internally is also reasonable, and if it's being rejected on the grounds of supposed slowness

No... my main point was interaction with the underlying OS. I made a SWAG (Scientific Wild Ass Guess :-) and stated that UTF-8 is probably slower for various types of operations. As always, your infernal meddling has dashed that hypothesis, so I must retreat...

... I expect either would work well. It's at least curious that Perl and Tcl both went with UTF-8 -- does anyone think they know *why*? I don't. The people here saying UCS-2 is the obviously better choice are all from the Microsoft camp <wink>. It's not obvious to me, but then neither do I claim that UTF-8 is obviously better.

Probably for the exact reason that you stated in your messages: many 8-bit (7-bit?) functions continue to work quite well when given a UTF-8-encoded string. i.e. they didn't have to rewrite the entire Perl/TCL interpreter to deal with a new string type. I'd guess it is a helluva lot easier for us to add a Python Type than for Perl or TCL to whack around with new string types (since they use strings so heavily). Cheers, -g -- Greg Stein, http://www.lyra.org/

Greg Stein wrote:

On Wed, 10 Nov 1999, M.-A. Lemburg wrote:

...

... Well almost... it depends on the current value of <default encoding>.

Default encodings are kind of nasty when they can be altered. The same problem occurred with import hooks. Only one can be present at a time. This implies that modules, packages, subsystems, whatever, cannot set a default encoding because something else might depend on it having a different value. In the end, nobody uses the default encoding because it is unreliable, so you end up with extra implementation/semantics that aren't used/needed.

I know, but this is a little different: you use strings a lot while import hooks are rarely used directly by the user. E.g. people in Europe will probably prefer Latin-1 as default encoding while people in Asia will use one of the common CJK encodings. The <default encoding> decides what encoding to use for many typical tasks: printing, str(u), "s" argument parsing, etc. Note that setting the <default encoding> is not intended to be done prior to single operations. It is meant to be settable at thread creation time.

[...]

...

BTW, I'm still not too sure about the underlying internal format. The problem here is that Unicode started out as 2-byte fixed length representation (UCS2) but then shifted towards a 4-byte fixed length reprensetation known as UCS4. Since having 4 bytes per character is hard sell to customers, UTF16 was created to stuff the UCS4 code points (this is how character entities are called in Unicode) into 2 bytes... with a variable length encoding.

History is basically irrelevant. What is the situation today? What is in use, and what are people planning for right now?

...

... The downside of using UTF16: it is a variable length format, so iterations over it will be slower than for UCS4.

Bzzt. May as well go with UTF-8 as the internal format, much like Perl is doing (as I recall).

Why go with a variable length format, when people seem to be doing fine with UCS-2?

The reason for UTF-16 is simply that it is identical to UCS-2 over large ranges which makes optimizations (e.g. the UCS2 flag I mentioned in an earlier post) feasable and effective. UTF-8 slows things down for CJK encodings, since the APIs will very often have to scan the string to find the correct logical position in the data. Here's a quote from the Unicode FAQ (http://www.unicode.org/unicode/faq/ ): """ Q: How about using UCS-4 interfaces in my APIs? Given an internal UTF-16 storage, you can, of course, still index into text using UCS-4 indices. However, while converting from a UCS-4 index to a UTF-16 index or vice versa is fairly straightforward, it does involve a scan through the 16-bit units up to the index point. In a test run, for example, accessing UTF-16 storage as UCS-4 characters results in a 10X degradation. Of course, the precise differences will depend on the compiler, and there are some interesting optimizations that can be performed, but it will always be slower on average. This kind of performance hit is unacceptable in many environments. Most Unicode APIs are using UTF-16. The low-level character indexing are at the common storage level, with higher-level mechanisms for graphemes or words specifying their boundaries in terms of the storage units. This provides efficiency at the low levels, and the required functionality at the high levels. Convenience APIs can be produced that take parameters in UCS-4 methods for common utilities: e.g. converting UCS-4 indices back and forth, accessing character properties, etc. Outside of indexing, differences between UCS-4 and UTF-16 are not as important. For most other APIs outside of indexing, characters values cannot really be considered outside of their context--not when you are writing internationalized code. For such operations as display, input, collation, editing, and even upper and lowercasing, characters need to be considered in the context of a string. That means that in any event you end up looking at more than one character. In our experience, the incremental cost of doing surrogates is pretty small. """

Like I said in the other mail note: two large platforms out there are UCS-2 based. They seem to be doing quite well with that approach.

If people truly need UCS-4, then they can work with that on their own. One of the major reasons for putting Unicode into Python is to increase/simplify its ability to speak to the underlying platform. Hey! Guess what? That generally means UCS2.

All those formats are upward compatible (within certain ranges) and the Python Unicode API will provide converters between its internal format and the few common Unicode implementations, e.g. for MS compilers (16-bit UCS2 AFAIK), GLIBC (32-bit UCS4).

If we didn't need to speak to the OS with these Unicode values, then people can work with the values entirely in Python, PyUnicodeType-be-damned.

Are we digging a hole for ourselves? Maybe. But there are two other big platforms that have the same hole to dig out of *IF* it ever comes to that. I posit that it won't be necessary; that the people needing UCS-4 can do so entirely in Python.

Maybe we can allow the encoder to do UCS-4 to UTF-8 encoding and vice-versa. But: it only does it from String to String -- you can't use Unicode objects anywhere in there.

See above.

...

Simply sticking to UCS2 is probably out of the question, since Unicode 3.0 requires UCS4 and we are targetting Unicode 3.0.

Oh? Who says?

From the FAQ: """ Q: What is UTF-16?

Unicode was originally designed as a pure 16-bit encoding, aimed at representing all modern scripts. (Ancient scripts were to be represented with private-use characters.) Over time, and especially after the addition of over 14,500 composite characters for compatibility with legacy sets, it became clear that 16-bits were not sufficient for the user community. Out of this arose UTF-16. """ Note that there currently are no defined surrogate pairs for UTF-16, meaning that in practice the difference between UCS-2 and UTF-16 is probably negligable, e.g. we could define the internal format to be UTF-16 and raise exception whenever the border between UTF-16 and UCS-2 is crossed -- sort of as political compromise ;-). But... I think HP has the last word on this one. -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 51 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/

Tim Peters wrote:

UTF-8 is a very nice encoding scheme, but is very hard for people "to do" by hand -- it breaks apart and rearranges bytes at the bit level, and everything other than 7-bit ASCII requires solid strings of "high-bit" characters.

unless you're using a UTF-8 aware editor, of course ;-) (some days, I think we need some way to tell the compiler what encoding we're using for the source file...)

This is painful for people to enter manually on both counts -- and no common reference gives the UTF-8 encoding of glyphs directly. So, as discussed earlier, we should follow Java's lead and also introduce a \u escape sequence:

octet: hexdigit hexdigit unicodecode: octet octet unicode_escape: "\\u" unicodecode

Inside a u'' string, I guess this should expand to the UTF-8 encoding of the Unicode character at the unicodecode code position. For consistency, then, it should probably expand the same way inside "regular strings" too. Unlike Java does, I'd rather not give it a meaning outside string literals.

good idea. and by some reason, patches for this is included in the unicode distribution (see the attached str2utf.c).

The other point is a nit: The vast bulk of UTF-8 encodings encode characters in UCS-4 space outside of Unicode. In good Pythonic fashion, those must either be explicitly outlawed, or explicitly defined. I vote for outlawed, in the sense of detected error that raises an exception. That leaves our future options open.

I vote for 'outlaw'. </F> /* A small code snippet that translates \uxxxx syntax to UTF-8 text. To be cut and pasted into Python/compile.c */ /* Written by Fredrik Lundh, January 1999. */ /* Documentation (for the language reference): \uxxxx -- Unicode character with hexadecimal value xxxx. The character is stored using UTF-8 encoding, which means that this sequence can result in up to three encoded characters. Note that the 'u' must be followed by four hexadecimal digits. If fewer digits are given, the sequence is left in the resulting string exactly as given. If more digits are given, only the first four are translated to Unicode, and the remaining digits are left in the resulting string. */ #define Py_CHARMASK(ch) ch void convert(const char *s, char *p) { while (*s) { if (*s != '\\') { *p++ = *s++; continue; } s++; switch (*s++) { /* -------------------------------------------------------------------- */ /* copy this section to the appropriate place in compile.c... */ case 'u': /* \uxxxx => UTF-8 encoded unicode character */ if (isxdigit(Py_CHARMASK(s[0])) && isxdigit(Py_CHARMASK(s[1])) && isxdigit(Py_CHARMASK(s[2])) && isxdigit(Py_CHARMASK(s[3]))) { /* fetch hexadecimal character value */ unsigned int n, ch = 0; for (n = 0; n < 4; n++) { int c = Py_CHARMASK(*s); s++; ch = (ch << 4) & ~0xF; if (isdigit(c)) ch += c - '0'; else if (islower(c)) ch += 10 + c - 'a'; else ch += 10 + c - 'A'; } /* store as UTF-8 */ if (ch < 0x80) *p++ = (char) ch; else { if (ch < 0x800) { *p++ = 0xc0 | (ch >> 6); *p++ = 0x80 | (ch & 0x3f); } else { *p++ = 0xe0 | (ch >> 12); *p++ = 0x80 | ((ch >> 6) & 0x3f); *p++ = 0x80 | (ch & 0x3f); } } break; } else goto bogus; /* -------------------------------------------------------------------- */ default: bogus: *p++ = '\\'; *p++ = s[-1]; break; } } *p++ = '\0'; } main() { int i; unsigned char buffer[100]; convert("Link\\u00f6ping", buffer); for (i = 0; buffer[i]; i++) if (buffer[i] < 0x20 || buffer[i] >= 0x80) printf("\\%03o", buffer[i]); else printf("%c", buffer[i]); }

Tim Peters wrote:

[/F, dripping with code]

...

... Note that the 'u' must be followed by four hexadecimal digits. If fewer digits are given, the sequence is left in the resulting string exactly as given.

Yuck -- don't let probable error pass without comment. "must be" == "must be"!

I second that.

[moving backwards]

...

\uxxxx -- Unicode character with hexadecimal value xxxx. The character is stored using UTF-8 encoding, which means that this sequence can result in up to three encoded characters.

The code is fine, but I've gotten confused about what the intent is now. Expanding \uxxxx to its UTF-8 encoding made sense when MAL had UTF-8 literals, but now he's got Unicode-escaped literals instead -- and you favor an internal 2-byte-per-char Unicode storage format. In that combination of worlds, is there any use in the *language* (as opposed to in a runtime module) for \uxxxx -> UTF-8 conversion?

No, no... :-) I think it was a simple misunderstanding... \uXXXX is only to be used within u'' strings and then gets expanded to *one* character encoded in the internal Python format (which is heading towards UTF-16 without surrogates).

And MAL, if you're listening, I'm not clear on what a Unicode-escaped literal means. When you had UTF-8 literals, the meaning of something like

u"a\340\341"

was clear, since UTF-8 is defined as a byte stream and UTF-8 string literals were just a way of specifying a byte stream. As a Unicode-escaped string, I assume the "a" maps to the Unicode "a", but what of the rest? Are the octal escapes to be taken as two separate Latin-1 characters (in their role as a Unicode subset), or as an especially clumsy way to specify a single 16-bit Unicode character? I'm afraid I'd vote for the former. Same issue wrt \x escapes.

Good points. The conversion goes as follows: · for single characters (and this includes all \XXX sequences except \uXXXX), take the ordinal and interpret it as Unicode ordinal · for \uXXXX sequences, insert the Unicode character with ordinal 0xXXXX instead

One other issue: are there "raw" Unicode strings too, as in ur"\u20ac"? There probably should be; and while Guido will hate this, a ur string should probably *not* leave \uxxxx escapes untouched. Nasties like this are why Java defines \uxxxx expansion as occurring in a preprocessing step.

Not sure whether we really need to make this even more complicated... The \uXXXX strings look ugly, adding a few \\\\ for e.g. REs or filenames won't hurt much in the context of those \uXXXX monsters :-)

BTW, the meaning of \uxxxx in a non-Unicode string is now also unclear (or isn't \uxxxx allowed in a non-Unicode string? that's what I would do ...).

Right. \uXXXX will only be allowed in u'' strings, not in "normal" strings. BTW, if you want to type in UTF-8 strings and have them converted to Unicode, you can use the standard: u = unicode('...string with UTF-8 encoded characters...','utf-8') -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 50 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/

Tim Peters wrote:

[MAL]

...

... The conversion goes as follows: · for single characters (and this includes all \XXX sequences except \uXXXX), take the ordinal and interpret it as Unicode ordinal for \uXXXX sequences, insert the Unicode character with ordinal 0xXXXX instead

Perfect!

Thanks :-)

[about "raw" Unicode strings]

...

... Not sure whether we really need to make this even more complicated... The \uXXXX strings look ugly, adding a few \\\\ for e.g. REs or filenames won't hurt much in the context of those \uXXXX monsters :-)

Alas, this won't stand over the long term. Eventually people will write Python using nothing but Unicode strings -- "regular strings" will eventurally become a backward compatibility headache <0.7 wink>. IOW, Unicode regexps and Unicode docstrings and Unicode formatting ops ... nothing will escape. Nor should it.

I don't think it all needs to be done at once, though -- existing languages usually take years to graft in gimmicks to cover all the fine points. So, happy to let raw Unicode strings pass for now, as a relatively minor point, but without agreeing it can be ignored forever.

Agreed... note that you could also write your own codec for just this reason and then use: u = unicode('....\u1234...\...\...','raw-unicode-escaped') Put that into a function called 'ur' and you have: u = ur('...\u4545...\...\...') which is not that far away from ur'...' w/r to cosmetics.

...

... BTW, if you want to type in UTF-8 strings and have them converted to Unicode, you can use the standard:

u = unicode('...string with UTF-8 encoded characters...','utf-8')

That's what I figured, and thanks for the confirmation.

-- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 49 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/

Tim Peters wrote:

(\b is supposed to be a word boundary assertion).

in some places, that is. </F> Main Entry: reg·u·lar Pronunciation: 're-gy&-l&r, 're-g(&-)l&r 1 : belonging to a religious order 2 a : formed, built, arranged, or ordered according to some established rule, law, principle, or type ... 3 a : ORDERLY, METHODICAL <regular habits> ... 4 a : constituted, conducted, or done in conformity with established or prescribed usages, rules, or discipline ...

[MAL]

... Here is a sample implementation of what I had in mind:

""" Demo for 'unicode-escape' encoding. """ import struct,string,re

pack_format = '>H'

def convert_string(s):

l = map(None,s) for i in range(len(l)): l[i] = struct.pack(pack_format,ord(l[i])) return l

u_escape = re.compile(r'\\u([0-9a-fA-F]{0,4})')

def unicode_unescape(s):

l = [] start = 0 while start < len(s): m = u_escape.search(s,start) if not m: l[len(l):] = convert_string(s[start:]) break m_start,m_end = m.span() if m_start > start: l[len(l):] = convert_string(s[start:m_start]) hexcode = m.group(1) #print hexcode,start,m_start if len(hexcode) != 4: raise SyntaxError,'illegal \\uXXXX sequence: \\u%s' % hexcode ordinal = string.atoi(hexcode,16) l.append(struct.pack(pack_format,ordinal)) start = m_end #print l return string.join(l,'')

def hexstr(s,sep=''):

return string.join(map(lambda x,hex=hex,ord=ord: '%02x' % ord(x),s),sep)

It looks like r'\\u0000' will get translated into a 2-character Unicode string. That's probably not good, if for no other reason than that Java would not do this (it would create the obvious 7-character Unicode string), and having something that looks like a Java escape that doesn't *work* like the Java escape will be confusing as heck for JPython users. Keeping track of even-vs-odd number of backslashes can't be done with a regexp search, but is easy if the code is simple <wink>: def unicode_unescape(s): from string import atoi import array i, n = 0, len(s) result = array.array('H') # unsigned short, native order while i < n: ch = s[i] i = i+1 if ch != "\\": result.append(ord(ch)) continue if i == n: raise ValueError("string ends with lone backslash") ch = s[i] i = i+1 if ch != "u": result.append(ord("\\")) result.append(ord(ch)) continue hexchars = s[i:i+4] if len(hexchars) != 4: raise ValueError("\\u escape at end not followed by " "at least 4 characters") i = i+4 for ch in hexchars: if ch not in "01234567890abcdefABCDEF": raise ValueError("\\u" + hexchars + " contains " "non-hex characters") result.append(atoi(hexchars, 16)) # print result return result.tostring()

[MAL]

Guido and I have decided to turn \uXXXX into a standard escape sequence with no further magic applied. \uXXXX will only be expanded in u"" strings.

Does that exclude ur"" strings? Not arguing either way, just don't know what all this means.

Here's the new scheme:

With the 'unicode-escape' encoding being defined as:

· all non-escape characters represent themselves as a Unicode ordinal (e.g. 'a' -> U+0061).

Same as before (scream if that's wrong).

· all existing defined Python escape sequences are interpreted as Unicode ordinals;

Same as before (ditto).

note that \xXXXX can represent all Unicode ordinals,

This means that the definition of \xXXXX has changed, then -- as you pointed out just yesterday <wink>, \xABCDq currently acts like \xCDq. Does the new \x definition apply only in u"" strings, or in "" strings too? What is the new \x definition?

and \OOO (octal) can represent Unicode ordinals up to U+01FF.

Same as before (ditto).

· a new escape sequence, \uXXXX, represents U+XXXX; it is a syntax error to have fewer than 4 digits after \u.

Same as before (ditto). IOW, I don't see anything that's changed other than an unspecified new treatment of \x escapes, and possibly that ur"" strings don't expand \u escapes.

Examples:

u'abc' -> U+0061 U+0062 U+0063 u'\u1234' -> U+1234 u'abc\u1234\n' -> U+0061 U+0062 U+0063 U+1234 U+05c

The last example is damaged (U+05c isn't legit). Other than that, these look the same as before.

Now how should we define ur"abc\u1234\n" ... ?

If strings carried an encoding tag with them, the obvious answer is that this acts exactly like r"abc\u1234\n" acts today except gets a "unicode-escaped" encoding tag instead of a "[whatever the default is today]" encoding tag. If strings don't carry an encoding tag with them, you're in a bit of a pickle: you'll have to convert it to a regular string or a Unicode string, but in either case have no way to communicate that it may need further processing; i.e., no way to distinguish it from a regular or Unicode string produced by any other mechanism. The code I posted yesterday remains my best answer to that unpleasant puzzle (i.e., produce a Unicode string, fiddling with backslashes just enough to get the \u escapes expanded, in the same way Java's (conceptual) preprocessor does it).

...

...

Now how should we define ur"abc\u1234\n" ... ?

If strings carried an encoding tag with them, the obvious answer is that this acts exactly like r"abc\u1234\n" acts today except gets a "unicode-escaped" encoding tag instead of a "[whatever the default is today]" encoding tag.

If strings don't carry an encoding tag with them, you're in a bit of a pickle: you'll have to convert it to a regular string or a Unicode string, but in either case have no way to communicate that it may need further processing; i.e., no way to distinguish it from a regular or Unicode string produced by any other mechanism. The code I posted yesterday remains my best answer to that unpleasant puzzle (i.e., produce a Unicode string, fiddling with backslashes just enough to get the \u escapes expanded, in the same way Java's (conceptual) preprocessor does it).

They don't have such tags... so I guess we're in trouble ;-)

I guess to make ur"" have a meaning at all, we'd need to go the Java preprocessor way here, i.e. scan the string *only* for \uXXXX sequences, decode these and convert the rest as-is to Unicode ordinals.

Would that be ok ?

Read Tim's code (posted about 40 messages ago in this list). Like Java, it interprets \u.... when the number of backslashes is odd, but not when it's even. So \\u.... returns exactly that, while \\\u.... returns two backslashes and a unicode character. This is nice and can be done regardless of whether we are going to interpret other \ escapes or not. --Guido van Rossum (home page: http://www.python.org/~guido/)

...

Like Java, it interprets \u.... when the number of backslashes is odd, but not when it's even. So \\u.... returns exactly that, while \\\u.... returns two backslashes and a unicode character.

This is nice and can be done regardless of whether we are going to interpret other \ escapes or not.

So I'll take that as: this is what we want in Python too :-)

I'll reserve judgement until we've got some experience with it in the field, but it seems the best compromise. It also gives a clear explanation about why we have \uXXXX when we already have \xXXXX. --Guido van Rossum (home page: http://www.python.org/~guido/)

Would this definition be fine ? """

u = ur'<raw-unicode-escape encoded Python string>'

The 'raw-unicode-escape' encoding is defined as follows:

· \uXXXX sequence represent the U+XXXX Unicode character if and only if the number of leading backslashes is odd

· all other characters represent themselves as Unicode ordinal (e.g. 'b' -> U+0062)

"""

Yes. --Guido van Rossum (home page: http://www.python.org/~guido/)

On Wed, 10 Nov 1999, M.-A. Lemburg wrote:

Tim Peters wrote:

...

BTW, is ord(unicode_char) defined? And as what? And does ord have an inverse in the Unicode world? Both seem essential.

Good points.

How about

uniord(u[:1]) --> Unicode ordinal number (32-bit)

unichr(i) --> Unicode object for character i (provided it is 32-bit); ValueError otherwise

Why new functions? Why not extend the definition of ord() and chr()? In terms of backwards compatibility, the only issue could possibly be that people relied on chr(x) to throw an error when x>=256. They certainly couldn't pass a Unicode object to ord(), so that function can safely be extended to accept a Unicode object and return a larger integer. Cheers, -g -- Greg Stein, http://www.lyra.org/

[MAL, on Unicode chr() and ord()

... Because unichr() will always have to return Unicode objects. You don't want chr(i) to return Unicode for i>255 and strings for i<256.

Indeed I do not!

OTOH, ord() could probably be extended to also work on Unicode objects.

I think should be -- it's a good & natural use of polymorphism; introducing a new function *here* would be as odd as introducing a unilen() function to get the length of a Unicode string.

Guido van Rossum wrote:

http://starship.skyport.net/~lemburg/unicode-proposal.txt

Marc-Andre writes: The internal format for Unicode objects should either use a Python specific fixed cross-platform format <PythonUnicode> (e.g. 2-byte little endian byte order) or a compiler provided wchar_t format (if available). Using the wchar_t format will ease embedding of Python in other Unicode aware applications, but will also make internal format dumps platform dependent. having been there and done that, I strongly suggest a third option: a 16-bit unsigned integer, in platform specific byte order (PY_UNICODE_T). along all other roads lie code bloat and speed penalties... (besides, this is exactly how it's already done in unicode.c and what 'sre' prefers...) </F>

What I don't like is using wchar_t if available (and then addressing it as if it were defined as unsigned integer). IMO, it's better to define a Python Unicode representation which then gets converted to whatever wchar_t represents on the target machine.

you should read the unicode.h file a bit more carefully: ... /* Unicode declarations. Tweak these to match your platform */ /* set this flag if the platform has "wchar.h", "wctype.h" and the wchar_t type is a 16-bit unsigned type */ #define HAVE_USABLE_WCHAR_H #if defined(WIN32) || defined(HAVE_USABLE_WCHAR_H) (this uses wchar_t, and also iswspace and friends) ... #else /* Use if you have a standard ANSI compiler, without wchar_t support. If a short is not 16 bits on your platform, you have to fix the typedef below, or the module initialization code will complain. */ (this maps iswspace to isspace, for 8-bit characters). #endif ... the plan was to use the second solution (using "configure" to figure out what integer type to use), and its own uni- code database table for the is/to primitives (iirc, the unicode.txt file discussed this, but that one seems to be missing from the zip archive). </F>

Fredrik Lundh writes:

having been there and done that, I strongly suggest a third option: a 16-bit unsigned integer, in platform specific byte order (PY_UNICODE_T). along all other

I actually like this best, but I understand that there are reasons for using wchar_t, especially for interfacing with other code that uses Unicode. Perhaps someone who knows more about the specific issues with interfacing using wchar_t can summarize them, or point me to whatever I've already missed. p-) -Fred -- Fred L. Drake, Jr. Corporation for National Research Initiatives

Guido van Rossum wrote:

One specific question: in you discussion of typed strings, I'm not sure why you couldn't convert everything to Unicode and be done with it. I have a feeling that the answer is somewhere in your case study -- maybe you can elaborate?

Marc-Andre writes: Unicode objects should have a pointer to a cached (read-only) char buffer <defencbuf> holding the object's value using the current <default encoding>. This is needed for performance and internal parsing (see below) reasons. The buffer is filled when the first conversion request to the <default encoding> is issued on the object. keeping track of an external encoding is better left for the application programmers -- I'm pretty sure that different application builders will want to handle this in radically different ways, depending on their environ- ment, underlying user interface toolkit, etc. besides, this is how Tcl would have done it. Python's not Tcl, and I think you need *very* good arguments for moving in that direction. </F>

Just a couple observations from the peanut gallery... 1. I'm glad I don't have to do this Unicode/UTF/internationalization stuff. Seems like it would be easier to just get the whole world speaking Esperanto. 2. Are there plans for an internationalization session at IPC8? Perhaps a few key players could be locked into a room for a couple days, to emerge bloodied, but with an implementation in-hand... Skip Montanaro | http://www.mojam.com/ skip@mojam.com | http://www.musi-cal.com/ 847-971-7098 | Python: Programming the way Guido indented...