On 27/07/13 11:01, Steven D'Aprano wrote:

Unicode's standard notation for code points is U+ followed by a 4, 5 or 6 hex digit string, such as π = U+03C0. This notation is found throughout the Unicode Consortium's website, e.g.:

http://www.unicode.org/versions/corrigendum2.html

as well as in third party sites that have reason to discuss Unicode code points, e.g.:

https://en.wikipedia.org/wiki/Eth#Computer_input

I propose that Python strings support this as the preferred escape notation for Unicode code points:

'\U+03C0' => 'π'

The existing \U and \u variants must be kept for backwards compatibility, but should be (mildly) discouraged in new code.

Doesn't this violate "Only One Way To Do It"? ---------------------------------------------

That's not what the Zen says. The Zen says there should be One Obvious Way to do it, not Only One. It is my hope that we can agree that the One Obvious Way to refer to a Unicode character by its code point is by using the same notation that the Unicode Consortium uses:

d <=> U+0064

and leave legacy escape sequences as the not-so-obvious ways to do it:

\x64 \144 \u0064 \U00000064

Why do we need yet another way of writing escape sequences? -----------------------------------------------------------

We don't need another one, we need a better one. U+xxxx is the standard Unicode notation, while existing Python escapes have various problems.

One-byte hex and oct escapes are a throwback to the old one-byte ASCII days, and reflect an obsolete idea of strings being equivalent to bytes. Backwards compatibility requires that we continue to support them, but they shouldn't be encouraged in strings.

Two-byte \u escapes are harmless, so long as you imagine that Unicode is a 16-bit character set. Unfortunately, it is not. \u does not support code points in the Supplementary Multilingual Planes (those with ordinal value greater than 0xFFFF), and can silently give the wrong result if you make a mistake in counting digits:

# I want EGYPTIAN HIEROGLYPH D010 (Eye of Horus) s = '\u13080' => oops, I get 'ገ0' (ETHIOPIC SYLLABLE GA, ZERO)

Four-byte \U escape sequences support the entire Unicode character set, but they are terribly verbose, and the first three digits are *always* zero. Python doesn't (and shouldn't) support \U escapes beyond 10FFFF, so the first three digits of the eight digit hex value are pointless.

What is the U+ escape specification? ------------------------------------

http://docs.python.org/3/reference/lexical_analysis.html#string-and-bytes-li...

lists the escape sequences, including:

\uxxxx Character with 16-bit hex value xxxx \Uxxxxxxxx Character with 32-bit hex value xxxxxxxx

To this should be added:

\U+xxxx Character at code point xxxx (hex)

with the note:

Exactly 4, 5 or 6 hexadecimal digits are required.

Upper or lower case? --------------------

Uppercase should be preferred, as the Unicode Consortium uses it, but both should be accepted.

Variable number of digits? Isn't that a bad thing? --------------------------------------------------

It's neither good nor bad. Octal escapes already support from 1 to 3 oct digits. In some languages (but not Python), hex escapes support from 1 to an unlimited number of hex digits.

Is this backwards compatible? -----------------------------

I believe it is. As of Python 3.3, strings using \U+ give a syntax error:

py> '\U+13080' File "<stdin>", line 1 SyntaxError: (unicode error) 'unicodeescape' codec can't decode bytes in position 0-7: end of string in escape sequence

What deprecation schedule are you proposing? --------------------------------------------

I'm not. At least, the existing features should not be considered for removal before Python 4000. In the meantime, the U+ form should be noted as the preferred way, and perhaps blessed in PEP 8.

Should string reprs use the U+ form? ------------------------------------

\u escapes are sometimes used in string reprs, e.g. for private-use characters:

py> chr(0xE034) '\ue034'

Should this change to '\U+E034'? My personal preference is that it should, but I fear backwards compatibility may prevent it. Even if the exact form of str.__repr__ is not guaranteed, changing the repr would break (e.g.) some doctests.

This proposal defers any discussion of changing the repr of strings to use U+ escapes.

What should 'U+12345' be? U+12345 CUNEIFORM SIGN URU TIMES KI or U+1234 ETHIOPIC SYLLABLE SEE and a digit 5? -1 without a clear way to disambiguate. Regards, Ian

On Jul 27, 2013, at 12:22, Ian Foote wrote:

On 27/07/13 11:01, Steven D'Aprano wrote:

...

Unicode's standard notation for code points is U+ followed by a 4, 5 or 6 hex digit string, such as π = U+03C0. This notation is found throughout the Unicode Consortium's website, e.g.:

http://www.unicode.org/versions/corrigendum2.html

as well as in third party sites that have reason to discuss Unicode code points, e.g.:

https://en.wikipedia.org/wiki/Eth#Computer_input

I propose that Python strings support this as the preferred escape notation for Unicode code points:

'\U+03C0' => 'π'

The existing \U and \u variants must be kept for backwards compatibility, but should be (mildly) discouraged in new code.

Doesn't this violate "Only One Way To Do It"? ---------------------------------------------

That's not what the Zen says. The Zen says there should be One Obvious Way to do it, not Only One. It is my hope that we can agree that the One Obvious Way to refer to a Unicode character by its code point is by using the same notation that the Unicode Consortium uses:

d <=> U+0064

and leave legacy escape sequences as the not-so-obvious ways to do it:

\x64 \144 \u0064 \U00000064

Why do we need yet another way of writing escape sequences? -----------------------------------------------------------

We don't need another one, we need a better one. U+xxxx is the standard Unicode notation, while existing Python escapes have various problems.

One-byte hex and oct escapes are a throwback to the old one-byte ASCII days, and reflect an obsolete idea of strings being equivalent to bytes. Backwards compatibility requires that we continue to support them, but they shouldn't be encouraged in strings.

Two-byte \u escapes are harmless, so long as you imagine that Unicode is a 16-bit character set. Unfortunately, it is not. \u does not support code points in the Supplementary Multilingual Planes (those with ordinal value greater than 0xFFFF), and can silently give the wrong result if you make a mistake in counting digits:

# I want EGYPTIAN HIEROGLYPH D010 (Eye of Horus) s = '\u13080' => oops, I get 'ገ0' (ETHIOPIC SYLLABLE GA, ZERO)

Four-byte \U escape sequences support the entire Unicode character set, but they are terribly verbose, and the first three digits are *always* zero. Python doesn't (and shouldn't) support \U escapes beyond 10FFFF, so the first three digits of the eight digit hex value are pointless.

What is the U+ escape specification? ------------------------------------

http://docs.python.org/3/reference/lexical_analysis.html#string-and-bytes-li...

lists the escape sequences, including:

\uxxxx Character with 16-bit hex value xxxx \Uxxxxxxxx Character with 32-bit hex value xxxxxxxx

To this should be added:

\U+xxxx Character at code point xxxx (hex)

with the note:

Exactly 4, 5 or 6 hexadecimal digits are required.

Upper or lower case? --------------------

Uppercase should be preferred, as the Unicode Consortium uses it, but both should be accepted.

Variable number of digits? Isn't that a bad thing? --------------------------------------------------

It's neither good nor bad. Octal escapes already support from 1 to 3 oct digits. In some languages (but not Python), hex escapes support from 1 to an unlimited number of hex digits.

Is this backwards compatible? -----------------------------

I believe it is. As of Python 3.3, strings using \U+ give a syntax error:

py> '\U+13080' File "<stdin>", line 1 SyntaxError: (unicode error) 'unicodeescape' codec can't decode bytes in position 0-7: end of string in escape sequence

What deprecation schedule are you proposing? --------------------------------------------

I'm not. At least, the existing features should not be considered for removal before Python 4000. In the meantime, the U+ form should be noted as the preferred way, and perhaps blessed in PEP 8.

Should string reprs use the U+ form? ------------------------------------

\u escapes are sometimes used in string reprs, e.g. for private-use characters:

py> chr(0xE034) '\ue034'

Should this change to '\U+E034'? My personal preference is that it should, but I fear backwards compatibility may prevent it. Even if the exact form of str.__repr__ is not guaranteed, changing the repr would break (e.g.) some doctests.

This proposal defers any discussion of changing the repr of strings to use U+ escapes.

What should 'U+12345' be? U+12345 CUNEIFORM SIGN URU TIMES KI or U+1234 ETHIOPIC SYLLABLE SEE and a digit 5?

-1 without a clear way to disambiguate.

We already have the exact same problem with octal literals. They can be one to three digits, ending at the first non-octal-digit character (or end of string). So '\123' is unambiguously 'S', while '\128' is unambiguously '\n8'. Not exactly beautiful, but simple, and a precedent going back to the earliest days of Python, and beyond it to C. So if we followed the same rule, '\U+12345' would unambiguously be character U+12345, while '\U+1234@' would be U+1234 and a @. That doesn't mean it's necessarily a good idea. After all, we don't allow 1-char hex escapes. And octal escapes are already pretty weird, in that they don't encode only characters up to 127 (as in C) or all of Unicode, but everything up to 511 (because that happens to be the max you can fit into the rules), so maybe they're not a great precedent to follow.

Regards, Ian

_______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas

On 27/07/13 20:22, Ian Foote wrote:

On 27/07/13 11:01, Steven D'Aprano wrote:

...

Variable number of digits? Isn't that a bad thing? --------------------------------------------------

It's neither good nor bad. Octal escapes already support from 1 to 3 oct digits. In some languages (but not Python), hex escapes support from 1 to an unlimited number of hex digits.

What should 'U+12345' be? U+12345 CUNEIFORM SIGN URU TIMES KI or U+1234 ETHIOPIC SYLLABLE SEE and a digit 5?

There is no ambiguity. Just like oct escapes, the longest valid sequence (up to the maximum) would be used. If you used the shortest, then there would be no way to specify 5 or 6 digit sequences. -- Steven

On Sat, Jul 27, 2013 at 12:22 PM, Steven D'Aprano wrote:

On 27/07/13 20:22, Ian Foote wrote:

...

On 27/07/13 11:01, Steven D'Aprano wrote:

...

...

Variable number of digits? Isn't that a bad thing? --------------------------------------------------

It's neither good nor bad. Octal escapes already support from 1 to 3 oct digits. In some languages (but not Python), hex escapes support from 1 to an unlimited number of hex digits.

...

What should 'U+12345' be? U+12345 CUNEIFORM SIGN URU TIMES KI or U+1234 ETHIOPIC SYLLABLE SEE and a digit 5?

There is no ambiguity. Just like oct escapes, the longest valid sequence (up to the maximum) would be used. If you used the shortest, then there would be no way to specify 5 or 6 digit sequences.

In a vacuum, \U+12345 seems like a good thing. But two issues dog it: incompatibility with *every other language*, and the inability to follow it with a hex digit. With octal escapes, there's a limit of three digits, so you can simply stuff in an extra zero or two:

...

...

"\1234" 'S4' "\01234" '\n34' "\001234" '\x01234'

Granted, this isn't the case in all languages, but it's a reasonable convention to stick to. How many digits should be permitted in \U+ notation? Six? Eight? Will a quick eyeball of a string literal be able to figure out the correct interpretation of "\U+0012345678"? Also, this is a problem with a lot more characters than it is with octal, which unambiguously stops after any non-digit; in hex, there are two additional digits (8, 9) and twelve very common ASCII letters (A-F, a-f) which can cause problems. I foresee issues like with Windows paths in non-raw strings:

...

...

"c:\qwer" 'c:\\qwer' "c:\asdf" 'c:\x07sdf'

Some work, some don't. You'll put in a convenient four or five digit Unicode escape, follow it with a non-hex letter, and then later on come and edit and confuse yourself no end. I'm -1 on the proposal, primarily because it's different from everything else without being a significant improvement over them. On Sat, Jul 27, 2013 at 12:25 PM, Steven D'Aprano wrote:

On 27/07/13 20:07, M.-A. Lemburg wrote:

...

The \u and \U notations are standard in several programming languages, e.g. Java and C++, so we're in good company.

Given the problems with both \u and \U escapes, I think it is better to say we're in bad company.

Good or bad, it's a large company, and that *in itself* is of value. ChrisA

On 27/07/2013 12:17, Chris “Kwpolska” Warrick wrote:

On Sat, Jul 27, 2013 at 12:01 PM, Steven D'Aprano wrote:

...

Unicode's standard notation for code points is U+ followed by a 4, 5 or 6 hex digit string, such as π = U+03C0. This notation is found throughout the Unicode Consortium's website, e.g.:

http://www.unicode.org/versions/corrigendum2.html

as well as in third party sites that have reason to discuss Unicode code points, e.g.:

https://en.wikipedia.org/wiki/Eth#Computer_input

I propose that Python strings support this as the preferred escape notation for Unicode code points:

'\U+03C0' => 'π'

The existing \U and \u variants must be kept for backwards compatibility, but should be (mildly) discouraged in new code.

As Marc-Andre Lemburg said, C, C++ and Java use the same notation as Python does.

And there is NO programming language implementing the U+ syntax. Why should we? Why should we violate de-facto standards?

Existing programming languages use one or more of:

a) \uHHHH b) \UHHHHHHHH c) \u{H..HHHHHH} (eg. Ruby) c) \xH..HH d) \x{H..HHHHHH} e) \O..OOO

and probably some more variants I am not aware of or forgot about, but there is probably no programming language that does \U+{H..HHHHHH}, so why should we?

[snip] Perl supports "\N{U+1234}" and "\x{1234}", and I believe that some languages also support "\x{41 42 43}" as an abbreviation for "\x{41}\x{42}\x{43}". As others have said, "\U+1234" suffers from the same problem as octal escapes. -1

On Sat, Jul 27, 2013 at 4:47 PM, Steven D'Aprano wrote:

Unicode's notation is nice and simple. If we had it first, would we prefer \uxxxx and \U00xxxxxx over it? I don't think so.

Almost certainly not. Like I said, I think your idea is great *in a vacuum*. Obviously the removal of the current notations is out of the question, which means that this is yet another way to specify a codepoint; and it's one that most programmers won't be looking for. (I stand corrected, though: I had thought that there were *no* other languages using this notation. Of course, this is a silly thought. There is almost nothing that hasn't already been done, somewhere.) If Python had supported this notation from the beginning of Unicode strings, or at least since 3.0, then adding \uxxxx would have been purely as a sop to C/Java/etc programmers, and it would likely have gone nowhere. How much value is gained by creating a new syntax, which now Python programmers have to understand in addition to the existing ones? Consistency across languages is fairly important; have you ever used \123 notation in a BIND file? http://rosuav.blogspot.com/2012/12/i-want-my-octal.html Maybe Python will start a new trend, and \U+1234 will become the new convention. Maybe that's a good thing. But how beneficial will it be, and how complicating? I'm weakening my stance to -0. ChrisA

On 27/07/2013 17:46, Stephen J. Turnbull wrote:

Steven D'Aprano writes:

...

I propose that Python strings support this as the preferred escape notation for Unicode code points:

'\U+03C0' => 'π'

-1. Because:

...

The existing \U and \u variants must be kept for backwards compatibility, but should be (mildly) discouraged in new code.

OTOH, supporting "\N{U+03C0}" seems harmless, if not particularly useful, to me. However, I don't find it hard to imagine that some people would use it in preference to the \U and \u escpes, despite being somewhat verbose.

I think the point of "\N{U+03C0}" is that it lets you name all of the codepoints, even those that are as yet unnamed. :-)

On 7/27/2013 7:22 AM, Steven D'Aprano wrote:

On 27/07/13 20:22, Ian Foote wrote:

...

On 27/07/13 11:01, Steven D'Aprano wrote:

...

...

Variable number of digits? Isn't that a bad thing? --------------------------------------------------

It's neither good nor bad.

It is wretched. In the unicode standard, the U+ notation is used for single codepoints and as near as I can tell from checking a few chapters, always has a trailing delimiter (space or punctuation). This is true even for successive codepoints. For example: "katakana letter ainu to can simply be mapped to the Unicode character sequence ". Note that the authors did not simple write "U+30C8U+309A" as in this proposal. In other words, the proposal does not conform to the usage of the notation in the standard. In tables, the 'U+' is omitted. Sequential codepoints are separated by spaces for readability. For instance, '0069 0307 0301' in one table stands for the single grapheme 'i̇́' (Lithuanian char) == '\u0069\u0307\u0301' Even though a computer could parse 'U+0069U+0307U+0301' correctly, most humans eyes will see '+' as the separator. I find this more painful to read than the '\' form.

...

...

Octal escapes already support from 1 to 3 oct digits.

And there are awful to use in string literals, as opposed to numbers.

...

...

In some languages (but not Python), hex escapes support from 1 to an unlimited number of hex digits.

That is fine for numbers. For strings, 2*n hex digits often (typically?) means n bytes.

...

What should 'U+12345' be? U+12345 CUNEIFORM SIGN URU TIMES KI or U+1234 ETHIOPIC SYLLABLE SEE and a digit 5?

There is no ambiguity.

But there is a problem. What if a persons (an Ethiopian?) *wants* to write U+1234 ETHIOPIC SYLLABLE SEE and a digit 5 as a 2 character identifier? You really expect someone to tranlate '5' into 'U+00xx'?

Just like oct escapes, the longest valid sequence (up to the maximum) would be used. If you used the shortest, then there would be no way to specify 5 or 6 digit sequences.

As I said above, there is no ambiguity in the standard because they do not jam codepoints (with or without 'U+') together without non-alphanumeric delimiters. -- Terry Jan Reedy

On Sun, Jul 28, 2013 at 12:14 AM, Greg Ewing wrote:

Steven D'Aprano wrote:

...

Aside: you keep writing H..HHHHHH for Unicode code points. Unicode code points go up to hex 10FFFF,

They do *now*, but we can't be sure that they will stay that way in the future.

They will for as long as UTF-16 is supported. Really, it would have been better all round if UTF-16 had never existed, and everyone just had to switch up to UTF-32; sure, memory would have been wasted, but concepts like PEP 393 would have been devised to deal with that, and we wouldn't have stupid bugs in 99% of programming languages. ChrisA

On 28 Jul 2013 10:34, "Andrew Barnert" wrote:

On Jul 28, 2013, at 1:18, Chris Angelico wrote:

...

On Sun, Jul 28, 2013 at 12:14 AM, Greg Ewing wrote:

...

Steven D'Aprano wrote:

...

Aside: you keep writing H..HHHHHH for Unicode code points. Unicode

...

...

...

points go up to hex 10FFFF,

They do *now*, but we can't be sure that they will stay that way in the future.

They will for as long as UTF-16 is supported. Really, it would have been better all round if UTF-16 had never existed, and everyone just had to switch up to UTF-32; sure, memory would have been wasted, but concepts like PEP 393 would have been devised to deal with that, and we wouldn't have stupid bugs in 99% of programming languages.

UTF-16 wouldn't have been a problem if it weren't almost compatible with UCS2, allowing all kinds of Unicode 1.0 software to misleadingly claim Unicode 2.0 support. (For example, for a long time, both Windows and Java "supported" UTF-16 by treating surrogate pairs as two characters instead of one, which is like "supporting" UTF-8 by treating it like ASCII--except

code that the bugs are much less likely to hit developers early in the cycle.) There are use cases for which UTF-16 is perfectly reasonable. For example, strings with lots of BMP CJK characters and an occasional non-BMP character aren't helped by PEP 393, or by UTF-8, but they are helped by UTF-16. (So long as you can rely on software not treating it as UCS2…) But anyway, this is pretty far off topic.

Unicode could go past 10FFFF without dropping UTF-16, either by adding

more surrogate pair ranges, or by adding surrogate triplets. It's really no different from extending UTF-8, which is no problem.

The problem is that we have no way to predict how they will extend

UTF-16, UTF-8, or code point notation if that ever happens. Assuming that the max length for a code point is six nibbles does sound like assuming nobody will ever need more than 640k characters. The idea of enhancing name based lookup by accepting the "U+" prefix as specifying a code point sounds good to me. It's already a delimited notation, doesn't require a new escape and, as someone else pointed out, allows \N to be used consistently, even if a code point doesn't have a name yet. Cheers, Nick.

_______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas

On 28/07/13 10:30, Andrew Barnert wrote:

Unicode could go past 10FFFF without dropping UTF-16, either by adding more surrogate pair ranges, or by adding surrogate triplets. It's really no different from extending UTF-8, which is no problem.

The problem is that we have no way to predict how they will extend UTF-16, UTF-8, or code point notation if that ever happens. Assuming that the max length for a code point is six nibbles does sound like assuming nobody will ever need more than 640k characters.

The Unicode Consortium formally guarantees stability of the character range U+0000 - U+10FFFF. http://www.unicode.org/faq/utf_bom.html#utf16-6 -- Steven

On Sun, Jul 28, 2013 at 4:57 AM, Steven D'Aprano wrote:

On 28/07/13 10:30, Andrew Barnert wrote:

...

Unicode could go past 10FFFF without dropping UTF-16, either by adding more surrogate pair ranges, or by adding surrogate triplets. It's really no different from extending UTF-8, which is no problem.

The problem is that we have no way to predict how they will extend UTF-16, UTF-8, or code point notation if that ever happens. Assuming that the max length for a code point is six nibbles does sound like assuming nobody will ever need more than 640k characters.

The Unicode Consortium formally guarantees stability of the character range U+0000 - U+10FFFF.

http://www.unicode.org/faq/utf_bom.html#utf16-6

And to add to this: Surrogate triplets would majorly break one of the fundamentals of UTF-16, namely that it guarantees synchronizability. You can look at any 16-bit code unit and know whether it's a lead or trail surrogate. (Obviously if you write to a file or other byte stream, you have to have some out-of-band way to synchronize on bytes, that's separate.) So there's unlikely ever to be a scheme that extends UTF-16 to more characters. UTF-8 can in theory handle longer codes (and some encoders can simply use the same mathematical technique to encode numbers larger than 10FFFF, as we've already seen). The only way would be to declare UTF-16 as a flawed system, just as UCS-2 is. It's a system that can encode only the first planes of Unicode. I doubt it'll ever happen, though, as there's no need for more space. ChrisA

Steven D'Aprano writes:

if you're writing Ethiopian text, I don't think it is actually very likely that you will want to follow ETHIOPIC SYLLABLE SEE by a Latin digit 5 with no separator between them. Yes, it "might" happen,

If you're writing Ethiopic text, I doubt you'll be using escape sequences to denote Ethiopic characters in the first place. I think it's hard to predict how these sequences are going to be used in the future. What I would worry about it not whether writers would "want" to use such sequences, but whether they'll bother to clean them up if they occur in the first place. The writer knows what she wants; it's the reader who has to parse the resulting mess.

(Sorry, I have forgotten who made that suggestion originally.) That could be extended to allow multiple space-separated code points:

\N{U+xxxx U+yyyy U+zzzzz}

or

\N{U+xxxx yyyy zzzzz}

This is a modal encoding, which has proved to be a really bad idea in its past incarnations. I hope that extension is never added to Python.

On 28/07/13 17:41, Stephen J. Turnbull wrote:

...

(Sorry, I have forgotten who made that suggestion originally.) That could be extended to allow multiple space-separated code points:

\N{U+xxxx U+yyyy U+zzzzz}

or

\N{U+xxxx yyyy zzzzz}

This is a modal encoding, which has proved to be a really bad idea in its past incarnations. I hope that extension is never added to Python.

Could you elaborate please? What do you mean "modal encoding", and what past incarnations are you referring to? -- Steven

Steven D'Aprano writes:

On 28/07/13 17:41, Stephen J. Turnbull wrote:

...

...

(Sorry, I have forgotten who made that suggestion originally.) That could be extended to allow multiple space-separated code points:

\N{U+xxxx U+yyyy U+zzzzz}

or

\N{U+xxxx yyyy zzzzz}

This is a modal encoding, which has proved to be a really bad idea in its past incarnations. I hope that extension is never added to Python.

Could you elaborate please? What do you mean "modal encoding", and what past incarnations are you referring to?

A "modal encoding" is one in which the same combination of code units (here, ASCII characters) is interpreted differently depending on arbitrarily distant context. One only has to look at certain web pages or mail messages to see similar encodings (SGML numeric character entities, quoted-printable encoding of text using non-Latin character sets) abused to represent many lines of text. In such (ab)uses, it's very easy to corrupt the whole stream accidentally by losing one of the braces or by interpolating text encoded differently. Sure, it's easy for humans to recognize what's going on, and recover, when they encounter corrupted text interactively, but this is obviously not a convention that's intended for interactive human use! The main past incarnation is the ISO 2022 family. I see no advantage in "readability" of "\N{U+xxxx U+yyyy U+zzzzz}" or "\N{U+xxxx yyyy zzzzz}" over "\N{U+xxxx}\N{U+yyyy}\N{U+zzzzz}", and very little space savings. Worst, it violates the basic understanding that "\N{...}" is the name of one character or code point.

Nick Coghlan writes:

It doesn't bother me that much personally, especially if it was a general comma delimited capability that also worked with other code point names,

I think it should bother you, though. It's not a problem for Python core developers, it's true. Similarly, ISO 2022 was a great idea in theory, and works fine for communication of text over streams. The problem is when you want to embed that stream in some higher-level protocol. So, for example, the original space-separated syntax breaks one-argument split-string, while your comma-separated version breaks CSV. You could fix both of those by using no separator and simply finishing the current code point on encountering "U+" or "}", but I doubt anybody would find that variant appealing. Now, for program literals this isn't going to matter because a string will be converted to internal representation by the compiler, and the program never sees that syntax. But what about applications like web frameworks which often eval client-supplied strings? I hope we are not going to recommend they eval them before validating them!<wink/>

but my inclination is to call YAGNI on the additional complexity.

"Using 'complexity' to refer to this syntax isn't really valid though - what it is, is 'complicated'."<wink/>

Using "modal encoding" to refer to that change isn't really valid though

No, it's quite correct, at least in ISO-land. There, a modal encoding is one which must maintain state across *code points*. The single- code-point "\N" syntax needs to maintain state across *code units*, but when it's done with a code *point*, it's done - there's no state to worry about before starting to parse the next one. By your definition, UTF-8 is modal, but that doesn't seem a very useful categorization to me.

On 28/07/13 23:06, Nick Coghlan wrote:

It would also be more consistent if unicodedata.lookup() was updated to handle numeric code point names. Something like:

...

...

...

import unicodedata def enhanced_lookup(name): ... if name.startswith("U+"): ... return chr(int(name[2:], 16)) ... return unicodedata.lookup(name) ... enhanced_lookup("GREEK SMALL LETTER ALPHA") 'α' enhanced_lookup("U+03B1") 'α'

Earlier, MRAB suggested that unicodedata.name() could return the U+ code point in the case of unnamed characters. I think it would be better to have a separate unicodedata function to return the code point, and leave the current behaviour of name() alone. def codepoint(c): return 'U+{:04X}'.format(ord(c)) This should always succeed for any character. -- Steven

Steven D'Aprano writes:

Earlier, MRAB suggested that unicodedata.name() could return the U+ code point in the case of unnamed characters. I think it would be better to have a separate unicodedata function to return the code point, and leave the current behaviour of name() alone.

His point, and I agree, is that it's not useful to have name() error, as it does for unicodedata.name(chr(65535)). In that case I would prefer that it return "U+FFFF NOT A CHARACTER" or something like that. And for chr(65535*2) it would return "U+1FFFE UNASSIGNED IN VERSION ". Similarly for unassigned private use area code points and surrogates (with their blocks being mentioned). It would be nice if assigned private use area code points could have names added to the database. If a private use character wasn't named, it could have its name algorithmically determined as "U+XXXX PRIVATE USE: UNNAMED".

def codepoint(c): return 'U+{:04X}'.format(ord(c))

This should always succeed for any character.

Or code point: it will succeed for things that aren't characters, such as chr(65535). As one-liners go, this does seem a reasonable candidate for the stdlib. Steve

I wonder if this should also support the special labels for characters without names: control-NNNN reserved-NNNN noncharacter-NNNN private-use-NNNN surrogate-NNNN see p. 138 of http://www.unicode.org/versions/Unicode6.2.0/ch04.pdf I would think that unicodedata.name should not return these, but perhaps unicodedata.lookup should accept them. Note that the doc says that these are frequently displayed enclosed in <>, so perhaps unicodedata.lookup('U+0001') == unicodedata.lookup('control-0001') == unicodedata.lookup('<control-0001>') == '\x01' --- Bruce I'm hiring: http://www.cadencemd.com/info/jobs Latest blog post: Alice's Puzzle Page http://www.vroospeak.com Learn how hackers think: http://j.mp/gruyere-security

On Thu, Aug 1, 2013 at 6:09 PM, Terry Reedy wrote:

Why would someone write 'control-' instead of 'U+'?

Because this is the recommended way to form the code-point labels: "For each code point type without character names, code point labels are constructed by using a lowercase prefix derived from the code point type, followed by a hyphen-minus and then a 4- to 6-digit hexadecimal representation of the code point." "To avoid any possible confusion with actual, non-null Name property values, constructed Unicode code point labels are often displayed between angle brackets: <control-0009>, <noncharacter-FFFF>, and so on. This convention is used consistently in the data files for the Unicode Character Database." "A constructed code point label is distinguished from the designation of the code point itself (for example, “U+0009” or “U+FFFF”), which is also a unique identifier, as described in Appendix A, Notational Conventions." < http://www.unicode.org/versions/Unicode6.2.0/ch04.pdf> I would rather see unicodedata.lookup() to be extended to accept code-point labels rather than "the designation of the code point itself." The same applies to \N escape: I would rather see \N{control-NNNN} or \N{surrogate-NNNN} in string literals than some mysterious \N{U+NNNN}.

On Thu, Aug 1, 2013 at 7:20 PM, Nick Coghlan wrote:

I'd never even heard of code point labels before this thread, while the "U+" notation is incredibly common.

Nick, Did you see this part: "A constructed code point label is distinguished from the designation of the code point itself (for example, “U+0009” or “U+FFFF”), which is also a unique identifier"? The purpose of unicode.lookup() is to look up the unicode code point by name and "U+NNNN" is not a name - it is "the designation of the code point itself." There is no need to look up anything if you want to process an occasional s = "U+FFFF" string: chr(int(s[2:], 16) ) will do the job. The original proposal was to allow \U+NNNN escape as a shortcut for \U0000NNNN. This is a clear readability improvement while \N{U+001B}, for example, is not an improvement over \N{ESCAPE}. However, for more obscure control characters, \N{control-NNNN} may be clearer than any currently available spelling. For example, \N{control-001E} is easier to understand than \036, \x1e, \u001E, \N{RS} or even the most verbose \N{INFORMATION SEPARATOR TWO}.

On Thu, Aug 1, 2013 at 8:04 PM, Bruce Leban wrote:

I'm +/-0 on having 'control-' and 'reserved-' etc. simply being different spellings of 'U+' so that '\N{control-0021}' == '\N{U+0021}' == '\x21' == '!' even though that isn't a control character.

This misses the point of adding the code point type prefix. If you fat-finger \N{control-0021} instead of intended \N{control-0012} you would want a quick syntax error rather than an obscure bug. Similarly, when you are reading someone else's code, you don't want to consult the code table every time you see \N{control-NNNN} to assure that this is really a control character rather than a surrogate- or private-use- one.

Alexander Belopolsky writes:

On Thu, Aug 1, 2013 at 8:04 PM, Bruce Leban wrote:

...

I'm +/-0 on having 'control-' and 'reserved-' etc. simply being different spellings of 'U+' so that '\N{control-0021}' == '\N{U+0021}' == '\x21' == '!' even though that isn't a control character.

This misses the point of adding the code point type prefix.

Not really. That would just pass the responsibility for enforcing consistency to linters, instead of the translator. You can't just make this a syntax error because a code point may be reserved one Python version and a letter in another, depending on which versions of the Unicode tables are being used by those versions of Python. That would conflict with Unicode itself, which says that unknown code points must be treated as characters. This is way too fragile to be allowed to cause syntax errors.

 If you fat-finger \N{control-0021} instead of intended \N{control-0012} you would want a quick syntax error rather than an obscure bug.  Similarly, when you are reading someone else's code, you don't want to consult the code table every time you see \N{control-NNNN} to assure that this is really a control character rather than a surrogate- or private-use- one.

+0 on Bruce's idea, -1 on syntax errors It might be on rare occasions be useful to be strict about fixed-for- all-time types like surrogate and private use. (But even those weren't fixed for all time in the past!) Really, this is an editor or linter function.

On Thu, Aug 1, 2013 at 9:15 PM, Stephen J. Turnbull wrote:

Alexander Belopolsky writes:

...

On Thu, Aug 1, 2013 at 8:04 PM, Bruce Leban wrote: .. This misses the point of adding the code point type prefix.

Not really. That would just pass the responsibility for enforcing consistency to linters, instead of the translator.

I have not seen a linter yet that would suggest that "\x41" should be written as "A". The choice of the best literal syntax requires human judgement. A linter cannot tell you when 1.00 is better than 1.0 or 1. I would choose a more verbose \N{control-NNNN} over shorter \uNNNN when I want to make it obvious to the human reader of my code that I use a control character rather than anything else.

You can't just make this a syntax error because a code point may be reserved one Python version and a letter in another, depending on which versions of the Unicode tables are being used by those versions of Python.

That's true, but why would you write \N{reserved-NNNN} instead of \uNNNN to begin with? I would assume you would only choose a longer spelling when it is important for your program that you use a reserved character and your program will not work correctly with the UCD version where the NNNN code point is assigned.

That would conflict with Unicode itself, which says that unknown code points must be treated as characters. This is way too fragile to be allowed to cause syntax errors.

You can always avoid syntax errors by using \uNNNN. If you choose to specify the character type you hopefully do it for a good reason.

..

It might be on rare occasions be useful to be strict about fixed-for- all-time types like surrogate and private use.

There are only five type prefixes: control-, reserved-, non-character-, private-use-, and surrogate-. With the possible exception or reserved-, on a rare occasion when you want to be explicit about the character type, it is useful to be strict. In case of reserved-, I cannot think of any legitimate use for a reserved character in a string literal, so if strictness is a problem in this case, I would disallow \N{reserved-NNNN} altogether.

(But even those weren't fixed for all time in the past!)

Now they are: control- property is immutable since version 1.1.5, surrogate- and private-use- since 2.0, and noncharacter- since 3.1.0. (See http://www.unicode.org/policies/stability_policy.html.) Moreover, since 2.1.0, "The enumeration of General_Category property values is fixed. No new values will be added."

Alexander Belopolsky writes:

On Thu, Aug 1, 2013 at 9:46 PM, MRAB wrote:

...

We could follow Perl or Ruby, or both of them, or even allow braces with any of the hex escapes.

That choice is unfortunately precluded by backwards compatibility because both "\u1FFFF" and "\x1FFFF" are valid strings. (Are braces optional in Perl's \x{..} or Ruby's \u{..}?) Also, the upper-case U is more in-line with U+ notation and \N escape.  If we are looking for "one obvious way," I think it should be \U with \x and \u remaining the other less obvious ways.

-1. The obvious way forward is \N{U+1FFFF}. That *looks* like an algorithmically generated name, and (wow!) that's what it *is*.[1] The existing \U, \u, and \x escapes are fine as they are. They can't really be deprecated because they're needed for portability to older Python versions which won't have any of the proposed extensions. Changing the syntax of \U to allow braces with a variable-width hexadecimal argument is only a minor compatibility break, but please have pity on the folks who support python-list. They'll forever be dealing with questions like "I know I've seen other people write '\U3bb', why do I get a weird syntax error?" and "I use Python 3.3. Why do I get a syntax error with '\U{3BB}'?" On the other hand, \N{U+1FFFF} will currently get a lookup failure. I think that's OK, since currently code needs to be prepared for that to fail anyway since it raises an error, and users will be used to it because it's easy to typo Unicode names when typing from memory -- they're pretty regular but not 100% so. Footnotes: [1] Of course, it's also an invalid code point in any Unicode stream. ;-)

On Thu, Aug 1, 2013 at 11:30 PM, Stephen J. Turnbull wrote:

-1. The obvious way forward is \N{U+1FFFF}. That *looks* like an algorithmically generated name, and (wow!) that's what it *is*.[1]

The only problem is that this is not a conforming name according to the Unicode standard. The standard is very explicit in its recommendation on how the names should be generated: "Use in APIs. APIs which return the value of a Unicode “character name” for a given code point might vary somewhat in their behavior. An API which is defined as strictly returning the value of the Unicode Name property (the “na” attribute), should return a null string for any Unicode code point other than graphic or format characters, as that is the actual value of the property for such code points. On the other hand, an API which returns a name for Unicode code points, but which is expected to provide useful, unique labels for unassigned, reserved code points and other special code point types, should return the value of the Unicode Name property for any code point for which it is non-null, but should otherwise construct a code point label to stand in for a character name." The recommendation on what should be accepted as a valid name is more relaxed: "... it can be more effective for a user interface to use names that were translated or otherwise adjusted to meet the expectations of the targeted user community. By also listing the formal character name, a user interface could ensure that users can unambiguously refer to the character by the name documented in the Unicode Standard." This does not literally preclude treating U+NNNN as a character name, but it looks like such use is discouraged: "A constructed code point label is distinguished from the designation of the code point itself (for example, “U+0009” or “U+FFFF”), which is also a unique identifier."

[1] Of course, it's also an invalid code point in any Unicode stream. ;-)

This is not accurate. U+1FFFF is a valid code point and its generated label is <noncharacter-1FFFF>. Noncharacters "are forbidden for use in open interchange of Unicode text data. ... Applications are free to use any of these noncharacter code points internally but should never attempt to exchange them." (See Chapter 16.7 Noncharacters.) In Python 0x1FFFF is a valid code point:

...

...

chr(0x1FFFF) '\U0001ffff'

An application written in Python can use strings containing '\U0001ffff' internally, but should not interchange such strings with other applications.

Alexander Belopolsky writes:

On Thu, Aug 1, 2013 at 11:30 PM, Stephen J. Turnbull wrote:

...

-1.  The obvious way forward is \N{U+1FFFF}.  That *looks* like an algorithmically generated name, and (wow!) that's what it *is*.

The only problem is that this is not a conforming name according to the Unicode standard. The standard is very explicit in its recommendation on how the names should be generated: "Use in APIs. APIs which return the value of a Unicode “character name” [...]

This whole section of the standard is irrelevant. Of course unicodedata.name('A') should *return* 'LATIN CAPITAL LETTER A', but we're discussing the possibility of extending what unicodedata.lookup() should *accept*.

The recommendation on what should be accepted as a valid name is more relaxed: "... it can be more effective for a user interface to use names that were translated or otherwise adjusted to meet the expectations of the targeted user community."

It seems to me that's exactly what those of us who advocate using \N{} are saying.

This does not literally preclude treating U+NNNN as a character name, but it looks like such use is discouraged: "A constructed code point label is distinguished from the designation of the code point itself (for example, “U+0009” or “U+FFFF”), which is also a unique identifier."

I don't see any such implication. What's being said here is that an application should not expect a conforming implementation to treat "U+0009" and "control-0009" identically in all respects. For example, "control-0009" might be subjected to the kind of consistency check you want. Or only one of the two might be acceptable to a name lookup function. Or you might have to use different functions to convert them to characters. Steve