On 2013-10-08, at 13:43 , Serhiy Storchaka wrote:

08.10.13 14:38, Masklinn написав(ла):

...

On 2013-10-08, at 13:17 , Serhiy Storchaka wrote:

...

Here is an idea about adding a mark to PyUnicode object which allows fast answer to the question if a string has surrogate code. This mark has one of three possible states:

* String doesn't contain surrogates. * String contains surrogates. * It is still unknown.

We can combine this with "is_ascii" flag in 2-bit value:

* String is ASCII-only (and doesn't contain surrogates). * String is not ASCII-only and doesn't contain surrogates. * String is not ASCII-only and contains surrogates. * String is not ASCII-only and it is still unknown if it contains surrogate.

Isn't that redundant with the kind under shortest form representation?

No, it isn't redundant. '\udc80' is UCS2 string with surrogate code, and '\udc80\U00010000' is UCS4 string with surrogate code.

I don't know the details of the flexible string representation, but I believed the names fit what was actually in memory. UCS2 does not have surrogate pairs, thus surrogate codes make no sense in UCS2, they're a UTF-16 concept. Likewise for UCS4. Surrogate codes are not codepoints, they have no reason to appear in either UCS2 or UCS4 outside of encoding errors.

UCS2 string without surrogate codes can be encoded in UTF-16 by memcpy().

Surrogate codes prevent that (modulo objections above) for slicing (not that it's a big issue I think, a guard can just check whether it's slicing within a surrogate pair, that only requires checking the first and last 2 bytes of the range) but not for concatenation right?

On 2013-10-08, at 15:02 , Steven D'Aprano wrote: [snipped early part as any response would be superseded by or redundant with the stuff below]

However, you cannot encode single surrogates to UTF-8:

py> surr.encode('utf-8') Traceback (most recent call last): File "<stdin>", line 1, in <module> UnicodeEncodeError: 'utf-8' codec can't encode character '\udc80' in position 0: surrogates not allowed

as per the standard:

http://www.unicode.org/faq/utf_bom.html#utf8-5

I *think* you are supposed to be able to encode surrogate *pairs* to UTF-8, if I'm reading the FAQ correctly

I'm reading the opposite, from http://www.unicode.org/faq/utf_bom.html#utf8-4:

there is a widespread practice of generating pairs of three byte sequences in older software, especially software which pre-dates the introduction of UTF-16 or that is interoperating with UTF-16 environments under particular constraints. Such an encoding is not conformant to UTF-8 as defined.

Pairs of 3-byte sequences would be encoding each surrogate directly to UTF-8, whereas a single 4-byte sequence would be decoding the surrogate pair to a codepoint and encoding that codepoint to UTF-8. My reading of the FAQ makes the second interpretation the only valid one. So you can't encode surrogates (either lone or paired) to UTF-8, you can encode the codepoint encoded by a surrogate pair.

In any case, it is certainly legal to have Unicode strings containing non-characters, including surrogates, and you can encode them to UTF-16 and −32.

The UTF-32 section has similar note to UTF-8: http://www.unicode.org/faq/utf_bom.html#utf32-7

A: If an unpaired surrogate is encountered when converting ill-formed UTF-16 data, any conformant converter must treat this as an error. By representing such an unpaired surrogate on its own, the resulting UTF-32 data stream would become ill-formed. While it faithfully reflects the nature of the input, Unicode conformance requires that encoding form conversion always results in valid data stream.

and the UTF-16 section points out: http://www.unicode.org/faq/utf_bom.html#utf16-7

Q: Are there any 16-bit values that are invalid?

A: Unpaired surrogates are invalid in UTFs. These include any value in the range D80016 to DBFF16 not followed by a value in the range DC0016 to DFFF16, or any value in the range DC0016 to DFFF16 not preceded by a value in the range D80016 to DBFF16.

As far as I can read the FAQ, it is always invalid to encode a surrogate, surrogates are not to be considered codepoints (they're not just noncharacters[0], noncharacters are codepoints), and a lone surrogate in a UTF-16 stream means the stream is corrupted, which should result in an error during transcoding to anything (unless some recovery mode is used to replace corrupted characters by some mark during decoding I guess).

So... I'm not sure why this will be useful. Presumably Unicode strings containing surrogate code points will be rare

And they're a sign of corrupted stream. The FAQ reads a bit strangely, I think because it's written from the viewpoint that the "internal encoding" will be UTF-16, and UTF-8 and UTF-32 are transcoding from that. Which does not apply to CPython and the FSR. Parsing the FAQ with that viewpoint, I believe a CPython string (unicode) must not contain surrogate codes: a surrogate pair should have been decoded from UTF-16 to a codepoint (then identity-encoded to UCS4) and a single surrogate should have been caught by the UTF-16 decoder and should have triggered the error handler at that point. A surrogate code in a CPython string means the string is corrupted[1]. Surrogates *may* appear in binary data, while building a UTF-16 bytestream by hand. [0] since "noncharacter" has a well-defined meaning in unicode, and only applies to 66 codepoints, a much smaller range than surrogates: http://www.unicode.org/faq/private_use.html#noncharacters [1] note that this hinges on my understanding of "UCS2" in FSR being actual UCS2, if it's UCS2-with-surrogates with a heuristic for switching between UCS2 and UCS4 depending on the number of surrogate pairs in the string it does not apply

On 2013-10-08, at 16:20 , Steven D'Aprano wrote

I'd actually be disappointed if that were the case; I think that would be a poor design. But if that's what the Unicode standard demands, Python ought to support it.

That would be really weird, it'd mean an *encoder* has to translate a surrogate pair into the actual codepoint in some sort of weird UTF-specific normalization pass.

But hopefully somebody will explain to me why my interpretation is wrong :-)

[...]

...

The FAQ reads a bit strangely, I think because it's written from the viewpoint that the "internal encoding" will be UTF-16, and UTF-8 and UTF-32 are transcoding from that. Which does not apply to CPython and the FSR.

Hmmm... well, that might explain it. If it's written by Java programmers for Java programmers, they may very well decide that having spent 20 years trying to convince people that string != ASCII, they're now going to convince them that string == UTF-16 instead :/

To be fair, it's not just java programmers, IIRC ICU uses UTF-16 as the internal encoding.

...

Parsing the FAQ with that viewpoint, I believe a CPython string (unicode) must not contain surrogate codes: a surrogate pair should have been decoded from UTF-16 to a codepoint (then identity-encoded to UCS4) and a single surrogate should have been caught by the UTF-16 decoder and should have triggered the error handler at that point. A surrogate code in a CPython string means the string is corrupted[1].

I think that interpretation is a bit strong. I think it would be fair to say that CPython strings may contain surrogates, but you can't encode them to bytes using the UTFs. Nor are there any byte sequences that can be decoded to surrogates using the UTFs.

This essentially means that you can only get surrogates in a string using (e.g.) chr() or \u escapes, and you can't then encode them to bytes using UTF encodings.

...

Surrogates *may* appear in binary data, while building a UTF-16 bytestream by hand.

But there you're talking about bytes, not byte strings. Byte strings can contain any bytes you like :-)

Yes, that's basically what I mean: I think surrogates only make sense in a bytestream, not in a unicode stream. Although I did not remember/was not aware of PEP 383 (thank you Stephen) which makes the Unicode spec irrelevant to what Python string contains. On 2013-10-08, at 16:31 , Stephen J. Turnbull wrote:

Furthermore, there is nothing to stop a Python unicode from containing any code unit (including both surrogates and other non-characters like 0xFFFF). Checking of the rules you cite is done by codecs, at encoding and decoding time.

noncharacters are a very different case for what it's worth, their own FAQ clearly notes that they are valid full-fledged codepoints and must be encoded and preserved by UTFs: http://www.unicode.org/faq/private_use.html#nonchar7

Bruce Leban wrote:

The FAQ is explicit that this is wrong: "The definition of UTF-8 requires that supplementary characters (those using surrogate pairs in UTF-16) be encoded with a single four byte sequence." http://www.unicode.org/faq/utf_bom.html#utf8-4

Python's internal string representation is not UTF-16, though, so this doesn't apply directly. Seems to me it hinges on whether a pair of surrogate code points appearing in a Python string are meant to represent a single character or not. I would say not, because otherwise they would have been stored as a single code unit. -- Greg

On Tue, Oct 08, 2013 at 01:37:54PM -0700, Bruce Leban wrote:

On Tue, Oct 8, 2013 at 7:20 AM, Steven D'Aprano <steve@pearwood.info> wrote:

...

Given:

c = '\N{LINEAR B SYLLABLE B038 E}' # \U00010001 c.encode('utf-8') => b'\xf0\x90\x80\x81'

and:

c.encode('utf-16BE') # encodes as a surrogate pair => b'\xd8\x00\xdc\x01'

then those same surrogates, taken as codepoints, should be encodable as UTF-8:

'\ud800\udc01'.encode('utf-8') => b'\xf0\x90\x80\x81'

I'd actually be disappointed if that were the case; I think that would be a poor design. But if that's what the Unicode standard demands, Python ought to support it.

The FAQ is explicit that this is wrong: "The definition of UTF-8 requires that supplementary characters (those using surrogate pairs in UTF-16) be encoded with a single four byte sequence." http://www.unicode.org/faq/utf_bom.html#utf8-4

And if you count the number of bytes, you will see four of them: '\ud800\udc01'.encode('utf-8') => b'\xf0' b'\x90' b'\x80' b'\x81' I stress that Python 3.3 doesn't actually do this, but my reading of the FAQ suggests that it should. The question isn't what UTF-8 should do with supplmentary characters (those outside the BMP). That is well-defined, and Python 3.3 gets it right. The question is what it should do with pairs of surrogates. Ill-formed surrogates are rightly illegal when encoding to UTF-8: # a lone surrogate is illegal '\ud800'.encode('utf-8') must be treated as an error # two high surrogates, or two low surrogates '\udc01\udc01'.encode('utf-8') must be treated as an error '\ud800\ud800'.encode('utf-8') must be treated as an error # if they're in the wrong order '\udc01\ud800'.encode('utf-8') must be treated as an error The only thing that I'm not sure is how to deal with *valid* pairs of surrogates: '\ud800\udc01'.encode('utf-8') should do what? I personally would hope that this too should raise, which is Python's current behaviour, but my reading of the FAQs is that it should be treated as if there were an implicit UTF-16 conversion. (I hope I'm wrong!) That is: 1) treat the sequence of code points as if it were a sequence of two 16-bit values b'\xd8\x00' b'\xdc\x01' 2) implicitly decode it using UTF-16 to get U+10001 3) encode U+10001 using UTF-8 to get b'\xf0\x90\x80\x81' That would be (in my opinion) *horrible*, but that's my reading of the Unicode FAQ. The question asks: "How do I convert a UTF-16 surrogate pair such as <D800 DC00> to UTF-8?" and the answer seems to be: "The definition of UTF-8 requires that supplementary characters (those using surrogate pairs in UTF-16) be encoded with a single four byte sequence." which doesn't actually answer the question (the question is about SURROGATE PAIRS, the answer is about SUPPLEMENTARY CHARACTERS) but suggests the above horrible interpretation. What I'm hoping for is a definite source that explains what the UTF-8 encoder is supposed to do with a Unicode string containing surrogates. (And presumably the other UTF encoders as well, although I haven't tried thinking about them yet.)

It goes on to say that there is a widespread practice of doing it anyway in older software. Therefore, it might be acceptable to accept these mis-encoded characters when *decoding* but they should never be generated when *encoding*.

They are talking about the practice of generating six bytes, two three-byte sequences. You should notice that I'm not generating six bytes anywhere. -- Steven

On 10/8/2013 8:55 PM, Steven D'Aprano wrote:

'\ud800\udc01'.encode('utf-8') => b'\xf0' b'\x90' b'\x80' b'\x81'

I stress that Python 3.3 doesn't actually do this, but my reading of the FAQ suggests that it should.

And I already explained on python-list why that reading is wrong; transcoding a utf-16 string (sequence of 2-byte words, subject to validity rules) is different from encoding unicode text (character sequence, and surrogates are not characters). A utf-16 to utf-8 transcoder should (must) do the above, but in 3.3+, the utf-8 codec is no longer the utf-16 trancoder that it effectively was for narrow builds. Each utf form defines a one to one mapping between unicode texts and valid code unit sequences. (Unicode Standard, Chapter 3, definition D79.) Having both '\U00010001' and '\ud800\udc01' map to b'\xf0\x90\x80\x81' would violate that important property. '\ud800\udc01' represents a character in utf-16 but not in python's flexible string representation. The latter uses one code unit (of variable size per string) per character, instead of a variable number of code units (of one size for all strings) per character. Because machines have not conceptual, visual, or aural memory, but only byte memory, they must re-encode abstract characters to bytes to remember them. In pre 3.3 narrow builds, where utf-16 was used internally, decoding and encoding amounted to transcoding bytes encodings into the utf-16 encoding, and vice versa. So utf-8 b'\xf0\x90\x80\x81' and utf-16 '\ud800\udc01' were mapped into each other. Whether the mapping was done directly or indirectly, via the character codepoint value, did not matter to the user. In any case FSR no longer uses multiple-code-unit encodings internally, and '\ud800\udc01', even though allowed for practical reasons, does not represent and is not the same as '\U00010001'. The proposed 'has_surrogates' flag amounts to an 'not strictly valid' flag. Only the FSR implementors can decide if it is worth the trouble. -- Terry Jan Reedy

Masklinn writes:

The FAQ reads a bit strangely, I think because it's written from the viewpoint that the "internal encoding" will be UTF-16, and UTF-8 and UTF-32 are transcoding from that. Which does not apply to CPython and the FSR.

No, it's written from the viewpoint that it says *nothing* about internal encodings, only about the encodings used in interchange of textual data, and about certain aspects of the processes that may receive and generate such data (eg, when data matches a Unicode regular expression, or how bidirectional text should appear visually).

Parsing the FAQ with that viewpoint, I believe a CPython string (unicode) must not contain surrogate codes:

No, it says no such thing. All the Unicode Standard (and the FAQ) says is that if Python generates output that purports to be text encoded in Unicode, it may not contain surrogate codes except where those codes are used according to UTF-16 to encode characters in planes 2 to 17, and if it receives data alleged to be Unicode in some transformation format, it must raise an error if it receives surrogates other than a correctly formed surrogate pair in text known to be encoded as UTF-16. In fact (as I wrote before without proper citation), the internal encoding of Python has been extended by PEP 383 to use a subset of the surrogate space to represent undecodable bytes in an octet stream, when the error handler is set to "surrogateescape". Furthermore, there is nothing to stop a Python unicode from containing any code unit (including both surrogates and other non-characters like 0xFFFF). Checking of the rules you cite is done by codecs, at encoding and decoding time.

Masklinn writes:

I don't know the details of the flexible string representation, but I believed the names fit what was actually in memory. UCS2 does not have surrogate pairs, thus surrogate codes make no sense in UCS2, they're a UTF-16 concept. Likewise for UCS4. Surrogate codes are not codepoints, they have no reason to appear in either UCS2 or UCS4 outside of encoding errors.

True, but Python doesn't actually use UCS2 or UCS4 internally. It uses UCS2 or UCS4 plus a row of codes from the surrogate area to represent undecodable bytes. This feature is optional (enabled by using the appropriate error= setting in the codec), but I don't suppose it's going to go away.

08.10.13 14:58, M.-A. Lemburg написав(ла):

I guess you could use one bit from the kind structure for that:

The kind of string should be equal to the size of character unit. This assumption is used in a lot of code.

On 2013-10-08, at 13:43 , Serhiy Storchaka wrote:

08.10.13 14:38, Masklinn написав(ла):

...

On 2013-10-08, at 13:17 , Serhiy Storchaka wrote:

...

Here is an idea about adding a mark to PyUnicode object which allows fast answer to the question if a string has surrogate code. This mark has one of three possible states:

* String doesn't contain surrogates. * String contains surrogates. * It is still unknown.

We can combine this with "is_ascii" flag in 2-bit value:

* String is ASCII-only (and doesn't contain surrogates). * String is not ASCII-only and doesn't contain surrogates. * String is not ASCII-only and contains surrogates. * String is not ASCII-only and it is still unknown if it contains surrogate.

Isn't that redundant with the kind under shortest form representation?

No, it isn't redundant. '\udc80' is UCS2 string with surrogate code, and '\udc80\U00010000' is UCS4 string with surrogate code.

I don't know the details of the flexible string representation, but I believed the names fit what was actually in memory. UCS2 does not have surrogate pairs, thus surrogate codes make no sense in UCS2, they're a UTF-16 concept. Likewise for UCS4. Surrogate codes are not codepoints, they have no reason to appear in either UCS2 or UCS4 outside of encoding errors.

UCS2 string without surrogate codes can be encoded in UTF-16 by memcpy().

Surrogate codes prevent that (modulo objections above) for slicing (not that it's a big issue I think, a guard can just check whether it's slicing within a surrogate pair, that only requires checking the first and last 2 bytes of the range) but not for concatenation right?

On 2013-10-08, at 15:02 , Steven D'Aprano wrote: [snipped early part as any response would be superseded by or redundant with the stuff below]

However, you cannot encode single surrogates to UTF-8:

py> surr.encode('utf-8') Traceback (most recent call last): File "<stdin>", line 1, in <module> UnicodeEncodeError: 'utf-8' codec can't encode character '\udc80' in position 0: surrogates not allowed

as per the standard:

http://www.unicode.org/faq/utf_bom.html#utf8-5

I *think* you are supposed to be able to encode surrogate *pairs* to UTF-8, if I'm reading the FAQ correctly

I'm reading the opposite, from http://www.unicode.org/faq/utf_bom.html#utf8-4:

there is a widespread practice of generating pairs of three byte sequences in older software, especially software which pre-dates the introduction of UTF-16 or that is interoperating with UTF-16 environments under particular constraints. Such an encoding is not conformant to UTF-8 as defined.

Pairs of 3-byte sequences would be encoding each surrogate directly to UTF-8, whereas a single 4-byte sequence would be decoding the surrogate pair to a codepoint and encoding that codepoint to UTF-8. My reading of the FAQ makes the second interpretation the only valid one. So you can't encode surrogates (either lone or paired) to UTF-8, you can encode the codepoint encoded by a surrogate pair.

In any case, it is certainly legal to have Unicode strings containing non-characters, including surrogates, and you can encode them to UTF-16 and −32.

The UTF-32 section has similar note to UTF-8: http://www.unicode.org/faq/utf_bom.html#utf32-7

A: If an unpaired surrogate is encountered when converting ill-formed UTF-16 data, any conformant converter must treat this as an error. By representing such an unpaired surrogate on its own, the resulting UTF-32 data stream would become ill-formed. While it faithfully reflects the nature of the input, Unicode conformance requires that encoding form conversion always results in valid data stream.

and the UTF-16 section points out: http://www.unicode.org/faq/utf_bom.html#utf16-7

Q: Are there any 16-bit values that are invalid?

A: Unpaired surrogates are invalid in UTFs. These include any value in the range D80016 to DBFF16 not followed by a value in the range DC0016 to DFFF16, or any value in the range DC0016 to DFFF16 not preceded by a value in the range D80016 to DBFF16.

As far as I can read the FAQ, it is always invalid to encode a surrogate, surrogates are not to be considered codepoints (they're not just noncharacters[0], noncharacters are codepoints), and a lone surrogate in a UTF-16 stream means the stream is corrupted, which should result in an error during transcoding to anything (unless some recovery mode is used to replace corrupted characters by some mark during decoding I guess).

So... I'm not sure why this will be useful. Presumably Unicode strings containing surrogate code points will be rare

And they're a sign of corrupted stream. The FAQ reads a bit strangely, I think because it's written from the viewpoint that the "internal encoding" will be UTF-16, and UTF-8 and UTF-32 are transcoding from that. Which does not apply to CPython and the FSR. Parsing the FAQ with that viewpoint, I believe a CPython string (unicode) must not contain surrogate codes: a surrogate pair should have been decoded from UTF-16 to a codepoint (then identity-encoded to UCS4) and a single surrogate should have been caught by the UTF-16 decoder and should have triggered the error handler at that point. A surrogate code in a CPython string means the string is corrupted[1]. Surrogates *may* appear in binary data, while building a UTF-16 bytestream by hand. [0] since "noncharacter" has a well-defined meaning in unicode, and only applies to 66 codepoints, a much smaller range than surrogates: http://www.unicode.org/faq/private_use.html#noncharacters [1] note that this hinges on my understanding of "UCS2" in FSR being actual UCS2, if it's UCS2-with-surrogates with a heuristic for switching between UCS2 and UCS4 depending on the number of surrogate pairs in the string it does not apply

On 2013-10-08, at 16:20 , Steven D'Aprano wrote

I'd actually be disappointed if that were the case; I think that would be a poor design. But if that's what the Unicode standard demands, Python ought to support it.

That would be really weird, it'd mean an *encoder* has to translate a surrogate pair into the actual codepoint in some sort of weird UTF-specific normalization pass.

But hopefully somebody will explain to me why my interpretation is wrong :-)

[...]

...

The FAQ reads a bit strangely, I think because it's written from the viewpoint that the "internal encoding" will be UTF-16, and UTF-8 and UTF-32 are transcoding from that. Which does not apply to CPython and the FSR.

Hmmm... well, that might explain it. If it's written by Java programmers for Java programmers, they may very well decide that having spent 20 years trying to convince people that string != ASCII, they're now going to convince them that string == UTF-16 instead :/

To be fair, it's not just java programmers, IIRC ICU uses UTF-16 as the internal encoding.

...

Parsing the FAQ with that viewpoint, I believe a CPython string (unicode) must not contain surrogate codes: a surrogate pair should have been decoded from UTF-16 to a codepoint (then identity-encoded to UCS4) and a single surrogate should have been caught by the UTF-16 decoder and should have triggered the error handler at that point. A surrogate code in a CPython string means the string is corrupted[1].

I think that interpretation is a bit strong. I think it would be fair to say that CPython strings may contain surrogates, but you can't encode them to bytes using the UTFs. Nor are there any byte sequences that can be decoded to surrogates using the UTFs.

This essentially means that you can only get surrogates in a string using (e.g.) chr() or \u escapes, and you can't then encode them to bytes using UTF encodings.

...

Surrogates *may* appear in binary data, while building a UTF-16 bytestream by hand.

But there you're talking about bytes, not byte strings. Byte strings can contain any bytes you like :-)

Yes, that's basically what I mean: I think surrogates only make sense in a bytestream, not in a unicode stream. Although I did not remember/was not aware of PEP 383 (thank you Stephen) which makes the Unicode spec irrelevant to what Python string contains. On 2013-10-08, at 16:31 , Stephen J. Turnbull wrote:

Furthermore, there is nothing to stop a Python unicode from containing any code unit (including both surrogates and other non-characters like 0xFFFF). Checking of the rules you cite is done by codecs, at encoding and decoding time.

noncharacters are a very different case for what it's worth, their own FAQ clearly notes that they are valid full-fledged codepoints and must be encoded and preserved by UTFs: http://www.unicode.org/faq/private_use.html#nonchar7

Bruce Leban wrote:

The FAQ is explicit that this is wrong: "The definition of UTF-8 requires that supplementary characters (those using surrogate pairs in UTF-16) be encoded with a single four byte sequence." http://www.unicode.org/faq/utf_bom.html#utf8-4

Python's internal string representation is not UTF-16, though, so this doesn't apply directly. Seems to me it hinges on whether a pair of surrogate code points appearing in a Python string are meant to represent a single character or not. I would say not, because otherwise they would have been stored as a single code unit. -- Greg

On Tue, Oct 08, 2013 at 01:37:54PM -0700, Bruce Leban wrote:

On Tue, Oct 8, 2013 at 7:20 AM, Steven D'Aprano <steve@pearwood.info> wrote:

...

Given:

c = '\N{LINEAR B SYLLABLE B038 E}' # \U00010001 c.encode('utf-8') => b'\xf0\x90\x80\x81'

and:

c.encode('utf-16BE') # encodes as a surrogate pair => b'\xd8\x00\xdc\x01'

then those same surrogates, taken as codepoints, should be encodable as UTF-8:

'\ud800\udc01'.encode('utf-8') => b'\xf0\x90\x80\x81'

I'd actually be disappointed if that were the case; I think that would be a poor design. But if that's what the Unicode standard demands, Python ought to support it.

The FAQ is explicit that this is wrong: "The definition of UTF-8 requires that supplementary characters (those using surrogate pairs in UTF-16) be encoded with a single four byte sequence." http://www.unicode.org/faq/utf_bom.html#utf8-4

And if you count the number of bytes, you will see four of them: '\ud800\udc01'.encode('utf-8') => b'\xf0' b'\x90' b'\x80' b'\x81' I stress that Python 3.3 doesn't actually do this, but my reading of the FAQ suggests that it should. The question isn't what UTF-8 should do with supplmentary characters (those outside the BMP). That is well-defined, and Python 3.3 gets it right. The question is what it should do with pairs of surrogates. Ill-formed surrogates are rightly illegal when encoding to UTF-8: # a lone surrogate is illegal '\ud800'.encode('utf-8') must be treated as an error # two high surrogates, or two low surrogates '\udc01\udc01'.encode('utf-8') must be treated as an error '\ud800\ud800'.encode('utf-8') must be treated as an error # if they're in the wrong order '\udc01\ud800'.encode('utf-8') must be treated as an error The only thing that I'm not sure is how to deal with *valid* pairs of surrogates: '\ud800\udc01'.encode('utf-8') should do what? I personally would hope that this too should raise, which is Python's current behaviour, but my reading of the FAQs is that it should be treated as if there were an implicit UTF-16 conversion. (I hope I'm wrong!) That is: 1) treat the sequence of code points as if it were a sequence of two 16-bit values b'\xd8\x00' b'\xdc\x01' 2) implicitly decode it using UTF-16 to get U+10001 3) encode U+10001 using UTF-8 to get b'\xf0\x90\x80\x81' That would be (in my opinion) *horrible*, but that's my reading of the Unicode FAQ. The question asks: "How do I convert a UTF-16 surrogate pair such as <D800 DC00> to UTF-8?" and the answer seems to be: "The definition of UTF-8 requires that supplementary characters (those using surrogate pairs in UTF-16) be encoded with a single four byte sequence." which doesn't actually answer the question (the question is about SURROGATE PAIRS, the answer is about SUPPLEMENTARY CHARACTERS) but suggests the above horrible interpretation. What I'm hoping for is a definite source that explains what the UTF-8 encoder is supposed to do with a Unicode string containing surrogates. (And presumably the other UTF encoders as well, although I haven't tried thinking about them yet.)

It goes on to say that there is a widespread practice of doing it anyway in older software. Therefore, it might be acceptable to accept these mis-encoded characters when *decoding* but they should never be generated when *encoding*.

They are talking about the practice of generating six bytes, two three-byte sequences. You should notice that I'm not generating six bytes anywhere. -- Steven

On 10/8/2013 8:55 PM, Steven D'Aprano wrote:

'\ud800\udc01'.encode('utf-8') => b'\xf0' b'\x90' b'\x80' b'\x81'

I stress that Python 3.3 doesn't actually do this, but my reading of the FAQ suggests that it should.

And I already explained on python-list why that reading is wrong; transcoding a utf-16 string (sequence of 2-byte words, subject to validity rules) is different from encoding unicode text (character sequence, and surrogates are not characters). A utf-16 to utf-8 transcoder should (must) do the above, but in 3.3+, the utf-8 codec is no longer the utf-16 trancoder that it effectively was for narrow builds. Each utf form defines a one to one mapping between unicode texts and valid code unit sequences. (Unicode Standard, Chapter 3, definition D79.) Having both '\U00010001' and '\ud800\udc01' map to b'\xf0\x90\x80\x81' would violate that important property. '\ud800\udc01' represents a character in utf-16 but not in python's flexible string representation. The latter uses one code unit (of variable size per string) per character, instead of a variable number of code units (of one size for all strings) per character. Because machines have not conceptual, visual, or aural memory, but only byte memory, they must re-encode abstract characters to bytes to remember them. In pre 3.3 narrow builds, where utf-16 was used internally, decoding and encoding amounted to transcoding bytes encodings into the utf-16 encoding, and vice versa. So utf-8 b'\xf0\x90\x80\x81' and utf-16 '\ud800\udc01' were mapped into each other. Whether the mapping was done directly or indirectly, via the character codepoint value, did not matter to the user. In any case FSR no longer uses multiple-code-unit encodings internally, and '\ud800\udc01', even though allowed for practical reasons, does not represent and is not the same as '\U00010001'. The proposed 'has_surrogates' flag amounts to an 'not strictly valid' flag. Only the FSR implementors can decide if it is worth the trouble. -- Terry Jan Reedy

Masklinn writes:

The FAQ reads a bit strangely, I think because it's written from the viewpoint that the "internal encoding" will be UTF-16, and UTF-8 and UTF-32 are transcoding from that. Which does not apply to CPython and the FSR.

No, it's written from the viewpoint that it says *nothing* about internal encodings, only about the encodings used in interchange of textual data, and about certain aspects of the processes that may receive and generate such data (eg, when data matches a Unicode regular expression, or how bidirectional text should appear visually).

Parsing the FAQ with that viewpoint, I believe a CPython string (unicode) must not contain surrogate codes:

No, it says no such thing. All the Unicode Standard (and the FAQ) says is that if Python generates output that purports to be text encoded in Unicode, it may not contain surrogate codes except where those codes are used according to UTF-16 to encode characters in planes 2 to 17, and if it receives data alleged to be Unicode in some transformation format, it must raise an error if it receives surrogates other than a correctly formed surrogate pair in text known to be encoded as UTF-16. In fact (as I wrote before without proper citation), the internal encoding of Python has been extended by PEP 383 to use a subset of the surrogate space to represent undecodable bytes in an octet stream, when the error handler is set to "surrogateescape". Furthermore, there is nothing to stop a Python unicode from containing any code unit (including both surrogates and other non-characters like 0xFFFF). Checking of the rules you cite is done by codecs, at encoding and decoding time.

Masklinn writes:

I don't know the details of the flexible string representation, but I believed the names fit what was actually in memory. UCS2 does not have surrogate pairs, thus surrogate codes make no sense in UCS2, they're a UTF-16 concept. Likewise for UCS4. Surrogate codes are not codepoints, they have no reason to appear in either UCS2 or UCS4 outside of encoding errors.

True, but Python doesn't actually use UCS2 or UCS4 internally. It uses UCS2 or UCS4 plus a row of codes from the surrogate area to represent undecodable bytes. This feature is optional (enabled by using the appropriate error= setting in the codec), but I don't suppose it's going to go away.

08.10.13 14:58, M.-A. Lemburg написав(ла):

I guess you could use one bit from the kind structure for that:

The kind of string should be equal to the size of character unit. This assumption is used in a lot of code.