[Python-Dev] PEP 383: Non-decodable Bytes in System Character Interfaces
google at mrabarnett.plus.com
Sat Apr 25 16:21:13 CEST 2009
Martin v. Löwis wrote:
>> If the bytes are mapped to single half surrogate codes instead of the
>> normal pairs (low+high), then I can see that decoding could never be
>> ambiguous and encoding could produce the original bytes.
> I was confused by Markus Kuhn's original UTF-8b specification. I have
> now changed the PEP to avoid using PUA characters at all.
I find the PEP easier to understand now.
In detail I'd say that if a sequence of bytes >=0x80 is found which is
not valid UTF-8, then the first byte is mapped to a half surrogate and
then decoding is continued from the next byte.
The only drawback I can see is if the UTF-8 bytes actually decode to a
half surrogate. However, half surrogates should really only occur in
UTF-16 (as I understand it), so they shouldn't be encoded in UTF-8
As for handling this case, you could either:
1. Raise an exception (which is what you're trying to avoid)
2. Treat it as invalid UTF-8 and map the bytes to half surrogates
(encoding would produce the original bytes).
I'd prefer option 2.
Anyway, +1 from me.
More information about the Python-Dev