[Python-Dev] PEP 383: Non-decodable Bytes in System Character Interfaces
James Y Knight
foom at fuhm.net
Fri Apr 24 16:54:07 CEST 2009
On Apr 24, 2009, at 8:00 AM, Paul Moore wrote:
> However, it *does* agree with the reality of Windows file systems. The
> fundamental problem here is that there is a strong OS disparity - for
> Windows, the OS uses Unicode, for POSIX, the OS uses bytes.
It's unfortunately the case that this isn't *precisely* true. Windows
uses arbitrary 16-bit sequences, just as unix uses arbitrary 8-bit
sequences. Neither one is required by the operating system to be a
proper unicode encoding. The main difference is that there is already
a widely accepted way to decode a improperly-encoded 16-bit-sequence
with the utf-16 codec: simply leave the lone surrogate pairs in place.
More information about the Python-Dev