[Python-Dev] PEP 383: Non-decodable Bytes in System Character Interfaces

[Hrvoje Niksic]

Zooko O'Whielacronx wrote:
>> If you switch to iso8859-15 only in the presence of undecodable  
>> UTF-8, then you have the same round-trip problem as the PEP: both  
>> b'\xff' and b'\xc3\xbf' will be converted to u'\u00ff' without a  
>> way to unambiguously recover the original file name.
> 
> Why do you say that?  It seems to work as I expected here:
> 
>  >>> '\xff'.decode('iso-8859-15')
> u'\xff'
>  >>> '\xc3\xbf'.decode('iso-8859-15')
> u'\xc3\xbf'

Here is what I mean by "switch to iso8859-15" only in the presence of 
undecodable UTF-8:

def file_name_to_unicode(fn, encoding):
     try:
         return fn.decode(encoding)
     except UnicodeDecodeError:
         return fn.decode('iso-8859-15')

Now, assume a UTF-8 locale and try to use it on the provided example 
file names.

 >>> file_name_to_unicode(b'\xff', 'utf-8')
'ÿ'
 >>> file_name_to_unicode(b'\xc3\xbf', 'utf-8')
'ÿ'

That is the ambiguity I was referring to -- to different byte sequences 
result in the same unicode string.