[issue1552880] [Python2] Use utf-8 in the import machinery on Windows to support unicode paths

STINNER Victor report at bugs.python.org
Sat Sep 4 03:00:41 CEST 2010

STINNER Victor <victor.stinner at haypocalc.com> added the comment:

Oh, I didn't see that the issue was specific to Python2. I updated the issue's title. If I understood correctly, the issue is also specific to Windows.

Do you know if your patch changes the public API? (break the compatibility)


FYI about Python3:

> That's an inventive way of breaking the unicode standard :)

It is described in the PEP 383 and it does solve a real and common issue: store a filename that cannot be decoded with the filesystem encoding. The operation is reversible. In Python 3.2, there are os.fsdecode() and os.fsencode() functions. On UNIX/BSD, os.encode(os.fsdecode(x)) is x, if x is a bytes object.

The PEP 383 introduces the surrogateescape error handler which does create surrogates on decode, and convert back surrogates to bytes on encode.

> Anyway, why would you worry about that? My patch doesn't use
> "surrogateescape" so there is no problem.

In Python3, filenames are stored as unicode. On UNIX/BSD, if a filename cannot be decode, it is encoded with surrogates. To get a full unicode support in Python3, you have to support surrogates.

title: Unicode Imports -> [Python2] Use utf-8 in the import machinery on Windows to support unicode paths

Python tracker <report at bugs.python.org>

More information about the Python-bugs-list mailing list