[issue13643] 'ascii' is a bad filesystem default encoding

Martin report at bugs.python.org
Wed Dec 21 20:47:05 CET 2011

Martin <gzlist at googlemail.com> added the comment:

> Nope. The locale encoding is chosen using LC_ALL, LC_CTYPE or LANG 
> variable: use the first non-empty variable. LC_MESSAGES doesn't affect 
> the encoding. Example:

That's good to know, thanks. Only leaves the case where setlocale is called again with a different value.

> Yes, and ASCII and UTF-8 are incompatible. ASCII is unable to decode an 
> UTF-8 encoded string.

I think we're envisioning different things here.

  os.stat("\u2601") # with LANG=C
    current -> UnicodeEncodeError
    changed -> works if utf-8 encoded file exists

  os.listdir() # with LANG=C
    current -> returns non-ascii as unicode with funky surrogates
    changed -> returns non-utf-8 as unicode with funky surrogates

> It affects everything because filenames are used everywhere.

But currently everything handling filenames as unicode on nix needs to worry about surrogates (that can't be encoded as ascii) already, or it will still be passing values that can't be interpreted by other processes as you highlighed earlier. Making utf-8 names come out correctly rather than as surrogates doesn't seem like it increases the burden.


Python tracker <report at bugs.python.org>

More information about the Python-bugs-list mailing list