[Python-Dev] Inconsistencies if locale and filesystem encodings are different
mal at egenix.com
Thu Oct 7 18:35:09 CEST 2010
Victor Stinner wrote:
> A PYTHONFSENCODING environment variable was added to Python 3.2: issue #8622.
> This variable introduces an inconstency because the filesystem and the locale
> encodings can now be different.
> There are (at least) four issues related to this problem. We have 2 choices to
> fix these issues:
> (a) use the same encoding to encode and decode values (it can be different
> for each issue)
> (b) remove PYTHONFSENCODING variable and raise an error if locale and
> filesystem encodings are different (ensure that both encodings are the same)
> Even if choice (a) is not easy to implement, it is feasible and I already
> wrote some patches.
> I don't understand how Python interact with other programs who ignore the
> PYTHONFSENCODING environment variable. It's like Python uses its own "locale".
> Choice (b) looks easy to implement, but... there is the problem of Mac OS X.
> Mac OS X uses utf-8 encoding for the filesystem (and not the locale encoding),
> whereas it looks like the locale encoding is used for the command line
> arguments. See issue #4388 for more information.
> There is also maybe an useful usecase of the PYTHONFSENCODING, but I don't
> remember which one :-)
You have to differentiate between the meaning of a file system
encoding and the locale:
A file system encoding defines how the applications interact
with the file system.
A locale defines how the user expects to interact with the
It is well possible that the two are different. Mac OS X is
just one example. Another common example is having a Unix
account using the C locale (=ASCII) while working on a UTF-8
BTW: We added that because just like I/O encoding, you need to be
able to override the setting determined by Python via locale
introspection, which may be wrong. The env var is only meant
as a way to solve encoding problems in special situations where
the local cannot be used to determine the file system or
Professional Python Services directly from the Source (#1, Oct 07 2010)
>>> Python/Zope Consulting and Support ... http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
::: Try our new mxODBC.Connect Python Database Interface for free ! ::::
eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
Registered at Amtsgericht Duesseldorf: HRB 46611
More information about the Python-Dev