[Python-Dev] Inconsistencies if locale and filesystem encodings are different

[M.-A. Lemburg]

Victor Stinner wrote:
> Hi,
> 
> A PYTHONFSENCODING environment variable was added to Python 3.2: issue #8622. 
> This variable introduces an inconstency because the filesystem and the locale 
> encodings can now be different.
> 
> There are (at least) four issues related to this problem. We have 2 choices to 
> fix these issues:
> 
>  (a) use the same encoding to encode and decode values (it can be different 
> for each issue)
> 
>  (b) remove PYTHONFSENCODING variable and raise an error if locale and 
> filesystem encodings are different (ensure that both encodings are the same)
> 
> Even if choice (a) is not easy to implement, it is feasible and I already 
> wrote some patches.
> 
> I don't understand how Python interact with other programs who ignore the 
> PYTHONFSENCODING environment variable. It's like Python uses its own "locale".
> 
> Choice (b) looks easy to implement, but... there is the problem of Mac OS X. 
> Mac OS X uses utf-8 encoding for the filesystem (and not the locale encoding), 
> whereas it looks like the locale encoding is used for the command line 
> arguments. See issue #4388 for more information.
> 
> There is also maybe an useful usecase of the PYTHONFSENCODING, but I don't 
> remember which one :-)

You have to differentiate between the meaning of a file system
encoding and the locale:

A file system encoding defines how the applications interact
with the file system.

A locale defines how the user expects to interact with the
application.

It is well possible that the two are different. Mac OS X is
just one example. Another common example is having a Unix
account using the C locale (=ASCII) while working on a UTF-8
file system.

BTW: We added that because just like I/O encoding, you need to be
able to override the setting determined by Python via locale
introspection, which may be wrong. The env var is only meant
as a way to solve encoding problems in special situations where
the local cannot be used to determine the file system or
input/output encoding.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Oct 07 2010)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/