[Python-Dev] PEP 538: Coercing the legacy C locale to a UTF-8 based locale

[Nick Coghlan]

On 12 March 2017 at 08:36, Jakub Wilk <jwilk at jwilk.net> wrote:

> This is a very bad idea.
>
> It seems to based on an assumption that the C locale is always some kind
> of pathology. Admittedly, it sometimes is a result of misconfiguration or a
> mistake. (But I don't see why it's the interpreter's job to correct such
> mistakes.) However, in some cases the C locale is a normal environment for
> system services, cron scripts, distro package builds and whatnot.


An environment in which Python 3's eager decoding of operating system
provided values to Unicode fails.


> It's possible to write Python programs that are locale-agnostic.
>

If a program is genuinely locale-agnostic, it will be unaffected by this
PEP.


> It's also possible to write programs that are locale-dependent, but handle
> ASCII as locale encoding gracefully.
>

No, it is not generally feasible to write such programs in Python 3. That's
the essence of the problem, and why the PEP deprecates support for the
legacy C locale in Python 3.


> Or you might want to write a program that intentionally aborts with an
> explanatory error message when the locale encoding doesn't have sufficient
> Unicode coverage. ("Errors should never pass silently" anyone?)
>

This is what click does, but it only does it because that isn't possible
for click to do the right thing given Python 3's eager decoding of various
values as ASCII.


> With this proposal, none of the above seems possible to correctly
> implement in Python.
>

The first case remains unchanged, the other two will need to use Python 2.7
or Tauthon. I'm fine with that.


> * Nick Coghlan <ncoghlan at gmail.com>, 2017-03-05, 17:50:
>
> While this PEP ensures that developers that need to do so can still opt-in
>> to running their Python code in the legacy C locale,
>>
>
> Yeah, no, it doesn't.
>
> It's impossible do disable coercion from Python code, because it happens
> to early. The best you can do is to write a wrapper script in a different
> language that sets PYTHONCOERCECLOCALE=0; but then you still get a spurious
> warning.
>

It's not a spurious warning, as Python 3's Unicode handling for
environmental interactions genuinely doesn't work properly in the legacy C
locale (unless you're genuinely promising to only ever feed it ASCII
values, but that isn't a realistic guarantee to make).

However, I'm also open to having that particular setting also disable the
runtime warning from the shared library.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170312/20052d08/attachment.html>