[Python-Dev] PEP 538: Coercing the legacy C locale to a UTF-8 based locale

[Nick Coghlan]

On 9 May 2017 at 13:44, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 8 May 2017 at 15:34, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> On 7 May 2017 at 15:22, INADA Naoki <songofacandy at gmail.com> wrote:
>>> ## Background
>>>
>>> Locale coercion in current PEP 538 has some downsides:
>>>
>>> * If user set `LANG=C LC_DATE=ja_JP.UTF-8`, locale coercion may
>>>   overrides LC_DATE.
>>
>> The fact it sets "LC_ALL" has previously been raised as a concern with
>> PEP 538, so it probably makes sense to drop that aspect and just
>> override "LANG". The scenarios where it makes a difference are
>> incredibly obscure (involving non-default SSH locale forwarding
>> settings for folks using SSH on Mac OS X to connect to remote Linux
>> systems), while just setting "LANG" will be sufficient to address the
>> "LANG=C" case that is the main driver for the PEP.
>
> It occurs to me we can even still handle the forwarded
> "LC_CTYPE=UTF-8" case by changing the locale coercion to set LC_CTYPE
> & LANG, rather than just setting LANG as I suggested above.
>
> That way `LANG=C LC_DATE=ja_JP.UTF-8` would still respect the explicit
> LC_DATE setting, `LC_CTYPE=C` would be handled the same way as
> `LANG=C`, and LC_ALL=C would continue to provide a way to force the C
> locale even for LC_CTYPE without needing to be aware of the Python
> specific PYTHONCOERCECLOCALE setting.

I've posted an updated reference implementation that works this way,
and it turned out to have some rather nice benefits: not only did it
make the handling of full locales (C.UTF-8, C.utf8) and partial
locales (UTF-8) more consistent (allowing for a net deletion of code),
it also meant I no longer needed a custom test case in _testembed to
check the locale warning. Instead, the affected test cases now just
set "LC_ALL" as a locale override that switches off CPython's locale
coercion without also switching off the locale warning.

Code changes: https://github.com/ncoghlan/cpython/commit/476a78133c94d82e19b89f50036cecd9b4214e7a

Rather than posting the PEP updates here though, I'll start a new
thread that explains what has changed since my initial posting to
python-dev back in March.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia