[Python-Dev] PEP 538 (review round 2): Coercing the legacy C locale to a UTF-8 based locale

[Nick Coghlan]

On 23 May 2017 at 18:38, INADA Naoki <songofacandy at gmail.com> wrote:
> Hi, Nick.
>
> I read again and I think PEP 538 is mostly ready for accepted,
> without waiting PEP 540.

Great to hear!

> One remaining my concern is setting LANG.
>
>> Setting LANG to C.UTF-8 ensures that even components that only check the LANG fallback for their locale settings will still use C.UTF-8 .
> https://www.python.org/dev/peps/pep-0538/#setting-both-lc-ctype-lang-for-utf-8-locale-coercion
>
> I feel setting only LC_CTYPE making PEP 538 simpler.
> Is there any real component using LANG for deciding encoding?
>
> For example, date command refers LC_TIME.
>
> $ LANG=ja_JP.UTF-8 LC_CTYPE=C date
> 2017年  5月 23日 火曜日 17:31:03 JST
>
> $ LANG=ja_JP.UTF-8 LC_CTYPE=C.UTF-8 date  # Coercing only LC_CTYPE
> 2017年  5月 23日 火曜日 17:32:58 JST
>
> $ LANG=C.UTF-8 LC_CTYPE=C.UTF-8 date  # Coercing both of LC_CTYPE and LANG
> Tue May 23 17:31:10 JST 2017
>
> In this case, coercing only LC_CTYPE has less side-effect.
>
> Would you add example demonstrates how coercing LANG helps people?

I'm honestly not sure it does - I think it's an assumption I added to
the PEP early on, and never actually tested. Looking at it more
closely now, all of the interpreter level checks are specifically for
LC_CTYPE, and experimenting with "LANG=C LC_CTYPE=C.UTF-8" indicates
that coercing only LC_CTYPE is still enough to fix the GNU readline
encoding compatibility problem covered in
https://www.python.org/dev/peps/pep-0538/#considering-locale-coercion-independently-of-utf-8-mode

So I'll take another pass through the implementation this weekend, and
simplify it to only set LC_CTYPE regardless of whether it's using
C.UTF-8, C.utf8, or UTF-8 as the target locale. Assuming that doesn't
uncover any hidden problems with the idea, I'll then update the PEP to
match.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia