[Python-Dev] PEP 540: Add a new UTF-8 mode

[Victor Stinner]

Chris:
> I just took another look at 538 -- and yes, the relationship between the two
> is really unclear. In particular, with 538, why do we need 540? I honestly
> don't know.

The PEP 538 only impacts platforms which provide the C.UTF-8 locale or
a variant: only a few recent Linux distribution. I know Fedora, maybe
a few other have it? FreeBSD and macOS are completely ignored by the
PEP 538. The PEP 540 uses the UTF-8 encoding for the POSIX locale on
*all* platforms.

Moreover, the PEP 538 only concerns the POSIX locale (locale "C"),
whereas the PEP 540 is usable with any locale. For example, using the
"fr_FR.iso88591" locale, the encoding is Latin1. But if you enable the
UTF-8 mode with this locale, Python will use UTF-8.

The other difference is that the PEP 538 is implemented with
setlocale(LC_CTYPE, "C.UTF-8"), whereas the PEP 540 is implemented in
Python internals and ignores the locale. The PEP 540 scope is limited
to Python, non-Python running in the same process is not aware of the
"Python UTF-8 mode".

Victor