[Python-ideas] PEP 540: Add a new UTF-8 mode

[INADA Naoki]

>> Always use UTF-8
>> ----------------
>>
>> Python already always use the UTF-8 encoding on Mac OS X, Android and Windows.
>> Since UTF-8 became the defacto encoding, it makes sense to always use it on all
>> platforms with any locale.
>
>    Please don't! I use different locales and encodings, sometimes it's
> utf-8, sometimes not - but I have properly configured LC_* settings and
> I prefer Python to follow my command. It'd be disgusting if Python
> starts to bend me to its preferences.

For stdio (including console), PYTHONIOENCODING can be used for
supporting legacy system.
e.g. `export PYTHONIOENCODING=$(locale charmap)`

For commandline argument and filepath, UTF-8/surrogateescape can round trip.
But mojibake may happens when pass the path to GUI.

If we chose "Always use UTF-8 for fs encoding", I think
PYTHONFSENCODING envvar should be
added again.  (It should be used from startup: decoding command line argument).

>
>> The risk is to introduce mojibake if the locale uses a different encoding,
>> especially for locales other than the POSIX locale.
>
>    There is no such risk for me as I already have mojibake in my
> systems. Two most notable sources of mojibake are:
>
> 1) FTP servers - people create files (both names and content) in
>    different encodings; w32 FTP clients usually send file names and
>    content in cp1251 (Russian Windows encoding), sometimes in cp866
>    (Russian Windows OEM encoding).
>
> 2) MP3 tags and play lists - almost always cp1251.
>
>    So whatever my personal encoding is - koi8-r or utf-8 - I have to
> deal with file names and content in different encodings.

3) unzip zip file sent by Windows.   Windows user use no-ASCII filenames, and
create legacy (no UTF-8) zip file very often.

I think people using non UTF-8 should solve encoding issue by themselves.
People should use ASCII or UTF-8 always if they don't want to see mojibake.

>
> Oleg.
> --
>      Oleg Broytman            http://phdru.name/            phd at phdru.name
>            Programmers don't die, they just GOSUB without RETURN.
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/