On Fri, Feb 12, 2021 at 12:28 PM Jim J. Jewett firstname.lastname@example.org wrote:
(I apologize if my summaries distort what Inada Naoki email@example.com explained.)
He said that some people use the default None when they really want either UTF-8 or ASCII.
Yes. Even Python core developers. For example: https://bugs.python.org/issue33684
This is just one example. I saw many codes using default encoding to read JSON, YAML, TOML, Markdown, etc...
My concern is that the warning will be a false alarm if they really do need whatever locale returns, and that case may still be common. (If web browsers had stopped bothering to sniff for other charsets, then maybe that situation really was getting rare.)
That's one of reason why this warning is opt-in, like BytesWarning.
I asked when encoding=None is actually different from encoding=locale, currently spelled encoding=locale.getpreferredencoding(False)
I don't understand this sentence. This PEP proposes `encoding="locale"` that is equal to encoding=None but don't emit EncodingWarning.
There was discussion about difference between `encoding=None` and `encoding=locale.getpreferredencoding(False)` in this thread.
They can be different on Windows console, presumably because the environment settings that control locale may differ from the charset actually used by the console. Even then, it only differs for open() when PYTHONLEGACYWINDOWSSTDIO is set, and for TextIOWrapper() When the file is not _WindowsConsoleIO
To me, that sounds narrow enough to be a windows issue, rather than an issue with open.
Yes. So if user want to specify locale-specific encoding and don't want to drop Python 3.9 support, user can use encoding=locale.getpreferredencoding(False).
But this PEP doesn't recommend it. Third party libraries can use `encoding="locale"` after they drop Python 3.9 support.
Is there some way to write an encoding that sniffs for charsets, particularly on windows, and to use that as the default instead of assuming that locale will be correct?
There is no reliable way, AFAIK.