PEP 597: Add optional EncodingWarning
Hi, all.
I updated the PEP 597 yesterday. Please review it to move it forward.
PEP: https://www.python.org/dev/peps/pep-0597/ Previous thread: https://discuss.python.org/t/3880
Main difference from the previous version:
* Added new warning category; EncodingWarning * Added dedicated option to enable the warning instead of using dev mode.
Abstract ========
Add a new warning category ``EncodingWarning``. It is emitted when ``encoding`` option is omitted and the default encoding is a locale encoding.
The warning is disabled by default. New ``-X warn_encoding`` command-line option and ``PYTHONWARNENCODING`` environment variable are used to enable the warnings.
Motivation ==========
Using the default encoding is a common mistake ----------------------------------------------
Developers using macOS or Linux may forget that the default encoding is not always UTF-8.
For example, ``long_description = open("README.md").read()`` in ``setup.py`` is a common mistake. Many Windows users can not install the package if there is at least one non-ASCII character (e.g. emoji) in the ``README.md`` file which is encoded in UTF-8.
For example, 489 packages of the 4000 most downloaded packages from PyPI used non-ASCII characters in README. And 82 packages of them can not be installed from source package when locale encoding is ASCII. [1_] They used the default encoding to read README or TOML file.
Another example is ``logging.basicConfig(filename="log.txt")``. Some users expect UTF-8 is used by default, but locale encoding is used actually. [2_]
Even Python experts assume that default encoding is UTF-8. It creates bugs that happen only on Windows. See [3_] and [4_].
Emitting a warning when the ``encoding`` option is omitted will help to find such mistakes.
Prepare to change the default encoding to UTF-8 -----------------------------------------------
We had chosen to use locale encoding for the default text encoding in Python 3.0. But UTF-8 has been adopted very widely since then.
We might change the default text encoding to UTF-8 in the future. But this change will affect many applications and libraries. Many ``DeprecationWarning`` will be emitted if we start emitting the warning by default. It will be too noisy.
Although this PEP doesn't propose to change the default encoding, this PEP will help to reduce the warning in the future if we decide to change the default encoding.
Specification =============
``EncodingWarning`` --------------------
Add new ``EncodingWarning`` warning class which is a subclass of ``Warning``. It is used to warn when the ``encoding`` option is omitted and the default encoding is locale-specific.
Options to enable the warning ------------------------------
``-X warn_encoding`` option and the ``PYTHONWARNENCODING`` environment variable are added. They are used to enable the ``EncodingWarning``.
``sys.flags.encoding_warning`` is also added. The flag represents ``EncodingWarning`` is enabled.
When the option is enabled, ``io.TextIOWrapper()``, ``open()``, and other modules using them will emit ``EncodingWarning`` when ``encoding`` is omitted.
``encoding="locale"`` option ----------------------------
``io.TextIOWrapper`` accepts ``encoding="locale"`` option. It means same to current ``encoding=None``. But ``io.TextIOWrapper`` doesn't emit ``EncodingWarning`` when ``encoding="locale"`` is specified.
Add ``io.LOCALE_ENCODING = "locale"`` constant too. This constant can be used to avoid confusing ``LookupError: unknown encoding: locale`` error when the code is run in old Python accidentally.
The constant can be used to test that ``encoding="locale"`` option is supported too. For example,
.. code-block::
# Want to suppress an EncodingWarning but still need support # old Python versions. locale_encoding = getattr(io, "LOCALE_ENCODING", None) with open(filename, encoding=locale_encoding) as f: ...
``io.text_encoding()`` -----------------------
``io.text_encoding()`` is a helper function for functions having ``encoding=None`` option and passing it to ``io.TextIOWrapper()`` or ``open()``.
Pure Python implementation will be like this::
def text_encoding(encoding, stacklevel=1): """Helper function to choose the text encoding.
When *encoding* is not None, just return it. Otherwise, return the default text encoding (i.e., "locale").
This function emits EncodingWarning if *encoding* is None and sys.flags.encoding_warning is true.
This function can be used in APIs having encoding=None option and pass it to TextIOWrapper or open. But please consider using encoding="utf-8" for new APIs. """ if encoding is None: if sys.flags.encoding_warning: import warnings warnings.warn("'encoding' option is omitted", EncodingWarning, stacklevel + 2) encoding = LOCALE_ENCODING return encoding
For example, ``pathlib.Path.read_text()`` can use the function like:
.. code-block::
def read_text(self, encoding=None, errors=None): encoding = io.text_encoding(encoding) with self.open(mode='r', encoding=encoding, errors=errors) as f: return f.read()
By using ``io.text_encoding()``, ``EncodingWarning`` is emitted for the caller of ``read_text()`` instead of ``read_text()``.
Affected stdlibs -------------------
Many stdlibs will be affected by this change.
Most APIs accepting ``encoding=None`` will use ``io.text_encoding()`` as written in the previous section.
Where using locale encoding as the default encoding is reasonable, ``encoding=io.LOCALE_ENCODING`` will be used instead. For example, ``subprocess`` module will use locale encoding for the default encoding of the pipes.
Many tests use ``open()`` without ``encoding`` specified to read ASCII text files. They should be rewritten with ``encoding="ascii"``.
Rationale =========
Opt-in warning ---------------
Although ``DeprecationWarning`` is suppressed by default, emitting ``DeprecationWarning`` always when ``encoding`` option is omitted would be too noisy.
Noisy warnings may lead developers to dismiss the ``DeprecationWarning``.
"locale" is not a codec alias -----------------------------
We don't add the "locale" to the codec alias because locale can be changed in runtime.
Additionally, ``TextIOWrapper`` checks ``os.device_encoding()`` when ``encoding=None``. This behavior can not be implemented in the codec.
Reference Implementation ========================
https://github.com/python/cpython/pull/19481
References ==========
.. [1] "Packages can't be installed when encoding is not UTF-8" (https://github.com/methane/pep597-pypi-ascii)
.. [2] "Logging - Inconsistent behaviour when handling unicode" (https://bugs.python.org/issue37111)
.. [3] Packaging tutorial in packaging.python.org didn't specify encoding to read a ``README.md`` (https://github.com/pypa/packaging.python.org/pull/682)
.. [4] ``json.tool`` had used locale encoding to read JSON files. (https://bugs.python.org/issue33684)
Copyright =========
This document has been placed in the public domain.
Hi Inada-san,
I followed the discussions on your different PEP and I like overall your latest PEP :-) I have some minor remarks.
On Mon, Feb 1, 2021 at 6:55 AM Inada Naoki songofacandy@gmail.com wrote:
The warning is disabled by default. New ``-X warn_encoding`` command-line option and ``PYTHONWARNENCODING`` environment variable are used to enable the warnings.
Maybe "warn implicit encoding" or "warn omit encoding" (not sure if it's make sense written like that in english ;-)) would be more explicit.
Options to enable the warning
``-X warn_encoding`` option and the ``PYTHONWARNENCODING`` environment variable are added. They are used to enable the ``EncodingWarning``.
``sys.flags.encoding_warning`` is also added. The flag represents ``EncodingWarning`` is enabled.
Nitpick: I would prefer using the same name for the -X option and the sys.flags attribute (ex: sys.flags.warn_encoding).
``encoding="locale"`` option
``io.TextIOWrapper`` accepts ``encoding="locale"`` option. It means same to current ``encoding=None``. But ``io.TextIOWrapper`` doesn't emit ``EncodingWarning`` when ``encoding="locale"`` is specified.
Can you please define if os.device_encoding(fd) is called if encoding="locale" is used? It seems so, so it's not obvious from the PEP.
In Python 3.10, I added _locale._get_locale_encoding() function which is exactly what the encoding used by open() when no encoding is specified (encoding=None) and when os.device_encoding(fd) returns None. See _Py_GetLocaleEncoding() for the C implementation (Python/fileutils.c).
Maybe we should add a public locale.get_locale_encoding() function? On Unix, this function uses nl_langinfo(CODESET) *without* setting LC_CTYPE locale to the user preferred locale.
I understand that encoding=locale.get_locale_encoding() would be different from encoding="locale": encoding=locale.get_locale_encoding() doesn't call os.device_encoding(), right?
Maybe the PEP should also explain (in a "How to teach this" section?) when encoding="locale" is better than a specific encoding, like encoding="utf-8" or encoding="cp1252". In my experience, it's mostly for the inter-operability which other applications which also use the current locale encoding.
By the way, I recently rewrote the documentation about the encodings used by Python:
* https://docs.python.org/dev/glossary.html#term-locale-encoding * https://docs.python.org/dev/glossary.html#term-locale-encoding * https://docs.python.org/dev/c-api/init_config.html#c.PyConfig.filesystem_enc... * https://docs.python.org/dev/c-api/init_config.html#c.PyConfig.stdio_encoding * https://docs.python.org/dev/library/os.html#utf8-mode
Add ``io.LOCALE_ENCODING = "locale"`` constant too. This constant can be used to avoid confusing ``LookupError: unknown encoding: locale`` error when the code is run in old Python accidentally.
I'm not sure that it is useful. I like a simple "locale" literal string. If there is a constant is io, people may start to think that it's specific and will add "import io" just to get the string "locale".
I don't think that we should care too much about the error message rased by old Python versions.
Opt-in warning
Although ``DeprecationWarning`` is suppressed by default, emitting ``DeprecationWarning`` always when ``encoding`` option is omitted would be too noisy.
The PEP is not very clear. Does "-X warn_encoding" only emits the warning, or does it also display it by default? Does it add a warning filter for EncodingWarning?
The PEP has no "Backward compatibility" section. Is it possible to monkey-patch Python to implement this PEP (maybe only partially) on old Python versions? I'm asking to prepare existing projects for future EncodingWarning.
The main question is if it's possible to use encoding="locale" on Python 3.6-3.9 (maybe using some ugly hacks). By the way, your PEP has no target Python version ;-) Do you want to get it in Python 3.10 or 3.11?
Victor
On Tue, Feb 2, 2021 at 12:23 AM Victor Stinner vstinner@python.org wrote:
Hi Inada-san,
I followed the discussions on your different PEP and I like overall your latest PEP :-) I have some minor remarks.
On Mon, Feb 1, 2021 at 6:55 AM Inada Naoki songofacandy@gmail.com wrote:
The warning is disabled by default. New ``-X warn_encoding`` command-line option and ``PYTHONWARNENCODING`` environment variable are used to enable the warnings.
Maybe "warn implicit encoding" or "warn omit encoding" (not sure if it's make sense written like that in english ;-)) would be more explicit.
Yes, it's explicit. So I used `PYTHONWARNDEFAULTENCODING` first. But I feel it's unreadable. That's why I shorten the option name.
I wait to see more feedback about naming.
Options to enable the warning
``-X warn_encoding`` option and the ``PYTHONWARNENCODING`` environment variable are added. They are used to enable the ``EncodingWarning``.
``sys.flags.encoding_warning`` is also added. The flag represents ``EncodingWarning`` is enabled.
Nitpick: I would prefer using the same name for the -X option and the sys.flags attribute (ex: sys.flags.warn_encoding).
OK, I will change the flag name same to option name.
``encoding="locale"`` option
``io.TextIOWrapper`` accepts ``encoding="locale"`` option. It means same to current ``encoding=None``. But ``io.TextIOWrapper`` doesn't emit ``EncodingWarning`` when ``encoding="locale"`` is specified.
Can you please define if os.device_encoding(fd) is called if encoding="locale" is used? It seems so, so it's not obvious from the PEP.
OK.
In Python 3.10, I added _locale._get_locale_encoding() function which is exactly what the encoding used by open() when no encoding is specified (encoding=None) and when os.device_encoding(fd) returns None. See _Py_GetLocaleEncoding() for the C implementation (Python/fileutils.c).
Maybe we should add a public locale.get_locale_encoding() function? On Unix, this function uses nl_langinfo(CODESET) *without* setting LC_CTYPE locale to the user preferred locale.
I can not imagine any use case. Isn't it just confusing?
I understand that encoding=locale.get_locale_encoding() would be different from encoding="locale": encoding=locale.get_locale_encoding() doesn't call os.device_encoding(), right?
Yes.
Maybe the PEP should also explain (in a "How to teach this" section?) when encoding="locale" is better than a specific encoding, like encoding="utf-8" or encoding="cp1252". In my experience, it's mostly for the inter-operability which other applications which also use the current locale encoding.
This option is for experts who are publishing cross-platform libraries, frameworks, etc.
For students, I am suggesting another idea that make UTF-8 mode more accessible.
Add ``io.LOCALE_ENCODING = "locale"`` constant too. This constant can be used to avoid confusing ``LookupError: unknown encoding: locale`` error when the code is run in old Python accidentally.
I'm not sure that it is useful. I like a simple "locale" literal string. If there is a constant is io, people may start to think that it's specific and will add "import io" just to get the string "locale".
I don't think that we should care too much about the error message rased by old Python versions.
This constant not only for replacing "locale" litera. As example code in the PEP, it can be used to test wheather TextIOWrapper supports `encoding="locale"` .
`open(fn, encoding=getattr(io, "LOCALE_ENCODING", None))` works both for Python ~3.9 and Python 3.10~.
Opt-in warning
Although ``DeprecationWarning`` is suppressed by default, emitting ``DeprecationWarning`` always when ``encoding`` option is omitted would be too noisy.
The PEP is not very clear. Does "-X warn_encoding" only emits the warning, or does it also display it by default? Does it add a warning filter for EncodingWarning?
This section is not the spec. This section is the rationale for adding EncodingWarning instead of using DeprecationWarning.
As spec saying, EncodingWarning is a subclass of Warning. So it is displayed by default. But it is not emitted by default.
When -X encoding_warning (or -X warn_default_encoding) is used, the warning is emitted and shown unless the user suppresses warnings.
The PEP has no "Backward compatibility" section. Is it possible to monkey-patch Python to implement this PEP (maybe only partially) on old Python versions? I'm asking to prepare existing projects for future EncodingWarning.
This PEP doesn't have "backward compatibility" section because the PEP doesn't break any backward compatibility. Unless the option is enabled, no warnings are emitted by the PEP, like `-b` option and BytesWarning.
And if developers want to support Python ~3.9 and use -X warn_default_encoding on 3.10, they need to write `encoding=getattr(io, "LOCALE_ENCODING", None)`, as written in the spec.
The main question is if it's possible to use encoding="locale" on Python 3.6-3.9 (maybe using some ugly hacks).
No.
By the way, your PEP has no target Python version ;-) Do you want to get it in Python 3.10 or 3.11?
Oh, I'm sorry. I want to make it in 3.10.
Precisely speaking, add option and warning in 3.10. But no need to fix all warnings in stdlib and tests by 3.10.
I expect we find many codes using locale-specific encoding without reason in stdlib and we need to fix them until 3.11 or even 3.12. After that, we can recommend using the option for many third-party libraries.
My long-term plan (out of scope of this PEP) is:
* Python 3.10 : Add the optional warning * Python 3.12 : Fix all warnings in stdlib, start the option for third parties. * Python 3.16 : Discuss changing the default encoding. If we decided to change, enable the warning by default. * Python 3.20 : Change the default encoding to UTF-8.
Regards,
On Tue, Feb 2, 2021 at 5:40 AM Inada Naoki songofacandy@gmail.com wrote:
In Python 3.10, I added _locale._get_locale_encoding() function which is exactly what the encoding used by open() when no encoding is specified (encoding=None) and when os.device_encoding(fd) returns None. See _Py_GetLocaleEncoding() for the C implementation (Python/fileutils.c).
Maybe we should add a public locale.get_locale_encoding() function? On Unix, this function uses nl_langinfo(CODESET) *without* setting LC_CTYPE locale to the user preferred locale.
I can not imagine any use case. Isn't it just confusing?
It's the same than locale.getpreferredencoding(False) but with a more explicit name, no argument and a *sane default behavior* (don't change the LC_CTYPE locale temporarily).
The use case is to pass text to the OS (or get text from the OS) when you cannot pass text directly, but must encode it (or decode it) manually. Not all use cases involve files ;-)
Example of locale.getpreferredencoding() usage:
* XML ElementTree uses locale.getpreferredencoding() when encoding="unicode" is used * Deprecate gettext functions use it to encode to bytes * the cgi module uses it to encode the URL query string for the CGI stdin (GET and HEAD methods)
I dislike getpreferredencoding() because by default it changes temporarily the LC_CTYPE locale which affects all threads, and this is bad.
Well, it doesn't have to be part of the PEP ;-)
I understand that encoding=locale.get_locale_encoding() would be different from encoding="locale": encoding=locale.get_locale_encoding() doesn't call os.device_encoding(), right?
Yes.
Would it be useful to add a io.get_locale_encoding(fd)->str (maybe "get_default_encoding"?) function which gives the chosen encoding from a file descriptor, similar to open(fd, encoding="locale").encoding? The os.device_encoding() call is not obvious.
Maybe the PEP should also explain (in a "How to teach this" section?) when encoding="locale" is better than a specific encoding, like encoding="utf-8" or encoding="cp1252". In my experience, it's mostly for the inter-operability which other applications which also use the current locale encoding.
This option is for experts who are publishing cross-platform libraries, frameworks, etc.
For students, I am suggesting another idea that make UTF-8 mode more accessible.
Maybe just say that in "How to teach this" section in the PEP?
In case of doubt, pass encoding="utf-8". Only use encoding="locale" if you understand that the encoding changes depending on the platform and the user locale. The common issue with encoding="locale" is that files should not be exchanged between two computers. encoding="locale" is good for files which remain local. It's also good for interoperability with other applications which use the locale encoding and with the terminal.
Opt-in warning
Although ``DeprecationWarning`` is suppressed by default, emitting ``DeprecationWarning`` always when ``encoding`` option is omitted would be too noisy.
The PEP is not very clear. Does "-X warn_encoding" only emits the warning, or does it also display it by default? Does it add a warning filter for EncodingWarning?
This section is not the spec. This section is the rationale for adding EncodingWarning instead of using DeprecationWarning.
As spec saying, EncodingWarning is a subclass of Warning. So it is displayed by default. But it is not emitted by default.
When -X encoding_warning (or -X warn_default_encoding) is used, the warning is emitted and shown unless the user suppresses warnings.
I understand that EncodingWarning is always displayed by default (default warning filters don't ignore it, whereas DeprecationWarning are ignored by default), but no warning is emitted by default. Ok, that makes sense. Maybe try to say it explicitly in the PEP.
This PEP doesn't have "backward compatibility" section because the PEP doesn't break any backward compatibility.
IMO it's a good thing to always have the section, just to say that you took time to think about backward compatibility ;-) The section can be empty, like just say "there is no incompatible change" ;-)
And if developers want to support Python ~3.9 and use -X warn_default_encoding on 3.10, they need to write `encoding=getattr(io, "LOCALE_ENCODING", None)`, as written in the spec.
Maybe repeat it in the Backward Compatibility section.
It's important to provide a way to prevent the warning without losing the support for old Python versions.
The main question is if it's possible to use encoding="locale" on Python 3.6-3.9 (maybe using some ugly hacks).
No.
Hum. To write code compatible with Python 3.9, I understand that encoding=None is the closest to encoding="locale".
And I understand that encoding=getattr(io, "LOCALE_ENCODING", None) is backward and forward compatible ;-)
Well, encoding=None will hopefully remain accepted with your PEP anyway for lazy developers ;-)
Oh, I'm sorry. I want to make it in 3.10.
Since it doesn't change anything by default, the warning is only displayed when you opt-in for it, IMO Python 3.10 target is reasonable.
Victor
On Tue, Feb 2, 2021 at 8:16 PM Victor Stinner vstinner@python.org wrote:
I understand that encoding=locale.get_locale_encoding() would be different from encoding="locale": encoding=locale.get_locale_encoding() doesn't call os.device_encoding(), right?
Yes.
Would it be useful to add a io.get_locale_encoding(fd)->str (maybe "get_default_encoding"?) function which gives the chosen encoding from a file descriptor, similar to open(fd, encoding="locale").encoding? The os.device_encoding() call is not obvious.
I don't think it's so useful. encoding=None is 99% same to encoding=locale.getpreferedencoding(False).
On Unix, os.device_encoding() just returns locale encoding. On Windows, os.device_encoding() is very unlikely used. open() uses WindowsConsoleIO for console unless PYTHONLEGACYWINDOWSSTDIO is set and encoding for it is UTF-8.
And that's why I removed the detailed behavior from the PEP. It is too detailed and almost unrelated to EncodingWarning. I wrote a simple comment in this section instead. https://www.python.org/dev/peps/pep-0597/#locale-is-not-a-codec-alias
Opt-in warning
Although ``DeprecationWarning`` is suppressed by default, emitting ``DeprecationWarning`` always when ``encoding`` option is omitted would be too noisy.
The PEP is not very clear. Does "-X warn_encoding" only emits the warning, or does it also display it by default? Does it add a warning filter for EncodingWarning?
This section is not the spec. This section is the rationale for adding EncodingWarning instead of using DeprecationWarning.
As spec saying, EncodingWarning is a subclass of Warning. So it is displayed by default. But it is not emitted by default.
When -X encoding_warning (or -X warn_default_encoding) is used, the warning is emitted and shown unless the user suppresses warnings.
I understand that EncodingWarning is always displayed by default (default warning filters don't ignore it, whereas DeprecationWarning are ignored by default), but no warning is emitted by default. Ok, that makes sense. Maybe try to say it explicitly in the PEP.
This PEP doesn't have "backward compatibility" section because the PEP doesn't break any backward compatibility.
IMO it's a good thing to always have the section, just to say that you took time to think about backward compatibility ;-) The section can be empty, like just say "there is no incompatible change" ;-)
And if developers want to support Python ~3.9 and use -X warn_default_encoding on 3.10, they need to write `encoding=getattr(io, "LOCALE_ENCODING", None)`, as written in the spec.
Maybe repeat it in the Backward Compatibility section.
It's important to provide a way to prevent the warning without losing the support for old Python versions.
will do.
The main question is if it's possible to use encoding="locale" on Python 3.6-3.9 (maybe using some ugly hacks).
No.
Hum. To write code compatible with Python 3.9, I understand that encoding=None is the closest to encoding="locale".
And I understand that encoding=getattr(io, "LOCALE_ENCODING", None) is backward and forward compatible ;-)
Well, encoding=None will hopefully remain accepted with your PEP anyway for lazy developers ;-)
Yes. I don't think this warning is enabled by default in near future. So developers can just use the option to find missing `encoding="utf-8"` bug.
Oh, I'm sorry. I want to make it in 3.10.
Since it doesn't change anything by default, the warning is only displayed when you opt-in for it, IMO Python 3.10 target is reasonable.
Victor
Night gathers, and now my watch begins. It shall not end until my death.
On Tue, Feb 2, 2021 at 1:40 PM Inada Naoki songofacandy@gmail.com wrote:
On Tue, Feb 2, 2021 at 12:23 AM Victor Stinner vstinner@python.org wrote:
Add ``io.LOCALE_ENCODING = "locale"`` constant too. This constant can be used to avoid confusing ``LookupError: unknown encoding: locale`` error when the code is run in old Python accidentally.
I'm not sure that it is useful. I like a simple "locale" literal string. If there is a constant is io, people may start to think that it's specific and will add "import io" just to get the string "locale".
I don't think that we should care too much about the error message rased by old Python versions.
This constant not only for replacing "locale" litera. As example code in the PEP, it can be used to test wheather TextIOWrapper supports `encoding="locale"` .
`open(fn, encoding=getattr(io, "LOCALE_ENCODING", None))` works both for Python ~3.9 and Python 3.10~.
I changed my mind. Since there is no plan to change the default encoding for now, no need to encourage `encoding="locale"` soon.
Until users can drop Python 3.9 support, they can use EncodingWarning only for finding missing `encoding="utf-8"` or `encoding="ascii"`.
I will remove the io.LOCALE_ENCODING.
I send a pull request https://github.com/python/peps/pull/1799
* Add Backward/Forward Compatibility section * Add How to teach this section * Remove io.LOCALE_ENCODING constant
On Sat, Feb 6, 2021 at 3:26 PM Inada Naoki songofacandy@gmail.com wrote:
I changed my mind. Since there is no plan to change the default encoding for now, no need to encourage `encoding="locale"` soon.
Until users can drop Python 3.9 support, they can use EncodingWarning only for finding missing `encoding="utf-8"` or `encoding="ascii"`.
I will remove the io.LOCALE_ENCODING.
I recall that something like 1 year ago, I basically tried to implement something like your PEP, to see if the stdlib calls open() without specifying an encoding. There were so many warnings, that the output was barely readable.
The warning would only be useful if there is a way to modify the code to make the warning quiet (fix the issue) without losing support with Python 3.9 and older.
I understand that open(filename) must be replaced with open(filename, encoding=("locale" if sys.version_info >= (3, 10) else None)) to make it backward and forward compatibility without emitting an EncodingWarning. One issue is that some people may blindly copy/paste this code pattern without thinking if "locale" is the proper encoding.
Victor
On Tue, Feb 9, 2021 at 7:23 PM Victor Stinner vstinner@python.org wrote:
I recall that something like 1 year ago, I basically tried to implement something like your PEP, to see if the stdlib calls open() without specifying an encoding. There were so many warnings, that the output was barely readable.
The warning would only be useful if there is a way to modify the code to make the warning quiet (fix the issue) without losing support with Python 3.9 and older.
I understand that open(filename) must be replaced with open(filename, encoding=("locale" if sys.version_info >= (3, 10) else None)) to make it backward and forward compatibility without emitting an EncodingWarning.
I think most of them must be replaced with encoding="ascii" or encoding="utf-8".
And encoding=locale.getpreferredencoding(False) is backward/forward compatible way. There is very little difference between encoding=None and encoding=locale.getpreferredencoding(False). But it is not a problem for most use cases. Only applications using PYTHONLEGACYWINDOWSSTDIO and open() for console I/O are affected by difference between them.
One issue is that some people may blindly copy/paste this code pattern without thinking if "locale" is the proper encoding.
Isn't it same if the code pattern become `encoding=getattr(io, "LOCALE_ENCODING", None)`, or `encoding=locale.getpreferredencoding(False)`?
I think only we can do is documenting the option like this:
""" EncodingWarning is warning to find missing encoding="utf-8" option. It is common pitfall that many Windows user Don't try to fix them if you need to use locale specific encoding. """
On Tue, 9 Feb 2021 at 11:55, Inada Naoki songofacandy@gmail.com wrote:
I think only we can do is documenting the option like this:
""" EncodingWarning is warning to find missing encoding="utf-8" option. It is common pitfall that many Windows user Don't try to fix them if you need to use locale specific encoding. """
I'm a very strong -1 on having programs generate warnings that the user isn't supposed to fix. If we can't provide a good recommendation to the user on what to do, we shouldn't be warning them that what they are currently doing is wrong. I've seen far too many examples of people thinking "well, users can ignore the warning, it's not shown by default" and then users' code being broken because of a situation we didn't think about (most recently, the Python test suite, which runs the venv tests with warnings converted to errors, which broke on a pip release that contains a deprecation warning from packaging).
IMO, if we issue a warning, we *must* be able to advise the user how to fix it. Otherwise we shouldn't be assuming we know what's correct better than the user.
Personally, I'm not at all keen on the idea of making users always specify encoding in the first place, even if it's "just for the transition". There are far too many people in my experience who wouldn't have a clue what to do when faced with that decision. And the people (again in my experience) who don't know how to make that choice are *precisely* the people for whom the system-defined default is what they want. Certainly, if they are getting stuff off the internet, they will more often get UTF-8, but I tend to find that people with limited understanding of these issues are much more comfortable with the idea that "stuff off the internet needs weird settings like this UTF-8 thing whatever it is", than they are with the idea that they have to tell Python how to read that text file they just got from their boss, who's still got Windows 7 on his PC...
If we want to switch the default encoding from the locale encoding to UTF-8, we should find a way to do that which *doesn't* mean that there's a "transitional" state where using the default is considered bad practice. That helps no-one, and just adds confusion, which will last far longer than that one release (there will be people encountering StackOverflow questions on the topic long after the default has changed).
Maybe we just have to accept that we can't work out what people are intending, and just state in advance in the documentation that the default will change, then it's documented as an upcoming breaking change that people can address (if they read the release notes, but we seem to be assuming they'll spot a warning, so why not assume they read the release notes, too?).
Paul
PS I've hesitated about saying this before, as I'm very aware that being from the UK, any problems I have with encodings are relatively minor, so I want to let the people with real problems have their say. But when we're talking about telling users not to fix warnings, I feel the need to speak up.
On Tue, Feb 9, 2021 at 1:31 PM Paul Moore p.f.moore@gmail.com wrote:
If we can't provide a good recommendation to the user on what to do, we shouldn't be warning them that what they are currently doing is wrong.
The warning can explicitly suggest to use encoding="utf8", it should work in almost all cases.
Victor
But people who currently don't specify the encoding, and *don't* have any issue (because the system locale is correct) will be getting told to introduce a bug into their code, if they follow that advice :-(
Paul
On Tue, 9 Feb 2021 at 16:03, Victor Stinner vstinner@python.org wrote:
On Tue, Feb 9, 2021 at 1:31 PM Paul Moore p.f.moore@gmail.com wrote:
If we can't provide a good recommendation to the user on what to do, we shouldn't be warning them that what they are currently doing is wrong.
The warning can explicitly suggest to use encoding="utf8", it should work in almost all cases.
Victor
Night gathers, and now my watch begins. It shall not end until my death.
On Wed, Feb 10, 2021 at 1:19 AM Paul Moore p.f.moore@gmail.com wrote:
But people who currently don't specify the encoding, and *don't* have any issue (because the system locale is correct) will be getting told to introduce a bug into their code, if they follow that advice :-(
This warning is opt-in warning like BytesWarning.
It will be a good tool to find potential problems for people knows what is the problem. But it is not recommended for users who don't understand what is the problem.
On Tue, 9 Feb 2021 at 16:28, Inada Naoki songofacandy@gmail.com wrote:
On Wed, Feb 10, 2021 at 1:19 AM Paul Moore p.f.moore@gmail.com wrote:
But people who currently don't specify the encoding, and *don't* have any issue (because the system locale is correct) will be getting told to introduce a bug into their code, if they follow that advice :-(
This warning is opt-in warning like BytesWarning.
It will be a good tool to find potential problems for people knows what is the problem. But it is not recommended for users who don't understand what is the problem.
Ah, OK. I missed that point in the long email chain. Sorry. Paul
On Tue, Feb 9, 2021 at 9:31 PM Paul Moore p.f.moore@gmail.com wrote:
Personally, I'm not at all keen on the idea of making users always specify encoding in the first place, even if it's "just for the transition".
I agree with you. But as I wrote in the PEP, omitted encoding caused much troubles already. Windows users can not just `pip install somepkg` because some library authors write `long_description=open("README.md").read()` in setup.py.
I am trying to fix this situation by two parallel approaches:
* (This PEP) Provide a tool for finding this type of bugs, and recommend `encoding="utf-8"` for cross-platform library authors. * (Author thread) Make UTF-8 mode more usable for Windows users, especially students.
If we want to switch the default encoding from the locale encoding to UTF-8, we should find a way to do that which *doesn't* mean that there's a "transitional" state where using the default is considered bad practice. That helps no-one, and just adds confusion, which will last far longer than that one release (there will be people encountering StackOverflow questions on the topic long after the default has changed).
Maybe we just have to accept that we can't work out what people are intending, and just state in advance in the documentation that the default will change, then it's documented as an upcoming breaking change that people can address (if they read the release notes, but we seem to be assuming they'll spot a warning, so why not assume they read the release notes, too?).
This PEP doesn't cover how to change the default encoding. So this is slightly off topic. I have two ideas for changing the default encoding:
(a) Regular deprecation period: Emitting EncodingWarning by default (3.14 or later), and change the default encoding later (3.17 or later). (b) Enable UTF-8 mode default on Windows. Users can disable UTF-8 mode for backward compatibility.
Steve Dower againsted to (b) very strongly. He suggested to emit DeprecationWarning. https://discuss.python.org/t/pep-597-enable-utf-8-mode-by-default-on-windows...
On the other hand, some core-dev don't like emitting Warning for all omitted `encoding` option.
So I don't have strong opinion about which approach is better. I want to see how EncodingWarning and UTF-8 mode are adopted.
I want to implement both EncodingWarning and per-site UTF-8 mode setting in Python 3.10. 5+ years later, we will see which approach is adopted by users.
* If EncodingWarning is widely adopted by many developers, we can discuss approach (a). * If UTF-8 mode becomes the best practice for Windows users, we can discuss approach (b).
Regards,
On Tue, 9 Feb 2021 at 16:54, Inada Naoki songofacandy@gmail.com wrote:
On Tue, Feb 9, 2021 at 9:31 PM Paul Moore p.f.moore@gmail.com wrote:
Personally, I'm not at all keen on the idea of making users always specify encoding in the first place, even if it's "just for the transition".
I agree with you. But as I wrote in the PEP, omitted encoding caused much troubles already. Windows users can not just `pip install somepkg` because some library authors write `long_description=open("README.md").read()` in setup.py.
I am trying to fix this situation by two parallel approaches:
- (This PEP) Provide a tool for finding this type of bugs, and
recommend `encoding="utf-8"` for cross-platform library authors.
- (Author thread) Make UTF-8 mode more usable for Windows users,
especially students.
Thanks for explaining (again). There's so much debate, across multiple proposals, that I can barely follow it. I'm impressed that you're managing to keep things straight at all :-)
I guess my views on this PEP come down to
* I see no harm in having a tool that helps developers spot platform-specific assumptions about encoding. * Realistically, I'd be surprised if developers actually use such a tool. If they were likely to do so, they could probably just as easily locate all the uses of open() in their code, and check that way. So I'm not sure this proposal is actually worth it, even if the end result would be very beneficial. * In the setup.py case, why don't those same Windows users complain that the library fails to install? A quick bug report, followed by a simple fix, seems more likely to happen than the developer suddenly deciding to scan their code for encoding issues.
Regarding the wider question of UTF8 as default, my views can probably be summarised as follows:
* If you want to write correct code to deal with encodings, there is no substitute for carefully considering every bytes/string conversion, deciding how you are going to identify the encoding to use, and then specifying that encoding explicitly. Default values for encodings have no place in such code. * In reality, though, that's far too much work for many situations. Default encodings are a necessary convenience, particularly for simple scripts, or for people who can't, or don't want to, do the analysis that the "correct" approach implies. * Picking the right default is *hard*. Changing the default is even harder, unfortunately. * I feel that we already have a number of mechanisms (PEPs 538 and 540) trying to tackle this issue. Adding yet more suggests to me that we'd be better off pausing and working out why we still have an issue. We should be moving towards *fewer* mechanisms, not more. * We have UTF-8 mode, and users can set it per-process (via flag or environment variable) per-user or per-site (by environment variable). I don't honestly believe that a user (whatever OS they work on) who is capable of writing Python code, can't be shown how to set an environment variable. I see no reason to suggest we need yet another way to set UTF-8 mode, or that a per-interpreter or per-virtualenv setting is particularly crucial (suggestions that have been made in the Python-Ideas threads). * UTF-8 is likely to be the most appropriate default encoding for Python in the longer term, and I agree that Windows is fast approaching the point where a UTF-8 encoding is more appropriate than the ANSI codepage for "new stuff". But there's a lot of legacy files and applications around, and I suspect that a UTF-8 default will inconvenience a lot of people working with such data. But equally, such people may not be in a huge rush to switch to the latest Python version. Whichever way we go, though, some people will be inconvenienced.
I'm also somewhat bemused by the rather negative view of "Windows beginners" that lies behind a lot of these discussions. People's experiences may well differ, but the people I see using (and learning) Python on Windows are often experienced computer users, maybe developers with significant experience in Java or other "enterprise languages", or data scientists who have a lot of knowledge of computers, but are relatively new to programming. Or systems admins, or database specialists, who want to use Python to write scripts on Windows. None of those people fit the picture of people who wouldn't know how to set an environment variable, or configure their environment. On the other hand, (in my experience) they often don't really have much knowledge of character encodings, and tend to just use whatever default their PC uses, and expect it to work. They *can*, however, understand when an encoding problem is explained to them, and can set an explicit encoding once they know they need to.
Paul
On Wed, Feb 10, 2021 at 5:50 AM Paul Moore p.f.moore@gmail.com wrote:
On Tue, 9 Feb 2021 at 16:54, Inada Naoki songofacandy@gmail.com wrote:
On Tue, Feb 9, 2021 at 9:31 PM Paul Moore p.f.moore@gmail.com wrote:
Personally, I'm not at all keen on the idea of making users always specify encoding in the first place, even if it's "just for the transition".
I agree with you. But as I wrote in the PEP, omitted encoding caused much troubles already. Windows users can not just `pip install somepkg` because some library authors write `long_description=open("README.md").read()` in setup.py.
I am trying to fix this situation by two parallel approaches:
- (This PEP) Provide a tool for finding this type of bugs, and
recommend `encoding="utf-8"` for cross-platform library authors.
- (Author thread) Make UTF-8 mode more usable for Windows users,
especially students.
Thanks for explaining (again). There's so much debate, across multiple proposals, that I can barely follow it. I'm impressed that you're managing to keep things straight at all :-)
I guess my views on this PEP come down to
- I see no harm in having a tool that helps developers spot
platform-specific assumptions about encoding.
- Realistically, I'd be surprised if developers actually use such a
tool. If they were likely to do so, they could probably just as easily locate all the uses of open() in their code, and check that way. So I'm not sure this proposal is actually worth it, even if the end result would be very beneficial.
- In the setup.py case, why don't those same Windows users complain
that the library fails to install? A quick bug report, followed by a simple fix, seems more likely to happen than the developer suddenly deciding to scan their code for encoding issues.
Yes, some issues are solved already. On the other hand, there are dozen question about UnicodeDecodeError in Q&A sites like Stack Overflow. Many people don't know what the error means, and how to report it correctly.
I sometime set PYTHONWARNINGS=deafult in my bashrc and find DeprecationWarnings in libraries I am using, and report them.
On the other hand, it is difficult to find omitted `encoding="utf-8"`, because I use macOS and Linux in daily development. If there is PYTHONWARNENCODING, I can write `export PYTHONWARNENCODING=1` in my .bashrc.
Regarding the wider question of UTF8 as default, my views can probably be summarised as follows:
- If you want to write correct code to deal with encodings, there is
no substitute for carefully considering every bytes/string conversion, deciding how you are going to identify the encoding to use, and then specifying that encoding explicitly. Default values for encodings have no place in such code.
- In reality, though, that's far too much work for many situations.
Default encodings are a necessary convenience, particularly for simple scripts, or for people who can't, or don't want to, do the analysis that the "correct" approach implies.
Yes. and the UTF-8 is the default encoding for s.encode() already.
- Picking the right default is *hard*. Changing the default is even
harder, unfortunately.
- I feel that we already have a number of mechanisms (PEPs 538 and
- trying to tackle this issue. Adding yet more suggests to me that
we'd be better off pausing and working out why we still have an issue. We should be moving towards *fewer* mechanisms, not more.
- We have UTF-8 mode, and users can set it per-process (via flag or
environment variable) per-user or per-site (by environment variable). I don't honestly believe that a user (whatever OS they work on) who is capable of writing Python code, can't be shown how to set an environment variable. I see no reason to suggest we need yet another way to set UTF-8 mode, or that a per-interpreter or per-virtualenv setting is particularly crucial (suggestions that have been made in the Python-Ideas threads).
Note that many Python users don't use consoles. They just starts Jupyter Notebook, or they just write .py file and run it in the Minecraft mods.
- UTF-8 is likely to be the most appropriate default encoding for
Python in the longer term, and I agree that Windows is fast approaching the point where a UTF-8 encoding is more appropriate than the ANSI codepage for "new stuff". But there's a lot of legacy files and applications around, and I suspect that a UTF-8 default will inconvenience a lot of people working with such data. But equally, such people may not be in a huge rush to switch to the latest Python version. Whichever way we go, though, some people will be inconvenienced.
I'm also somewhat bemused by the rather negative view of "Windows beginners" that lies behind a lot of these discussions. People's experiences may well differ, but the people I see using (and learning) Python on Windows are often experienced computer users, maybe developers with significant experience in Java or other "enterprise languages", or data scientists who have a lot of knowledge of computers, but are relatively new to programming. Or systems admins, or database specialists, who want to use Python to write scripts on Windows. None of those people fit the picture of people who wouldn't know how to set an environment variable, or configure their environment. On the other hand, (in my experience) they often don't really have much knowledge of character encodings, and tend to just use whatever default their PC uses, and expect it to work. They *can*, however, understand when an encoding problem is explained to them, and can set an explicit encoding once they know they need to.
Paul
Of course, it is not a problem for experts.
But Python is very widely used for learning Programming. There are many Programming courses for Junior high school students, or even for elementary school students. They may start learning Python on the browser. When they installed Python and run the program worked on the browser, it won't work because of encoding issue...
On 2/9/2021 8:28 PM, Inada Naoki wrote:
Note that many Python users don't use consoles.
Those of use who do may find it hard to imagine just how easy we have made computing.
My daughter minored in Computer Science about 6 years ago. She never saw Command Prompt until the summer after her Junior year when she watched me use it to install numpy and other packages for her. I had to do it because 'Run pip install numpy', etc, was met with a blank stare. I had taught her Python with IDLE, downloaded and install with a browser, and had neglected to teach her 'Dos' until then.
So had her CS classes. Those previous used Racket in a Dr. something environment and Java in, I believe, Eclipse. Also downloaded and installed with a browser.
They just starts Jupyter Notebook, or they just write .py file and run it in the Minecraft mods.
Also similar.
On Tue, Feb 9, 2021 at 11:29 PM Terry Reedy tjreedy@udel.edu wrote:
On 2/9/2021 8:28 PM, Inada Naoki wrote:
Note that many Python users don't use consoles.
Those of use who do may find it hard to imagine just how easy we have made computing.
My daughter minored in Computer Science about 6 years ago. She never saw Command Prompt until the summer after her Junior year when she watched me use it to install numpy and other packages for her. I had to do it because 'Run pip install numpy', etc, was met with a blank stare. I had taught her Python with IDLE, downloaded and install with a browser, and had neglected to teach her 'Dos' until then.
So had her CS classes. Those previous used Racket in a Dr. something environment and Java in, I believe, Eclipse. Also downloaded and installed with a browser.
Speaking as a current CS undergraduate student here (senior graduating in December 2021). At my university, the freshman/sophomore-level programming classes do not assume or expect any type of command line knowledge. They all rely on GUI tools (Eclipse, IntelliJ, or NetBeans for the freshman Java courses, Visual Studio for Data Structures in C++).
There is one course, typically taken in either the second or third semester for traditional students, called Operating Systems Concepts and Usage, that broadly discusses how operating systems function, but is also designed as a first introduction to Linux and to the command line. (Until this point, the only operating system students are assumed to be familiar with is Windows.) For many students, this course is their first ever exposure to the command prompt.
After that, students in this program don't generally *need* to touch the command line again in their studies until they hit 4000-level courses, and even then only a few courses require it. Outside of that one introductory course, I've only had two courses so far that actually required command line usage. Everything else so far has offered GUI options, even many upper level courses.
I think it's a disservice to fail to expose students to the command line more and earlier, but the fact is, that failure happens and happens often, and developers need to be conscious of that.
Despite my own ease and comfort with the command-line (which dates back to learning my way around DOS at the age of 5) to the point of almost always having a terminal window open on my daily Debian machine, I frequently find myself opting for point-and-click solutions to common problems, even Git operations (which are so easy and powerful in VS Code with the GitLens extension). GUI tools grow more powerful by the day, and it's very easy to get deep into a computer science program these days and not be comfortable with the command line and/or not know how to change environment variables.
Python, as a common introductory language used by many thousands of people who have never taken a university computer course, never mind majoring in computer science, shouldn't have basic features that depend on the likely false assumption that the user has ever seen a command prompt or an environment variable, much less comprehend how to use them.
On Wed, 10 Feb 2021 at 01:29, Inada Naoki songofacandy@gmail.com wrote:
Note that many Python users don't use consoles.
I never suggested that they did. There's a GUI for setting user-level and system-level environment variables. And whoever is introducing the user to Python can show them how to set the necessary environment variable - or do it for them.
Please be clear, I'm not saying I don't understand the difficulties. But I do question why PYTHONUTF8 is so much different than all the other environment variables that Python responds to that it needs special additional options.
Remember - I've already said that I'm not convinced that setting UTF8 mode globally is the right approach. So what you're saying in effect is that you want to convince me that we should add a new mechanism to globally set an option that I don't believe should be set globally. Let's just assume until you can convince me that setting UTF-8 mode globally is a good idea, there's no point trying to convince me that we need a new mechanism to do so because environment variables aren't good enough.
Paul
On Wed, Feb 10, 2021 at 5:00 PM Paul Moore p.f.moore@gmail.com wrote:
Let's just assume until you can convince me that setting UTF-8 mode globally is a good idea,
Oh, you misunderstood me. My proposal is not setting UTF-8 mode globally. What I proposed is setting UTF-8 mode per env (e.g. installation, venv, or conda env).
But this is off topic. The thread for this topic is here. https://mail.python.org/archives/list/python-ideas@python.org/thread/LQVK2UK...
Regards,
On Tue, Feb 9, 2021 at 9:51 PM Paul Moore p.f.moore@gmail.com wrote:
- Realistically, I'd be surprised if developers actually use such a
tool. If they were likely to do so, they could probably just as easily locate all the uses of open() in their code, and check that way. So I'm not sure this proposal is actually worth it, even if the end result would be very beneficial.
That's ok, they are many Python features which are only used by a minority of users. For me it's similar to the Python Development Mode: https://docs.python.org/dev/library/devmode.html
Most users and developers will never use it, but for developers who care, it's a useful tool (it ease the discovery of issues).
Victor
Who will benefit from this new warning?
Is this basically just changing builtins.open by adding:
if encoding is None and sys.flags.encoding_warning: # and not Android and not -X utf8 ? warnings.warn(EncodingWarning("Are you sure you want locale instead of utf-8?"))
Even for the few people with the knowledge, time, interest, and authority to fix the code, is that really helpful?
Helpful enough to put it directly in python as an optional mode, separate from the dev mode or show all warnings mode? Why not just add it to a linter, or write a 2to3 style checker? Or at least emit or not based on a warnings filter?
-jJ
On Thu, 11 Feb 2021 at 21:05, Jim J. Jewett jimjjewett@gmail.com wrote:
Who will benefit from this new warning?
Is this basically just changing builtins.open by adding:
if encoding is None and sys.flags.encoding_warning: # and not Android and not -X utf8 ? warnings.warn(EncodingWarning("Are you sure you want locale instead of utf-8?"))Even for the few people with the knowledge, time, interest, and authority to fix the code, is that really helpful?
Helpful enough to put it directly in python as an optional mode, separate from the dev mode or show all warnings mode? Why not just add it to a linter, or write a 2to3 style checker? Or at least emit or not based on a warnings filter?
That's a very good point. If this warning is of use, why have none of the well-known linters implemented it? And why not prototype the proposal in them, at least? Python-ideas posts routinely get pushed to justify "why can't this be done in an external library?" and that probably applies here too.
Paul
On Fri, Feb 12, 2021 at 6:34 AM Paul Moore p.f.moore@gmail.com wrote:
On Thu, 11 Feb 2021 at 21:05, Jim J. Jewett jimjjewett@gmail.com wrote:
Who will benefit from this new warning?
Is this basically just changing builtins.open by adding:
if encoding is None and sys.flags.encoding_warning: # and not Android and not -X utf8 ? warnings.warn(EncodingWarning("Are you sure you want locale instead of utf-8?"))Even for the few people with the knowledge, time, interest, and authority to fix the code, is that really helpful?
Helpful enough to put it directly in python as an optional mode, separate from the dev mode or show all warnings mode? Why not just add it to a linter, or write a 2to3 style checker? Or at least emit or not based on a warnings filter?
That's a very good point. If this warning is of use, why have none of the well-known linters implemented it? And why not prototype the proposal in them, at least? Python-ideas posts routinely get pushed to justify "why can't this be done in an external library?" and that probably applies here too.
* Linters can not add `encoding="locale"` to Python. * This PEP provides the way to shift where warnings is emitted.
def my_read_file(filename, encoding=None): encoding = io.text_encoding(encoding) with open(filename, encoding=encoding) with f: return f.read()
This function is not warned. Caller of this function is warned instead. It is difficult to implement in the Linter.
To demonstrate how this warning is useful, I used my reference implementation.
When I try `pip install`, I found these issues soon.
https://bugs.python.org/issue43214 (Open pth file with locale-encoding) https://github.com/pypa/pip/pull/9608 (Not a real bug, but open JSON file with locale-encoding)
And when creating a PR for pip, I found this issue in tox:
https://github.com/tox-dev/tox/issues/1908 (Open toml file with locale-encoding, may not work on Windows)
Although most developers won't use this option, I and few other developers can put `export PYTHONWARNENCODING=1` in .bashrc and will find many possible bugs that happen only on Windows, even if they don't use Windows daily development.
Isn't this option worth enough?
participants (6)
-
Inada Naoki -
Jim J. Jewett -
Jonathan Goble -
Paul Moore -
Terry Reedy -
Victor Stinner