Mailman 3 December 2017 - Python-Dev

"CPython loves your Pull Requests" talk by Stéphane Wirtel
by Victor Stinner Dec. 6, 2017

Dec. 6, 2017

Hi, Stéphane Wirtel gave a talk last month at Pycon CA about CPython pull requests. His slides: https://speakerdeck.com/matrixise/cpython-loves-your-pull-requests He produced interesting statistics that we didn't have before on pull requests (PR), from February 2017 to October 2017: * total number of merged PR: 4204 * number of contributors: 586 !!! (96%) * number of core developers: 27 (4%) * Time to merge a PR: 3 days in average, good! * etc. It would be nice to get these statistics updated regularly on a service running somewhere. By the way, I'm also looking for statistics on reviews on GitHub. Does someone know how to do that? Victor

7 7

[RELEASE] Python 3.6.4rc1 and 3.7.0a3 now available for testing
by Ned Deily Dec. 6, 2017

Dec. 6, 2017

Announcing the immediate availability of Python 3.6.4 release candidate 1 and of Python 3.7.0 alpha 3! Python 3.6.4rc1 is the first release candidate for Python 3.6.4, the next maintenance release of Python 3.6. While 3.6.4rc1 is a preview release and, thus, not intended for production environments, we encourage you to explore it and provide feedback via the Python bug tracker (https://bugs.python.org). 3.6.4 is planned for final release on 2017-12-18 with the next maintenance release expected to follow in about 3 months. You can find Python 3.6.4rc1 and more information here: https://www.python.org/downloads/release/python-364rc1/ Python 3.7.0a3 is the third of four planned alpha releases of Python 3.7, the next feature release of Python. During the alpha phase, Python 3.7 remains under heavy development: additional features will be added and existing features may be modified or deleted. Please keep in mind that this is a preview release and its use is not recommended for production environments. The next preview release, 3.7.0a4, is planned for 2018-01-08. You can find Python 3.7.0a3 and more information here: https://www.python.org/downloads/release/python-370a3/ -- Ned Deily nad(a)python.org -- []

1 0

PEP 540: Add a new UTF-8 mode
by Victor Stinner Dec. 5, 2017

Dec. 5, 2017

Hi, Since it's the PEP Acceptance Week, I try my luck! Here is my very long PEP to propose a tiny change. The PEP is very long to explain the rationale and limitations. Inaccurate tl; dr with the UTF-8 mode, Unicode "just works" as expected. Reminder: INADA Naoki was nominated as the BDFL-Delegate. https://www.python.org/dev/peps/pep-0540/ Full-text below. Victor PEP: 540 Title: Add a new UTF-8 mode Version: $Revision$ Last-Modified: $Date$ Author: Victor Stinner <victor.stinner(a)gmail.com>, Nick Coghlan <ncoghlan(a)gmail.com> BDFL-Delegate: INADA Naoki Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 5-January-2016 Python-Version: 3.7 Abstract ======== Add a new UTF-8 mode, enabled by default in the POSIX locale, to ignore the locale and force the usage of the UTF-8 encoding for external operating system interfaces, including the standard IO streams. Essentially, the UTF-8 mode behaves as Python 2 and other C based applications on \*nix systems: it aims to process text as best it can, but it errs on the side of producing or propagating mojibake to subsequent components in a processing pipeline rather than requiring strictly valid encodings at every step in the process. The UTF-8 mode can be configured as strict to reduce the risk of producing or propagating mojibake. A new ``-X utf8`` command line option and ``PYTHONUTF8`` environment variable are added to explicitly control the UTF-8 mode (including turning it off entirely, even in the POSIX locale). Rationale ========= "It's not a bug, you must fix your locale" is not an acceptable answer ---------------------------------------------------------------------- Since Python 3.0 was released in 2008, the usual answer to users getting Unicode errors is to ask developers to fix their code to handle Unicode properly. Most applications and Python modules were fixed, but users kept reporting Unicode errors regularly: see the long list of issues in the `Links`_ section below. In fact, a second class of bugs comes from a locale which is not properly configured. The usual answer to such a bug report is: "it is not a bug, you must fix your locale". Technically, the answer is correct, but from a practical point of view, the answer is not acceptable. In many cases, "fixing the issue" is a hard task. Moreover, sometimes, the usage of the POSIX locale is deliberate. A good example of a concrete issue are build systems which create a fresh environment for each build using a chroot, a container, a virtual machine or something else to get reproducible builds. Such a setup usually uses the POSIX locale. To get 100% reproducible builds, the POSIX locale is a good choice: see the `Locales section of reproducible-builds.org <https://reproducible-builds.org/docs/locales/>`_. PEP 538 lists additional problems related to the use of Linux containers to run network services and command line applications. UNIX users don't expect Unicode errors, since the common command lines tools like ``cat``, ``grep`` or ``sed`` never fail with Unicode errors - they produce mostly-readable text instead. These users similarly expect that tools written in Python 3 (including those updated from Python 2), continue to tolerate locale misconfigurations and avoid bothering them with text encoding details. >From their point of the view, the bug is not their locale but is obviously Python 3 ("Everything else works, including Python 2, so what's wrong with Python 3?"). Since Python 2 handles data as bytes, similar to system utilities written in C and C++, it's rarer in Python 2 compared to Python 3 to get explicit Unicode errors. It also contributes significantly to why many affected users perceive Python 3 as the root cause of their Unicode errors. At the same time, the stricter text handling model was deliberately introduced into Python 3 to reduce the frequency of data corruption bugs arising in production services due to mismatched assumptions regarding text encodings. It's one thing to emit mojibake to a user's terminal while listing a directory, but something else entirely to store that in a system manifest in a database, or to send it to a remote client attempting to retrieve files from the system. Since different group of users have different expectations, there is no silver bullet which solves all issues at once. Last but not least, backward compatibility should be preserved whenever possible. Locale and operating system data -------------------------------- .. _operating system data: Python uses an encoding called the "filesystem encoding" to decide how to encode and decode data from/to the operating system: * file content * command line arguments: ``sys.argv`` * standard streams: ``sys.stdin``, ``sys.stdout``, ``sys.stderr`` * environment variables: ``os.environ`` * filenames: ``os.listdir(str)`` for example * pipes: ``subprocess.Popen`` using ``subprocess.PIPE`` for example * error messages: ``os.strerror(code)`` for example * user and terminal names: ``os``, ``grp`` and ``pwd`` modules * host name, UNIX socket path: see the ``socket`` module * etc. At startup, Python calls ``setlocale(LC_CTYPE, "")`` to use the user ``LC_CTYPE`` locale and then store the locale encoding as the "filesystem error". It's possible to get this encoding using ``sys.getfilesystemencoding()``. In the whole lifetime of a Python process, the same encoding and error handler are used to encode and decode data from/to the operating system. The ``os.fsdecode()`` and ``os.fsencode()`` functions can be used to decode and encode operating system data. These functions use the filesystem error handler: ``sys.getfilesystemencodeerrors()``. .. note:: In some corner cases, the *current* ``LC_CTYPE`` locale must be used instead of ``sys.getfilesystemencoding()``. For example, the ``time`` module uses the *current* ``LC_CTYPE`` locale to decode timezone names. The POSIX locale and its encoding --------------------------------- The following environment variables are used to configure the locale, in this preference order: * ``LC_ALL``, most important variable * ``LC_CTYPE`` * ``LANG`` The POSIX locale, also known as "the C locale", is used: * if the first set variable is set to ``"C"`` * if all these variables are unset, for example when a program is started in an empty environment. The encoding of the POSIX locale must be ASCII or a superset of ASCII. On Linux, the POSIX locale uses the ASCII encoding. On FreeBSD and Solaris, ``nl_langinfo(CODESET)`` announces an alias of the ASCII encoding, whereas ``mbstowcs()`` and ``wcstombs()`` functions use the ISO 8859-1 encoding (Latin1) in practice. The problem is that ``os.fsencode()`` and ``os.fsdecode()`` use ``locale.getpreferredencoding()`` codec. For example, if command line arguments are decoded by ``mbstowcs()`` and encoded back by ``os.fsencode()``, an ``UnicodeEncodeError`` exception is raised instead of retrieving the original byte string. To fix this issue, Python checks since Python 3.4 if ``mbstowcs()`` really uses the ASCII encoding if the the ``LC_CTYPE`` uses the the POSIX locale and ``nl_langinfo(CODESET)`` returns ``"ASCII"`` (or an alias to ASCII). If not (the effective encoding is not ASCII), Python uses its own ASCII codec instead of using ``mbstowcs()`` and ``wcstombs()`` functions for `operating system data`_. See the `POSIX locale (2016 Edition) <http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap07.html>`_. POSIX locale used by mistake ---------------------------- In many cases, the POSIX locale is not really expected by users who get it by mistake. Examples: * program started in an empty environment * User forcing LANG=C to get messages in English * LANG=C used for bad reasons, without being aware of the ASCII encoding * SSH shell * Linux installed with no configured locale * chroot environment, Docker image, container, ... with no locale is configured * User locale set to a non-existing locale, typo in the locale name for example C.UTF-8 and C.utf8 locales -------------------------- Some UNIX operating systems provide a variant of the POSIX locale using the UTF-8 encoding: * Fedora 25: ``"C.utf8"`` or ``"C.UTF-8"`` * Debian (eglibc 2.13-1, 2011), Ubuntu: ``"C.UTF-8"`` * HP-UX: ``"C.utf8"`` It was proposed to add a ``C.UTF-8`` locale to the glibc: `glibc C.UTF-8 proposal <https://sourceware.org/glibc/wiki/Proposals/C.UTF-8>`_. It is not planned to add such locale to BSD systems. Popularity of the UTF-8 encoding -------------------------------- Python 3 uses UTF-8 by default for Python source files. On Mac OS X, Windows and Android, Python always use UTF-8 for operating system data. For Windows, see the `PEP 529`_: "Change Windows filesystem encoding to UTF-8". On Linux, UTF-8 became the de facto standard encoding, replacing legacy encodings like ISO 8859-1 or ShiftJIS. For example, using different encodings for filenames and standard streams is likely to create mojibake, so UTF-8 is now used *everywhere* (at least for modern distributions using their default settings). The UTF-8 encoding is the default encoding of XML and JSON file format. In January 2017, UTF-8 was used in `more than 88% of web pages <https://w3techs.com/technologies/details/en-utf8/all/all>`_ (HTML, Javascript, CSS, etc.). See `utf8everywhere.org <http://utf8everywhere.org/>`_ for more general information on the UTF-8 codec. .. note:: Some applications and operating systems (especially Windows) use Byte Order Markers (BOM) to indicate the used Unicode encoding: UTF-7, UTF-8, UTF-16-LE, etc. BOM are not well supported and rarely used in Python. Old data stored in different encodings and surrogateescape ---------------------------------------------------------- Even if UTF-8 became the de facto standard, there are still systems in the wild which don't use UTF-8. And there are a lot of data stored in different encodings. For example, an old USB key using the ext3 filesystem with filenames encoded to ISO 8859-1. The Linux kernel and libc don't decode filenames: a filename is used as a raw array of bytes. The common solution to support any filename is to store filenames as bytes and don't try to decode them. When displayed to stdout, mojibake is displayed if the filename and the terminal don't use the same encoding. Python 3 promotes Unicode everywhere including filenames. A solution to support filenames not decodable from the locale encoding was found: the ``surrogateescape`` error handler (`PEP 383`_), store undecodable bytes as surrogate characters. This error handler is used by default for `operating system data`_, by ``os.fsdecode()`` and ``os.fsencode()`` for example (except on Windows which uses the ``strict`` error handler). Standard streams ---------------- Python uses the locale encoding for standard streams: stdin, stdout and stderr. The ``strict`` error handler is used by stdin and stdout to prevent mojibake. The ``backslashreplace`` error handler is used by stderr to avoid Unicode encode errors when displaying non-ASCII text. It is especially useful when the POSIX locale is used, because this locale usually uses the ASCII encoding. The problem is that `operating system data`_ like filenames are decoded using the ``surrogateescape`` error handler (`PEP 383`_). Displaying a filename to stdout raises a Unicode encode error if the filename contains an undecoded byte stored as a surrogate character. Python 3.5+ now uses ``surrogateescape`` for stdin and stdout if the POSIX locale is used: `issue #19977 <http://bugs.python.org/issue19977>`_. The idea is to pass through `operating system data`_ even if it means mojibake, because most UNIX applications work like that. Such UNIX applications often store filenames as bytes, in many cases because their basic design principles (or those of the language they're implemented in) were laid down half a century ago when it was still a feat for computers to handle English text correctly, rather than humans having to work with raw numeric indexes. .. note:: The encoding and/or the error handler of standard streams can be overriden with the ``PYTHONIOENCODING`` environment variable. Proposal ======== Changes ------- Add a new UTF-8 mode, enabled by default in the POSIX locale, but otherwise disabled by default, to ignore the locale and force the usage of the UTF-8 encoding with the ``surrogateescape`` error handler, instead using the locale encoding (with ``strict`` or ``surrogateescape`` error handler depending on the case). The "normal" UTF-8 mode uses ``surrogateescape`` on the standard input and output streams and opened files, as well as on all operating system interfaces. This is the mode implicitly activated by the POSIX locale. The "strict" UTF-8 mode reduces the risk of producing or propogating mojibake: the UTF-8 encoding is used with the ``strict`` error handler for inputs and outputs, but the ``surrogateescape`` error handler is still used for `operating system data`_. This mode is never activated implicitly, but can be requested explicitly. The new ``-X utf8`` command line option and ``PYTHONUTF8`` environment variable are added to control the UTF-8 mode. The UTF-8 mode is enabled by ``-X utf8`` or ``PYTHONUTF8=1``. The UTF-8 Strict mode is configured by ``-X utf8=strict`` or ``PYTHONUTF8=strict``. The POSIX locale enables the UTF-8 mode. In this case, the UTF-8 mode can be explicitly disabled by ``-X utf8=0`` or ``PYTHONUTF8=0``. Other option values fail with an error. Options priority for the UTF-8 mode: * ``PYTHONLEGACYWINDOWSFSENCODING`` * ``-X utf8`` * ``PYTHONUTF8`` * POSIX locale For example, ``PYTHONUTF8=0 python3 -X utf8`` enables the UTF-8 mode, whereas ``LC_ALL=C python3.7 -X utf8=0`` disables the UTF-8 mode and so use the encoding of the POSIX locale. Encodings used by ``open()``, highest priority first: * *encoding* and *errors* parameters (if set) * UTF-8 mode * ``os.device_encoding(fd)`` * ``os.getpreferredencoding(False)`` Encoding and error handler -------------------------- The UTF-8 mode changes the default encoding and error handler used by ``open()``, ``os.fsdecode()``, ``os.fsencode()``, ``sys.stdin``, ``sys.stdout`` and ``sys.stderr``: ============================ ======================= ========================== ========================== Function Default UTF-8 mode or POSIX locale UTF-8 Strict mode ============================ ======================= ========================== ========================== open() locale/strict **UTF-8/surrogateescape** **UTF-8**/strict os.fsdecode(), os.fsencode() locale/surrogateescape **UTF-8**/surrogateescape **UTF-8**/surrogateescape sys.stdin, sys.stdout locale/strict **UTF-8/surrogateescape** **UTF-8**/strict sys.stderr locale/backslashreplace **UTF-8**/backslashreplace **UTF-8**/backslashreplace ============================ ======================= ========================== ========================== By comparison, Python 3.6 uses: ============================ ======================= ========================== Function Default POSIX locale ============================ ======================= ========================== open() locale/strict locale/strict os.fsdecode(), os.fsencode() locale/surrogateescape locale/surrogateescape sys.stdin, sys.stdout locale/strict locale/**surrogateescape** sys.stderr locale/backslashreplace locale/backslashreplace ============================ ======================= ========================== The UTF-8 mode uses the ``surrogateescape`` error handler instead of the strict mode for consistency with other standard \*nix operating system components: the idea is that data not encoded to UTF-8 are passed through "Python" without being modified, as raw bytes. The ``PYTHONIOENCODING`` environment variable has priority over the UTF-8 mode for standard streams. For example, ``PYTHONIOENCODING=latin1 python3 -X utf8`` uses the Latin1 encoding for stdin, stdout and stderr. Encoding and error handler on Windows ------------------------------------- On Windows, the encodings and error handlers are different: ============================ ======================= ========================== ========================== ========================== Function Default Legacy Windows FS encoding UTF-8 mode UTF-8 Strict mode ============================ ======================= ========================== ========================== ========================== open() mbcs/strict mbcs/strict **UTF-8/surrogateescape** **UTF-8**/strict os.fsdecode(), os.fsencode() UTF-8/surrogatepass **mbcs/replace** UTF-8/surrogatepass UTF-8/surrogatepass sys.stdin, sys.stdout UTF-8/surrogateescape UTF-8/surrogateescape UTF-8/surrogateescape **UTF-8/strict** sys.stderr UTF-8/backslashreplace UTF-8/backslashreplace UTF-8/backslashreplace UTF-8/backslashreplace ============================ ======================= ========================== ========================== ========================== By comparison, Python 3.6 uses: ============================ ======================= ========================== Function Default Legacy Windows FS encoding ============================ ======================= ========================== open() mbcs/strict mbcs/strict os.fsdecode(), os.fsencode() UTF-8/surrogatepass **mbcs/replace** sys.stdin, sys.stdout UTF-8/surrogateescape UTF-8/surrogateescape sys.stderr UTF-8/backslashreplace UTF-8/backslashreplace ============================ ======================= ========================== The "Legacy Windows FS encoding" is enabled by setting the ``PYTHONLEGACYWINDOWSFSENCODING`` environment variable to ``1`` as specified in `PEP 529` . Enabling the legacy Windows filesystem encoding disables the UTF-8 mode (as ``-X utf8=0``). If stdin and/or stdout is redirected to a pipe, ``sys.stdin`` and/or ``sys.output`` use ``mbcs`` encoding by default rather than UTF-8. But with the UTF-8 mode, ``sys.stdin`` and ``sys.stdout`` always use the UTF-8 encoding. There is no POSIX locale on Windows. The ANSI code page is used to the locale encoding, and this code page never uses the ASCII encoding. Rationale --------- The UTF-8 mode is disabled by default to keep hard Unicode errors when encoding or decoding `operating system data`_ failed, and to keep the backward compatibility. The user is responsible to enable explicitly the UTF-8 mode, and so is better prepared for mojibake than if the UTF-8 mode would be enabled *by default*. The UTF-8 mode should be used on systems known to be configured with UTF-8 where most applications speak UTF-8. It prevents Unicode errors if the user overrides a locale *by mistake* or if a Python program is started with no locale configured (and so with the POSIX locale). Most UNIX applications handle `operating system data`_ as bytes, so ``LC_ALL``, ``LC_CTYPE`` and ``LANG`` environment variables have a limited impact on how these data are handled by the application. The Python UTF-8 mode should help to make Python more interoperable with the other UNIX applications in the system assuming that *UTF-8* is used everywhere and that users *expect* UTF-8. Ignoring ``LC_ALL``, ``LC_CTYPE`` and ``LANG`` environment variables in Python is more convenient, since they are more commonly misconfigured *by mistake* (configured to use an encoding different than UTF-8, whereas the system uses UTF-8), rather than being misconfigured by intent. Expected mojibake and surrogate character issues ------------------------------------------------ The UTF-8 mode only affects code running directly in Python, especially code written in pure Python. The other code, called "external code" here, is not aware of this mode. Examples: * C libraries called by Python modules like OpenSSL * The application code when Python is embedded in an application In the UTF-8 mode, Python uses the ``surrogateescape`` error handler which stores bytes not decodable from UTF-8 as surrogate characters. If the external code uses the locale and the locale encoding is UTF-8, it should work fine. External code using bytes ^^^^^^^^^^^^^^^^^^^^^^^^^ If the external code processes data as bytes, surrogate characters are not an issue since they are only used inside Python. Python encodes back surrogate characters to bytes at the edges, before calling external code. The UTF-8 mode can produce mojibake since Python and external code don't both of invalid bytes, but it's a deliberate choice. The UTF-8 mode can be configured as strict to prevent mojibake and fail early when data is not decodable from UTF-8 or not encodable to UTF-8. External code using text ^^^^^^^^^^^^^^^^^^^^^^^^ If the external code uses text API, for example using the ``wchar_t*`` C type, mojibake should not occur, but the external code can fail on surrogate characters. Use Cases ========= The following use cases were written to help to understand the impact of chosen encodings and error handlers on concrete examples. The "Exception?" column shows the potential benefit of having a UTF-8 mode which is closer to the traditional Python 2 behaviour of passing along raw binary data even if it isn't valid UTF-8. The "Mojibake" column shows that ignoring the locale causes a practical issue: the UTF-8 mode produces mojibake if the terminal doesn't use the UTF-8 encoding. The ideal configuration is "No exception, no risk of mojibake", but that isn't always possible in the presence of non-UTF-8 encoded binary data. List a directory into stdout ---------------------------- Script listing the content of the current directory into stdout:: import os for name in os.listdir(os.curdir): print(name) Result: ======================== ========== ========= Python Exception? Mojibake? ======================== ========== ========= Python 2 No **Yes** Python 3 **Yes** No Python 3.5, POSIX locale No **Yes** UTF-8 mode No **Yes** UTF-8 Strict mode **Yes** No ======================== ========== ========= "Exception?" means that the script can fail on decoding or encoding a filename depending on the locale or the filename. To be able to never fail that way, the program must be able to produce mojibake. For automated and interactive process, mojibake is often more user friendly than an error with a truncated or empty output, since it confines the problem to the affected entry, rather than aborting the whole task. Example with a directory which contains the file called ``b'xxx\xff'`` (the byte ``0xFF`` is invalid in UTF-8). Default and UTF-8 Strict mode fail on ``print()`` with an encode error:: $ python3.7 ../ls.py Traceback (most recent call last): File "../ls.py", line 5, in <module> print(name) UnicodeEncodeError: 'utf-8' codec can't encode character '\udcff' ... $ python3.7 -X utf8=strict ../ls.py Traceback (most recent call last): File "../ls.py", line 5, in <module> print(name) UnicodeEncodeError: 'utf-8' codec can't encode character '\udcff' ... The UTF-8 mode, POSIX locale, Python 2 and the UNIX ``ls`` command work but display mojibake:: $ python3.7 -X utf8 ../ls.py xxx� $ LC_ALL=C /python3.6 ../ls.py xxx� $ python2 ../ls.py xxx� $ ls 'xxx'$'\377' List a directory into a text file --------------------------------- Similar to the previous example, except that the listing is written into a text file:: import os names = os.listdir(os.curdir) with open("/tmp/content.txt", "w") as fp: for name in names: fp.write("%s\n" % name) Result: ======================== ========== ========= Python Exception? Mojibake? ======================== ========== ========= Python 2 No **Yes** Python 3 **Yes** No Python 3.5, POSIX locale **Yes** No UTF-8 mode No **Yes** UTF-8 Strict mode **Yes** No ======================== ========== ========= Again, never throwing an exception requires that mojibake can be produced, while preventing mojibake means that the script can fail on decoding or encoding a filename depending on the locale or the filename. Typical error:: $ LC_ALL=C python3 test.py Traceback (most recent call last): File "test.py", line 5, in <module> fp.write("%s\n" % name) UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-1: ordinal not in range(128) Compared with native system tools:: $ ls > /tmp/content.txt $ cat /tmp/content.txt xxx� Display Unicode characters into stdout -------------------------------------- Very basic example used to illustrate a common issue, display the euro sign (U+20AC: €):: print("euro: \u20ac") Result: ======================== ========== ========= Python Exception? Mojibake? ======================== ========== ========= Python 2 **Yes** No Python 3 **Yes** No Python 3.5, POSIX locale **Yes** No UTF-8 mode No **Yes** UTF-8 Strict mode No **Yes** ======================== ========== ========= The UTF-8 and UTF-8 Strict modes will always encode the euro sign as UTF-8. If the terminal uses a different encoding, we get mojibake. For example, using ``iconv`` to emulate a GB-18030 terminal inside a UTF-8 one:: $ python3 -c 'print("euro: \u20ac")' | iconv -f gb18030 -t utf8 euro: 鈧iconv: illegal input sequence at position 8 The misencoding also corrupts the trailing newline such that the output stream isn't actually a valid GB-18030 sequence, hence the error message after the euro symbol is misinterpreted as a hanzi character. Replace a word in a text ------------------------ The following script replaces the word "apple" with "orange". It reads input from stdin and writes the output into stdout:: import sys text = sys.stdin.read() sys.stdout.write(text.replace("apple", "orange")) Result: ======================== ========== ========= Python Exception? Mojibake? ======================== ========== ========= Python 2 No **Yes** Python 3 **Yes** No Python 3.5, POSIX locale No **Yes** UTF-8 mode No **Yes** UTF-8 Strict mode **Yes** No ======================== ========== ========= This is a case where passing along the raw bytes (by way of the ``surrogateescape`` error handler) will bring Python 3's behaviour back into line with standard operating system tools like ``sed`` and ``awk``. Producer-consumer model using pipes ----------------------------------- Let's say that we have a "producer" program which writes data into its stdout and a "consumer" program which reads data from its stdin. On a shell, such programs are run with the command:: producer | consumer The question if these programs will work with any data and any locale. UNIX users don't expect Unicode errors, and so expect that such programs "just works", in the sense that Unicode errors may cause problems in the data stream, but won't cause the entire stream processing *itself* to abort. If the producer only produces ASCII output, no error should occur. Let's say that the producer writes at least one non-ASCII character (at least one byte in the range ``0x80..0xff``). To simplify the problem, let's say that the consumer has no output (doesn't write results into a file or stdout). A "Bytes producer" is an application which cannot fail with a Unicode error and produces bytes into stdout. Let's say that a "Bytes consumer" does not decode stdin but stores data as bytes: such consumer always work. Common UNIX command line tools like ``cat``, ``grep`` or ``sed`` are in this category. Many Python 2 applications are also in this category, as are applications that work with the lower level binary input and output stream in Python 3 rather than the default text mode streams. "Python producer" and "Python consumer" are producer and consumer implemented in Python using the default text mode input and output streams. Bytes producer, Bytes consumer ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ This won't through exceptions, but it is out of the scope of this PEP since it doesn't involve Python's default text mode input and output streams. Python producer, Bytes consumer ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Python producer:: print("euro: \u20ac") Result: ======================== ========== ========= Python Exception? Mojibake? ======================== ========== ========= Python 2 **Yes** No Python 3 **Yes** No Python 3.5, POSIX locale **Yes** No UTF-8 mode No **Yes** UTF-8 Strict mode No **Yes** ======================== ========== ========= The question here is not if the consumer is able to decode the input, but if Python is able to produce its output. So it's similar to the `Display Unicode characters into stdout`_ case. UTF-8 modes work with any locale since the consumer doesn't try to decode its stdin. Bytes producer, Python consumer ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Python consumer:: import sys text = sys.stdin.read() result = text.replace("apple", "orange") # ignore the result Result: ======================== ========== ========= Python Exception? Mojibake? ======================== ========== ========= Python 2 No **Yes** Python 3 **Yes** No Python 3.5, POSIX locale No **Yes** UTF-8 mode No **Yes** UTF-8 Strict mode **Yes** No ======================== ========== ========= Python 3 may throw an exception on decoding stdin depending on the input and the locale. Python producer, Python consumer ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Python producer:: print("euro: \u20ac") Python consumer:: import sys text = sys.stdin.read() result = text.replace("apple", "orange") # ignore the result Result, same Python version used for the producer and the consumer: ======================== ========== ========= Python Exception? Mojibake? ======================== ========== ========= Python 2 **Yes** No Python 3 **Yes** No Python 3.5, POSIX locale **Yes** No UTF-8 mode No No(!) UTF-8 Strict mode No No(!) ======================== ========== ========= This case combines a Python producer with a Python consumer, and the result is mainly the same as that for `Python producer, Bytes consumer`_, since the consumer can't read what the producer can't emit. However, the behaviour of the "UTF-8" and "UTF-8 Strict" modes in this configuration is notable: they don't produce an exception, *and* they shouldn't produce mojibake, as both the producer and the consumer are making *consistent* assumptions regarding the text encoding used on the pipe between them (i.e. UTF-8). Any mojibake generated would only be in the interfaces bween the consuming component and the outside world (e.g. the terminal, or when writing to a file). Backward Compatibility ====================== The main backward incompatible change is that the UTF-8 encoding is now used by default if the locale is POSIX. Since the UTF-8 encoding is used with the ``surrogateescape`` error handler, encoding errors should not occur and so the change should not break applications. The UTF-8 encoding is also quite restrictive regarding where it allows plain ASCII code points to appear in the byte stream, so even for ASCII-incompatible encodings, such byte values will often be escaped rather than being processed as ASCII characters. The more likely source of trouble comes from external libraries. Python can decode successfully data from UTF-8, but a library using the locale encoding can fail to encode the decoded text back to bytes. For example, GNU readline currently has problems on Android due to the mismatch between CPython's encoding assumptions there (always UTF-8) and GNU readline's encoding assumptions (which are based on the nominal locale). The PEP only changes the default behaviour if the locale is POSIX. For other locales, the *default* behaviour is unchanged. PEP 538 is a follow-up to this PEP that extends CPython's assumptions to other locale-aware components in the same process by explicitly coercing the POSIX locale to something more suitable for modern text processing. See that PEP for further details. Alternatives ============ Don't modify the encoding of the POSIX locale --------------------------------------------- A first version of the PEP did not change the encoding and error handler used of the POSIX locale. The problem is that adding the ``-X utf8`` command line option or setting the ``PYTHONUTF8`` environment variable is not possible in some cases, or at least not convenient. Moreover, many users simply expect that Python 3 behaves as Python 2: don't bother them with encodings and "just works" in all cases. These users don't worry about mojibake, or even expect mojibake because of complex documents using multiple incompatibles encodings. Always use UTF-8 ---------------- Python already always uses the UTF-8 encoding on Mac OS X, Android and Windows. Since UTF-8 became the de facto encoding, it makes sense to always use it on all platforms with any locale. The problem with this approach is that Python is also used extensively in desktop environments, and it is often a practical or even legal requirement to support locale encoding other than UTF-8 (for example, GB-18030 in China, and Shift-JIS or ISO-2022-JP in Japan) Force UTF-8 for the POSIX locale -------------------------------- An alternative to always using UTF-8 in any case is to only use UTF-8 when the ``LC_CTYPE`` locale is the POSIX locale. The `PEP 538`_ "Coercing the legacy C locale to C.UTF-8" of Nick Coghlan proposes to implement that using the ``C.UTF-8`` locale. Use the strict error handler for operating system data ------------------------------------------------------ Using the ``surrogateescape`` error handler for `operating system data`_ creates surprising surrogate characters. No Python codec (except of ``utf-7``) accept surrogates, and so encoding text coming from the operating system is likely to raise an error error. The problem is that the error comes late, very far from where the data was read. The ``strict`` error handler can be used instead to decode (``os.fsdecode()``) and encode (``os.fsencode()``) operating system data, to raise encoding errors as soon as possible. It helps to find bugs more quickly. The main drawback of this strategy is that it doesn't work in practice. Python 3 is designed on top on Unicode strings. Most functions expect Unicode and produce Unicode. Even if many operating system functions have two flavors, bytes and Unicode, the Unicode flavor is used in most cases. There are good reasons for that: Unicode is more convenient in Python 3 and using Unicode helps to support the full Unicode Character Set (UCS) on Windows (even if Python now uses UTF-8 since Python 3.6, see the `PEP 528`_ and the `PEP 529`_). For example, if ``os.fsdecode()`` uses ``utf8/strict``, ``os.listdir(str)`` fails to list filenames of a directory if a single filename is not decodable from UTF-8. As a consequence, ``shutil.rmtree(str)`` fails to remove a directory. Undecodable filenames, environment variables, etc. are simply too common to make this alternative viable. Links ===== PEPs: * `PEP 538 <https://www.python.org/dev/peps/pep-0538/>`_: "Coercing the legacy C locale to C.UTF-8" * `PEP 529 <https://www.python.org/dev/peps/pep-0529/>`_: "Change Windows filesystem encoding to UTF-8" * `PEP 528 <https://www.python.org/dev/peps/pep-0528/>`_: "Change Windows console encoding to UTF-8" * `PEP 383 <https://www.python.org/dev/peps/pep-0383/>`_: "Non-decodable Bytes in System Character Interfaces" Main Python issues: * `Issue #29240: Implementation of the PEP 540: Add a new UTF-8 mode <http://bugs.python.org/issue29240>`_ * `Issue #28180: sys.getfilesystemencoding() should default to utf-8 <http://bugs.python.org/issue28180>`_ * `Issue #19977: Use "surrogateescape" error handler for sys.stdin and sys.stdout on UNIX for the C locale <http://bugs.python.org/issue19977>`_ * `Issue #19847: Setting the default filesystem-encoding <http://bugs.python.org/issue19847>`_ * `Issue #8622: Add PYTHONFSENCODING environment variable <https://bugs.python.org/issue8622>`_: added but reverted because of many issues, read the `Inconsistencies if locale and filesystem encodings are different <https://mail.python.org/pipermail/python-dev/2010-October/104509.html>`_ thread on the python-dev mailing list Incomplete list of Python issues related to Unicode errors, especially with the POSIX locale: * 2016-12-22: `LANG=C python3 -c "import os; os.path.exists('\xff')" <http://bugs.python.org/issue29042#msg283821>`_ * 2014-07-20: `issue #22016: Add a new 'surrogatereplace' output only error handler <http://bugs.python.org/issue22016>`_ * 2014-04-27: `Issue #21368: Check for systemd locale on startup if current locale is set to POSIX <http://bugs.python.org/issue21368>`_ -- read manually /etc/locale.conf when the locale is POSIX * 2014-01-21: `Issue #20329: zipfile.extractall fails in Posix shell with utf-8 filename <http://bugs.python.org/issue20329>`_ * 2013-11-30: `Issue #19846: Python 3 raises Unicode errors with the C locale <http://bugs.python.org/issue19846>`_ * 2010-05-04: `Issue #8610: Python3/POSIX: errors if file system encoding is None <http://bugs.python.org/issue8610>`_ * 2013-08-12: `Issue #18713: Clearly document the use of PYTHONIOENCODING to set surrogateescape <http://bugs.python.org/issue18713>`_ * 2013-09-27: `Issue #19100: Use backslashreplace in pprint <http://bugs.python.org/issue19100>`_ * 2012-01-05: `Issue #13717: os.walk() + print fails with UnicodeEncodeError <http://bugs.python.org/issue13717>`_ * 2011-12-20: `Issue #13643: 'ascii' is a bad filesystem default encoding <http://bugs.python.org/issue13643>`_ * 2011-03-16: `issue #11574: TextIOWrapper should use UTF-8 by default for the POSIX locale <http://bugs.python.org/issue11574>`_, thread on python-dev: `Low-Level Encoding Behavior on Python 3 <https://mail.python.org/pipermail/python-dev/2011-March/109361.html>`_ * 2010-04-26: `Issue #8533: regrtest: use backslashreplace error handler for stdout <http://bugs.python.org/issue8533>`_, regrtest fails with Unicode encode error if the locale is POSIX Some issues are real bugs in applications which must explicitly set the encoding. Well, it just works in the common case (locale configured correctly), so what? The program "suddenly" fails when the POSIX locale is used (probably for bad reasons). Such bugs are not well understood by users. Example of such issues: * 2013-11-21: `pip: open() uses the locale encoding to parse Python script, instead of the encoding cookie <http://bugs.python.org/issue19685>`_ -- pip must use the encoding cookie to read a Python source code file * 2011-01-21: `IDLE 3.x can crash decoding recent file list <http://bugs.python.org/issue10974>`_ Prior Art ========= Perl has a ``-C`` command line option and a ``PERLUNICODE`` environment variable to force UTF-8: see `perlrun <http://perldoc.perl.org/perlrun.html>`_. It is possible to configure UTF-8 per standard stream, on input and output streams, etc. Post History ============ * 2017-04: `[Python-Dev] Proposed BDFL Delegate update for PEPs 538 & 540 (assuming UTF-8 for *nix system boundaries) <https://mail.python.org/pipermail/python-dev/2017-April/147795.html>`_ * 2017-01: `[Python-ideas] PEP 540: Add a new UTF-8 mode <https://mail.python.org/pipermail/python-ideas/2017-January/044089.html>`_ * 2017-01: `bpo-28180: Implementation of the PEP 538: coerce C locale to C.utf-8 (msg284764) <https://bugs.python.org/issue28180#msg284764>`_ * 2016-08-17: `bpo-27781: Change sys.getfilesystemencoding() on Windows to UTF-8 (msg272916) <https://bugs.python.org/issue27781#msg272916>`_ -- Victor proposed ``-X utf8`` for the :pep:`529` (Change Windows filesystem encoding to UTF-8) Copyright ========= This document has been placed in the public domain.

3 4

Re: [Python-Dev] PEPs: ``.. code:: python`` or ``::`` (syntax highlighting)
by Wes Turner Dec. 5, 2017

Dec. 5, 2017

Pending a transition of PEPs to ReadTheDocs (with HTTPS on a custom domain? and redirects?) (is there a gh issue for this task?), for the pythondotorg project is it as simple as `pip install pygments` and rebuilding each .rst with docutils with pygments installed? On Saturday, December 2, 2017, Mariatta Wijaya <mariatta.wijaya(a)gmail.com> wrote: > If we were to add Pygments support, it is to be done in pythondotorg > project. > > I recalled the decision was to get PEPs rendered using Sphinx and host it > at Read The Docs, so we don't have to worry about updating pythondotorg. > > Mariatta Wijaya > > >

3 3

PEP 557 Data Classes 5th posting
by Eric V. Smith Dec. 4, 2017

Dec. 4, 2017

I've pushed another version of PEP 557. The only difference is changing the default value of "order" to False instead of True. This matches regular classes: instances can be tested for equality, but are unordered. Discussion at https://github.com/ericvsmith/dataclasses/issues/104 It's already available at https://www.python.org/dev/peps/pep-0557/ I've updated the implementation on PyPI to reflect this change: https://pypi.python.org/pypi/dataclasses/0.3 Eric.

6 17

PEP 563: Postponed Evaluation of Annotations (Draft 3)
by Lukasz Langa Dec. 4, 2017

Dec. 4, 2017

Based on the feedback I gather in early November, I'm publishing the third draft for consideration on python-dev. I hope you like it! A nicely formatted rendering is available here: https://www.python.org/dev/peps/pep-0563/ The full list of changes between this version and the previous draft can be found here: https://github.com/ambv/static-annotations/compare/python-dev1...python-dev2 - Ł PEP: 563 Title: Postponed Evaluation of Annotations Version: $Revision$ Last-Modified: $Date$ Author: Łukasz Langa <lukasz(a)langa.pl> Discussions-To: Python-Dev <python-dev(a)python.org> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 8-Sep-2017 Python-Version: 3.7 Post-History: 1-Nov-2017, 21-Nov-2017 Resolution: Abstract ======== PEP 3107 introduced syntax for function annotations, but the semantics were deliberately left undefined. PEP 484 introduced a standard meaning to annotations: type hints. PEP 526 defined variable annotations, explicitly tying them with the type hinting use case. This PEP proposes changing function annotations and variable annotations so that they are no longer evaluated at function definition time. Instead, they are preserved in ``__annotations__`` in string form. This change is going to be introduced gradually, starting with a new ``__future__`` import in Python 3.7. Rationale and Goals =================== PEP 3107 added support for arbitrary annotations on parts of a function definition. Just like default values, annotations are evaluated at function definition time. This creates a number of issues for the type hinting use case: * forward references: when a type hint contains names that have not been defined yet, that definition needs to be expressed as a string literal; * type hints are executed at module import time, which is not computationally free. Postponing the evaluation of annotations solves both problems. Non-goals --------- Just like in PEP 484 and PEP 526, it should be emphasized that **Python will remain a dynamically typed language, and the authors have no desire to ever make type hints mandatory, even by convention.** This PEP is meant to solve the problem of forward references in type annotations. There are still cases outside of annotations where forward references will require usage of string literals. Those are listed in a later section of this document. Annotations without forced evaluation enable opportunities to improve the syntax of type hints. This idea will require its own separate PEP and is not discussed further in this document. Non-typing usage of annotations ------------------------------- While annotations are still available for arbitrary use besides type checking, it is worth mentioning that the design of this PEP, as well as its precursors (PEP 484 and PEP 526), is predominantly motivated by the type hinting use case. In Python 3.8 PEP 484 will graduate from provisional status. Other enhancements to the Python programming language like PEP 544, PEP 557, or PEP 560, are already being built on this basis as they depend on type annotations and the ``typing`` module as defined by PEP 484. In fact, the reason PEP 484 is staying provisional in Python 3.7 is to enable rapid evolution for another release cycle that some of the aforementioned enhancements require. With this in mind, uses for annotations incompatible with the aforementioned PEPs should be considered deprecated. Implementation ============== In Python 4.0, function and variable annotations will no longer be evaluated at definition time. Instead, a string form will be preserved in the respective ``__annotations__`` dictionary. Static type checkers will see no difference in behavior, whereas tools using annotations at runtime will have to perform postponed evaluation. The string form is obtained from the AST during the compilation step, which means that the string form might not preserve the exact formatting of the source. Note: if an annotation was a string literal already, it will still be wrapped in a string. Annotations need to be syntactically valid Python expressions, also when passed as literal strings (i.e. ``compile(literal, '', 'eval')``). Annotations can only use names present in the module scope as postponed evaluation using local names is not reliable (with the sole exception of class-level names resolved by ``typing.get_type_hints()``). Note that as per PEP 526, local variable annotations are not evaluated at all since they are not accessible outside of the function's closure. Enabling the future behavior in Python 3.7 ------------------------------------------ The functionality described above can be enabled starting from Python 3.7 using the following special import:: from __future__ import annotations A reference implementation of this functionality is available `on GitHub <https://github.com/ambv/cpython/tree/string_annotations>`_. Resolving Type Hints at Runtime =============================== To resolve an annotation at runtime from its string form to the result of the enclosed expression, user code needs to evaluate the string. For code that uses type hints, the ``typing.get_type_hints(obj, globalns=None, localns=None)`` function correctly evaluates expressions back from its string form. Note that all valid code currently using ``__annotations__`` should already be doing that since a type annotation can be expressed as a string literal. For code which uses annotations for other purposes, a regular ``eval(ann, globals, locals)`` call is enough to resolve the annotation. In both cases it's important to consider how globals and locals affect the postponed evaluation. An annotation is no longer evaluated at the time of definition and, more importantly, *in the same scope* where it was defined. Consequently, using local state in annotations is no longer possible in general. As for globals, the module where the annotation was defined is the correct context for postponed evaluation. The ``get_type_hints()`` function automatically resolves the correct value of ``globalns`` for functions and classes. It also automatically provides the correct ``localns`` for classes. When running ``eval()``, the value of globals can be gathered in the following way: * function objects hold a reference to their respective globals in an attribute called ``__globals__``; * classes hold the name of the module they were defined in, this can be used to retrieve the respective globals:: cls_globals = vars(sys.modules[SomeClass.__module__]) Note that this needs to be repeated for base classes to evaluate all ``__annotations__``. * modules should use their own ``__dict__``. The value of ``localns`` cannot be reliably retrieved for functions because in all likelihood the stack frame at the time of the call no longer exists. For classes, ``localns`` can be composed by chaining vars of the given class and its base classes (in the method resolution order). Since slots can only be filled after the class was defined, we don't need to consult them for this purpose. Runtime annotation resolution and class decorators -------------------------------------------------- Metaclasses and class decorators that need to resolve annotations for the current class will fail for annotations that use the name of the current class. Example:: def class_decorator(cls): annotations = get_type_hints(cls) # raises NameError on 'C' print(f'Annotations for {cls}: {annotations}') return cls @class_decorator class C: singleton: 'C' = None This was already true before this PEP. The class decorator acts on the class before it's assigned a name in the current definition scope. Runtime annotation resolution and ``TYPE_CHECKING`` --------------------------------------------------- Sometimes there's code that must be seen by a type checker but should not be executed. For such situations the ``typing`` module defines a constant, ``TYPE_CHECKING``, that is considered ``True`` during type checking but ``False`` at runtime. Example:: import typing if typing.TYPE_CHECKING: import expensive_mod def a_func(arg: expensive_mod.SomeClass) -> None: a_var: expensive_mod.SomeClass = arg ... This approach is also useful when handling import cycles. Trying to resolve annotations of ``a_func`` at runtime using ``typing.get_type_hints()`` will fail since the name ``expensive_mod`` is not defined (``TYPE_CHECKING`` variable being ``False`` at runtime). This was already true before this PEP. Backwards Compatibility ======================= This is a backwards incompatible change. Applications depending on arbitrary objects to be directly present in annotations will break if they are not using ``typing.get_type_hints()`` or ``eval()``. Annotations that depend on locals at the time of the function definition will not be resolvable later. Example:: def generate(): A = Optional[int] class C: field: A = 1 def method(self, arg: A) -> None: ... return C X = generate() Trying to resolve annotations of ``X`` later by using ``get_type_hints(X)`` will fail because ``A`` and its enclosing scope no longer exists. Python will make no attempt to disallow such annotations since they can often still be successfully statically analyzed, which is the predominant use case for annotations. Annotations using nested classes and their respective state are still valid. They can use local names or the fully qualified name. Example:: class C: field = 'c_field' def method(self) -> C.field: # this is OK ... def method(self) -> field: # this is OK ... def method(self) -> C.D: # this is OK ... def method(self) -> D: # this is OK ... class D: field2 = 'd_field' def method(self) -> C.D.field2: # this is OK ... def method(self) -> D.field2: # this is OK ... def method(self) -> field2: # this is OK ... def method(self) -> field: # this FAILS, class D doesn't ... # see C's attributes, This was # already true before this PEP. In the presence of an annotation that isn't a syntactically valid expression, SyntaxError is raised at compile time. However, since names aren't resolved at that time, no attempt is made to validate whether used names are correct or not. Deprecation policy ------------------ Starting with Python 3.7, a ``__future__`` import is required to use the described functionality. No warnings are raised. In Python 3.8 a ``PendingDeprecationWarning`` is raised by the compiler in the presence of type annotations in modules without the ``__future__`` import. Starting with Python 3.9 the warning becomes a ``DeprecationWarning``. In Python 4.0 this will become the default behavior. Use of annotations incompatible with this PEP is no longer supported. Forward References ================== Deliberately using a name before it was defined in the module is called a forward reference. For the purpose of this section, we'll call any name imported or defined within a ``if TYPE_CHECKING:`` block a forward reference, too. This PEP addresses the issue of forward references in *type annotations*. The use of string literals will no longer be required in this case. However, there are APIs in the ``typing`` module that use other syntactic constructs of the language, and those will still require working around forward references with string literals. The list includes: * type definitions:: T = TypeVar('T', bound='<type>') UserId = NewType('UserId', '<type>') Employee = NamedTuple('Employee', [('name', '<type>', ('id', '<type>')]) * aliases:: Alias = Optional['<type>'] AnotherAlias = Union['<type>', '<type>'] YetAnotherAlias = '<type>' * casting:: cast('<type>', value) * base classes:: class C(Tuple['<type>', '<type>']): ... Depending on the specific case, some of the cases listed above might be worked around by placing the usage in a ``if TYPE_CHECKING:`` block. This will not work for any code that needs to be available at runtime, notably for base classes and casting. For named tuples, using the new class definition syntax introduced in Python 3.6 solves the issue. In general, fixing the issue for *all* forward references requires changing how module instantiation is performed in Python, from the current single-pass top-down model. This would be a major change in the language and is out of scope for this PEP. Rejected Ideas ============== Keeping the ability to use function local state when defining annotations ------------------------------------------------------------------------- With postponed evaluation, this would require keeping a reference to the frame in which an annotation got created. This could be achieved for example by storing all annotations as lambdas instead of strings. This would be prohibitively expensive for highly annotated code as the frames would keep all their objects alive. That includes predominantly objects that won't ever be accessed again. To be able to address class-level scope, the lambda approach would require a new kind of cell in the interpreter. This would proliferate the number of types that can appear in ``__annotations__``, as well as wouldn't be as introspectable as strings. Note that in the case of nested classes, the functionality to get the effective "globals" and "locals" at definition time is provided by ``typing.get_type_hints()``. If a function generates a class or a function with annotations that have to use local variables, it can populate the given generated object's ``__annotations__`` dictionary directly, without relying on the compiler. Disallowing local state usage for classes, too ---------------------------------------------- This PEP originally proposed limiting names within annotations to only allow names from the model-level scope, including for classes. The author argued this makes name resolution unambiguous, including in cases of conflicts between local names and module-level names. This idea was ultimately rejected in case of classes. Instead, ``typing.get_type_hints()`` got modified to populate the local namespace correctly if class-level annotations are needed. The reasons for rejecting the idea were that it goes against the intuition of how scoping works in Python, and would break enough existing type annotations to make the transition cumbersome. Finally, local scope access is required for class decorators to be able to evaluate type annotations. This is because class decorators are applied before the class receives its name in the outer scope. Introducing a new dictionary for the string literal form instead ---------------------------------------------------------------- Yury Selivanov shared the following idea: 1. Add a new special attribute to functions: ``__annotations_text__``. 2. Make ``__annotations__`` a lazy dynamic mapping, evaluating expressions from the corresponding key in ``__annotations_text__`` just-in-time. This idea is supposed to solve the backwards compatibility issue, removing the need for a new ``__future__`` import. Sadly, this is not enough. Postponed evaluation changes which state the annotation has access to. While postponed evaluation fixes the forward reference problem, it also makes it impossible to access function-level locals anymore. This alone is a source of backwards incompatibility which justifies a deprecation period. A ``__future__`` import is an obvious and explicit indicator of opting in for the new functionality. It also makes it trivial for external tools to recognize the difference between a Python files using the old or the new approach. In the former case, that tool would recognize that local state access is allowed, whereas in the latter case it would recognize that forward references are allowed. Finally, just-in-time evaluation in ``__annotations__`` is an unnecessary step if ``get_type_hints()`` is used later. Dropping annotations with -O ---------------------------- There are two reasons this is not satisfying for the purpose of this PEP. First, this only addresses runtime cost, not forward references, those still cannot be safely used in source code. A library maintainer would never be able to use forward references since that would force the library users to use this new hypothetical -O switch. Second, this throws the baby out with the bath water. Now *no* runtime annotation use can be performed. PEP 557 is one example of a recent development where evaluating type annotations at runtime is useful. All that being said, a granular -O option to drop annotations is a possibility in the future, as it's conceptually compatible with existing -O behavior (dropping docstrings and assert statements). This PEP does not invalidate the idea. Pass string literals in annotations verbatim to ``__annotations__`` ------------------------------------------------------------------- This PEP originally suggested directly storing the contents of a string literal under its respective key in ``__annotations__``. This was meant to simplify support for runtime type checkers. Mark Shannon pointed out this idea was flawed since it wasn't handling situations where strings are only part of a type annotation. The inconsistency of it was always apparent but given that it doesn't fully prevent cases of double-wrapping strings anyway, it is not worth it. Make the name of the future import more verbose ----------------------------------------------- Instead of requiring the following import:: from __future__ import annotations the PEP could call the feature more explicitly, for example ``string_annotations``, ``stringify_annotations``, ``annotation_strings``, ``annotations_as_strings``, ``lazy_anotations``, ``static_annotations``, etc. The problem with those names is that they are very verbose. Each of them besides ``lazy_annotations`` would constitute the longest future feature name in Python. They are long to type and harder to remember than the single-word form. There is precedence of a future import name that sounds overly generic but in practice was obvious to users as to what it does:: from __future__ import division Prior discussion ================ In PEP 484 ---------- The forward reference problem was discussed when PEP 484 was originally drafted, leading to the following statement in the document: A compromise is possible where a ``__future__`` import could enable turning *all* annotations in a given module into string literals, as follows:: from __future__ import annotations class ImSet: def add(self, a: ImSet) -> List[ImSet]: ... assert ImSet.add.__annotations__ == { 'a': 'ImSet', 'return': 'List[ImSet]' } Such a ``__future__`` import statement may be proposed in a separate PEP. python/typing#400 ----------------- The problem was discussed at length on the typing module's GitHub project, under `Issue 400 <https://github.com/python/typing/issues/400>`_. The problem statement there includes critique of generic types requiring imports from ``typing``. This tends to be confusing to beginners: Why this:: from typing import List, Set def dir(o: object = ...) -> List[str]: ... def add_friends(friends: Set[Friend]) -> None: ... But not this:: def dir(o: object = ...) -> list[str]: ... def add_friends(friends: set[Friend]) -> None ... Why this:: up_to_ten = list(range(10)) friends = set() But not this:: from typing import List, Set up_to_ten = List[int](range(10)) friends = Set[Friend]() While typing usability is an interesting problem, it is out of scope of this PEP. Specifically, any extensions of the typing syntax standardized in PEP 484 will require their own respective PEPs and approval. Issue 400 ultimately suggests postponing evaluation of annotations and keeping them as strings in ``__annotations__``, just like this PEP specifies. This idea was received well. Ivan Levkivskyi supported using the ``__future__`` import and suggested unparsing the AST in ``compile.c``. Jukka Lehtosalo pointed out that there are some cases of forward references where types are used outside of annotations and postponed evaluation will not help those. For those cases using the string literal notation would still be required. Those cases are discussed briefly in the "Forward References" section of this PEP. The biggest controversy on the issue was Guido van Rossum's concern that untokenizing annotation expressions back to their string form has no precedent in the Python programming language and feels like a hacky workaround. He said: One thing that comes to mind is that it's a very random change to the language. It might be useful to have a more compact way to indicate deferred execution of expressions (using less syntax than ``lambda:``). But why would the use case of type annotations be so all-important to change the language to do it there first (rather than proposing a more general solution), given that there's already a solution for this particular use case that requires very minimal syntax? Eventually, Ethan Smith and schollii voiced that feedback gathered during PyCon US suggests that the state of forward references needs fixing. Guido van Rossum suggested coming back to the ``__future__`` idea, pointing out that to prevent abuse, it's important for the annotations to be kept both syntactically valid and evaluating correctly at runtime. First draft discussion on python-ideas -------------------------------------- Discussion happened largely in two threads, `the original announcement <https://mail.python.org/pipermail/python-ideas/2017-September/thread.html#4…>`_ and a follow-up called `PEP 563 and expensive backwards compatibility <https://mail.python.org/pipermail/python-ideas/2017-September/thread.html#4…>`_. The PEP received rather warm feedback (4 strongly in favor, 2 in favor with concerns, 2 against). The biggest voice of concern on the former thread being Steven D'Aprano's review stating that the problem definition of the PEP doesn't justify breaking backwards compatibility. In this response Steven seemed mostly concerned about Python no longer supporting evaluation of annotations that depended on local function/class state. A few people voiced concerns that there are libraries using annotations for non-typing purposes. However, none of the named libraries would be invalidated by this PEP. They do require adapting to the new requirement to call ``eval()`` on the annotation with the correct ``globals`` and ``locals`` set. This detail about ``globals`` and ``locals`` having to be correct was picked up by a number of commenters. Nick Coghlan benchmarked turning annotations into lambdas instead of strings, sadly this proved to be much slower at runtime than the current situation. The latter thread was started by Jim J. Jewett who stressed that the ability to properly evaluate annotations is an important requirement and backwards compatibility in that regard is valuable. After some discussion he admitted that side effects in annotations are a code smell and modal support to either perform or not perform evaluation is a messy solution. His biggest concern remained loss of functionality stemming from the evaluation restrictions on global and local scope. Nick Coghlan pointed out that some of those evaluation restrictions from the PEP could be lifted by a clever implementation of an evaluation helper, which could solve self-referencing classes even in the form of a class decorator. He suggested the PEP should provide this helper function in the standard library. Second draft discussion on python-dev ------------------------------------- Discussion happened mainly in the `announcement thread <https://mail.python.org/pipermail/python-dev/2017-November/150062.html>`_, followed by a brief discussion under `Mark Shannon's post <https://mail.python.org/pipermail/python-dev/2017-November/150637.html>`_. Steven D'Aprano was concerned whether it's acceptable for typos to be allowed in annotations after the change proposed by the PEP. Brett Cannon responded that type checkers and other static analyzers (like linters or programming text editors) will catch this type of error. Jukka Lehtosalo added that this situation is analogous to how names in function bodies are not resolved until the function is called. A major topic of discussion was Nick Coghlan's suggestion to store annotations in "thunk form", in other words as a specialized lambda which would be able to access class-level scope (and allow for scope customization at call time). He presented a possible design for it (`indirect attribute cells <https://mail.python.org/pipermail/python-dev/2017-November/150141.html>`_). This was later seen as equivalent to "special forms" in Lisp. Guido van Rossum expressed worry that this sort of feature cannot be safely implemented in twelve weeks (i.e. in time before the Python 3.7 beta freeze). After a while it became clear that the point of division between supporters of the string form vs. supporters of the thunk form is actually about whether annotations should be perceived as a general syntactic element vs. something tied to the type checking use case. Finally, Guido van Rossum declared he's rejecting the thunk idea based on the fact that it would require a new building block in the interpreter. This block would be exposed in annotations, multiplying possible types of values stored in ``__annotations__`` (arbitrary objects, strings, and now thunks). Moreover, thunks aren't as introspectable as strings. Most importantly, Guido van Rossum explicitly stated interest in gradually restricting the use of annotations to static typing (with an optional runtime component). Nick Coghlan got convinced to PEP 563, too, promptly beginning the mandatory bike shedding session on the name of the ``__future__`` import. Many debaters agreed that ``annotations`` seems like an overly broad name for the feature name. Guido van Rossum briefly decided to call it ``string_annotations`` but then changed his mind, arguing that ``division`` is a precedent of a broad name with a clear meaning. The final improvement to the PEP suggested in the discussion by Mark Shannon was the rejection of the temptation to pass string literals through to ``__annotations__`` verbatim. A side-thread of discussion started around the runtime penalty of static typing, with topic like the import time of the ``typing`` module (which is comparable to ``re`` without dependencies, and three times as heavy as ``re`` when counting dependencies). Acknowledgements ================ This document could not be completed without valuable input, encouragement and advice from Guido van Rossum, Jukka Lehtosalo, and Ivan Levkivskyi. Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

3 2

Accepting PEP 562 -- Module __getattr__ and __dir__
by Guido van Rossum Dec. 4, 2017

Dec. 4, 2017

Ivan, Congrats on your PEP. I believe the outstanding issues are now resolved and I am hereby accepting it. PS. Sorry, Larry, PEP 549 is rejected. But that happened a while ago. -- --Guido van Rossum (python.org/~guido)

4 4

3.7.0a3 still open
by Ned Deily Dec. 4, 2017

Dec. 4, 2017

Congratulations to the owners of the newly accepted PEPs. The code cutoff for 3.7.0 alpha 3 is scheduled for today (along with 3.6.3rc1). I know at least one of the PEPs has code ready to commit. I will hold off on tagging 3.7.0a3 for another 6 hours or so. If you feel your code is adequately reviewed and ready to go, go for it; likewise for normal bug fixes and doc changes. But keep in the mind that there is still one more alpha preview release coming prior to the beta 1 feature code freeze, so no need to panic. -- Ned Deily nad(a)python.org -- []

1 0

PEPs: ``.. code:: python`` or ``::`` (syntax highlighting)
by Wes Turner Dec. 3, 2017

Dec. 3, 2017

In ReStructuredText, this gets syntax highlighted because of the code directive [1][2][3]: .. code:: python import this def func(*args, **kwargs): pass This also gets syntax highlighted as python[3]: .. code:: python import this def func(*args, **kwargs): pass This does not:: import this def func(*args, **kwargs): pass Syntax highlighting in Docutils 0.9+ is powered by Pygments. If Pygments is not installed, or there is a syntax error, syntax highlighting is absent. GitHub does show Pygments syntax highlighting in .. code:: blocks for .rst and .restructuredtext documents [4] 1. Does the python.org PEP view support .. code:: blocks? [5] 2. Syntax highlighting is an advantage for writers, editors, and readers. 3. Should PEPs use .. code:: blocks to provide this advantage? [1] http://docutils.sourceforge.net/docs/ref/rst/directives.html#code [2] http://www.sphinx-doc.org/en/stable/markup/code.html [3] http://www.sphinx-doc.org/en/stable/config.html#confval-highlight_language [4] https://github.com/python/peps/blob/master/pep-0557.rst [5] https://www.python.org/dev/peps/pep-0557/ https://www.python.org/dev/peps/pep-0458/

4 3

Third and hopefully final post: PEP 557, Data Classes
by Eric V. Smith Dec. 3, 2017

Dec. 3, 2017

I've posted a new version of PEP 557, it should soon be available at https://www.python.org/dev/peps/pep-0557/. The only significant changes since the last version are: - changing the "compare" parameter to be "order", since that more accurately reflects what it does. - Having the combination of "eq=False" and "order=True" raise an exception instead of silently changing eq to True. There were no other issues raised with the previous version of the PEP. So with that, I think it's ready for a pronouncement. Eric.

10 21