Mailman 3 May 2019 - Python-Dev

PEP 594: update 1
by Christian Heimes 06 Jun '19

06 Jun '19

Hi, I have updated the PEP with feedback from discussions. The highlights are: * Deprecate parser module * Keep fileinput module * Elaborate why crypt and spwd are dangerous and bad * Improve sections for cgitb, colorsys, nntplib, and smtpd modules * The colorsys, crypt, imghdr, sndhdr, and spwd sections now list suitable substitutions. * Mention that socketserver is going to stay for http.server and xmlrpc.server * The future maintenance section now states that the deprecated modules may be adopted by Python community members. https://github.com/python/peps/compare/7799178a...2d536899?diff=unified#dif… I'll be traveling the next couple of days and will only have limited opportunities to respond on feedback. Christian ------------------------------------------------------------------- PEP: 594 Title: Removing dead batteries from the standard library Author: Christian Heimes <christian(a)python.org> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 20-May-2019 Post-History: 21-May-2019 Abstract ======== This PEP proposed a list of standard library modules to be removed from the standard library. The modules are mostly historic data formats and APIs that have been superseded a long time ago, e.g. Mac OS 9 and Commodore. Rationale ========= Back in the early days of Python, the interpreter came with a large set of useful modules. This was often refrained to as "batteries included" philosophy and was one of the corner stones to Python's success story. Users didn't have to figure out how to download and install separate packages in order to write a simple web server or parse email. Times have changed. The introduction of the cheese shop (PyPI), setuptools, and later pip, it became simple and straight forward to download and install packages. Nowadays Python has a rich and vibrant ecosystem of third party packages. It's pretty much standard to either install packages from PyPI or use one of the many Python or Linux distributions. On the other hand, Python's standard library is piling up cruft, unnecessary duplication of functionality, and dispensable features. This is undesirable for several reasons. * Any additional module increases the maintenance cost for the Python core development team. The team has limited resources, reduced maintenance cost frees development time for other improvements. * Modules in the standard library are generally favored and seen as the de-facto solution for a problem. A majority of users only pick 3rd party modules to replace a stdlib module, when they have a compelling reason, e.g. lxml instead of `xml`. The removal of an unmaintained stdlib module increases the chances of a community contributed module to become widely used. * A lean and mean standard library benefits platforms with limited resources like devices with just a few hundred kilobyte of storage (e.g. BBC Micro:bit). Python on mobile platforms like BeeWare or WebAssembly (e.g. pyodide) also benefit from reduced download size. The modules in the PEP have been selected for deprecation because their removal is either least controversial or most beneficial. For example least controversial are 30 years old multimedia formats like ``sunau`` audio format, which was used on SPARC and NeXT workstations in the late 1980ties. The ``crypt`` module has fundamental flaws that are better solved outside the standard library. This PEP also designates some modules as not scheduled for removal. Some modules have been deprecated for several releases or seem unnecessary at first glance. However it is beneficial to keep the modules in the standard library, mostly for environments where installing a package from PyPI is not an option. This can be cooperate environments or class rooms where external code is not permitted without legal approval. * The usage of FTP is declining, but some files are still provided over the FTP protocol or hosters offer FTP to upload content. Therefore ``ftplib`` is going to stay. * The ``optparse`` and ``getopt`` module are widely used. They are mature modules with very low maintenance overhead. * According to David Beazley [5]_ the ``wave`` module is easy to teach to kids and can make crazy sounds. Making a computer generate crazy sounds is powerful and highly motivating exercise for a 9yo aspiring developer. It's a fun battery to keep. Deprecation schedule ==================== 3.8 --- This PEP targets Python 3.8. Version 3.8.0 final is scheduled to be released a few months before Python 2.7 will reach its end of lifetime. We expect that Python 3.8 will be targeted by users that migrate to Python 3 in 2019 and 2020. To reduce churn and to allow a smooth transition from Python 2, Python 3.8 will neither raise `DeprecationWarning` nor remove any modules that have been scheduled for removal. Instead deprecated modules will just be *documented* as deprecated. Optionally modules may emit a `PendingDeprecationWarning`. All deprecated modules will also undergo a feature freeze. No additional features should be added. Bug should still be fixed. 3.9 --- Starting with Python 3.9, deprecated modules will start issuing `DeprecationWarning`. The `parser`_ module is removed and potentially replaced with a new module. All other deprecated modules are fully supported and will receive security updates until Python 3.9 reaches its end of lifetime. Python 3.9.0 will be released about 18 months after 3.8.0 (April 2021?) and most likely be supported for 5 years after the release. The estimated EOL of Python 3.9 is in 2026. 3.10 ---- In 3.10 all deprecated modules will be removed from the CPython repository together with tests, documentation, and autoconf rules. PEP acceptance process ====================== 3.8.0b1 is scheduled to be release shortly after the PEP is officially submitted. Since it's improbable that the PEP will pass all stages of the PEP process in time, I propose a two step acceptance process that is analogous Python's two release deprecation process. The first *provisionally accepted* phase targets Python 3.8.0b1. In the first phase no code is changes or removed. Modules are only documented as deprecated. The only exception is the `parser`_ module. It has been documented as deprecated since Python 2.5 and is scheduled for removal for 3.9 to make place for a more advanced parser. The final decision, which modules will be removed and how the removed code is preserved, can be delayed for another year. Deprecated modules ================== The modules are grouped as data encoding, multimedia, network, OS interface, and misc modules. The majority of modules are for old data formats or old APIs. Some others are rarely useful and have better replacements on PyPI, e.g. Pillow for image processing or NumPy-based projects to deal with audio processing. .. csv-table:: Table 1: Proposed modules deprecations :header: "Module", "Deprecated in", "To be removed", "Replacement" :widths: 1, 1, 1, 2 aifc,3.8,3.10,\- asynchat,3.8,3.10,asyncio asyncore,3.8,3.10,asyncio audioop,3.8,3.10,\- binhex,3.8,3.10,\- cgi,3.8,3.10,\- cgitb,3.8,3.10,\- chunk,3.8,3.10,\- colorsys,3.8,3.10,"colormath, colour, colorspacious, Pillow" crypt,3.8,3.10,"bcrypt, argon2cffi, hashlib, passlib" fileinput,\-,**keep**,argparse formatter,3.4,3.10,\- fpectl,**3.7**,**3.7**,\- getopt,**3.2**,**keep**,"argparse, optparse" imghdr,3.8,3.10,"filetype, puremagic, python-magic" imp,**3.4**,3.10,importlib lib2to3,\-,**keep**, macpath,**3.7**,**3.8**,\- msilib,3.8,3.10,\- nntplib,3.8,3.10,\- nis,3.8,3.10,\- optparse,\-,**keep**,argparse ossaudiodev,3.8,3.10,\- parser,**2.5**,**3.9**,"ast, lib2to3.pgen2" pipes,3.8,3.10,subprocess smtpd,"**3.4.7**, **3.5.4**",3.10,aiosmtpd sndhdr,3.8,3.10,"filetype, puremagic, python-magic" spwd,3.8,3.10,"python-pam, simplepam" sunau,3.8,3.10,\- uu,3.8,3.10,\- wave,\-,**keep**, xdrlib,3.8,3.10,\- Data encoding modules --------------------- binhex ~~~~~~ The `binhex <https://docs.python.org/3/library/binhex.html>`_ module encodes and decodes Apple Macintosh binhex4 data. It was originally developed for TSR-80. In the 1980s and early 1990s it was used on classic Mac OS 9 to encode binary email attachments. Module type pure Python Deprecated in 3.8 To be removed in 3.10 Substitute **none** uu ~~ The `uu <https://docs.python.org/3/library/uu.html>`_ module provides uuencode format, an old binary encoding format for email from 1980. The uu format has been replaced by MIME. The uu codec is provided by the binascii module. Module type pure Python Deprecated in 3.8 To be removed in 3.10 Substitute **none** xdrlib ~~~~~~ The `xdrlib <https://docs.python.org/3/library/xdrlib.html>`_ module supports the Sun External Data Representation Standard. XDR is an old binary serialization format from 1987. These days it's rarely used outside specialized domains like NFS. Module type pure Python Deprecated in 3.8 To be removed in 3.10 Substitute **none** Multimedia modules ------------------ aifc ~~~~ The `aifc <https://docs.python.org/3/library/aifc.html>`_ module provides support for reading and writing AIFF and AIFF-C files. The Audio Interchange File Format is an old audio format from 1988 based on Amiga IFF. It was most commonly used on the Apple Macintosh. These days only few specialized application use AIFF. Module type pure Python (depends on `audioop`_ C extension) Deprecated in 3.8 To be removed in 3.10 Substitute **none** audioop ~~~~~~~ The `audioop <https://docs.python.org/3/library/audioop.html>`_ module contains helper functions to manipulate raw audio data and adaptive differential pulse-code modulated audio data. The module is implemented in C without any additional dependencies. The `aifc`_, `sunau`_, and `wave`_ module depend on `audioop`_ for some operations. The byteswap operation in the `wave`_ module can be substituted with little work. Module type C extension Deprecated in 3.8 To be removed in 3.10 Substitute **none** colorsys ~~~~~~~~ The `colorsys <https://docs.python.org/3/library/colorsys.html>`_ module defines color conversion functions between RGB, YIQ, HSL, and HSV coordinate systems. The PyPI packages *colormath*, *colour*, and *colorspacious* provide more and advanced features. The Pillow library is better suited to transform images between color systems. Module type pure Python Deprecated in 3.8 To be removed in 3.10 Substitute `colormath <https://pypi.org/project/colormath/>`_, `colour <https://pypi.org/project/colour/>`_ `colorspacious <https://pypi.org/project/colorspacious/>`_, `Pillow <https://pypi.org/project/Pillow/>`_ chunk ~~~~~ The `chunk <https://docs.python.org/3/library/chunk.html>`_ module provides support for reading and writing Electronic Arts' Interchange File Format. IFF is an old audio file format originally introduced for Commodore and Amiga. The format is no longer relevant. Module type pure Python Deprecated in 3.8 To be removed in 3.10 Substitute **none** imghdr ~~~~~~ The `imghdr <https://docs.python.org/3/library/imghdr.html>`_ module is a simple tool to guess the image file format from the first 32 bytes of a file or buffer. It supports only a limited amount of formats and neither returns resolution nor color depth. Module type pure Python Deprecated in 3.8 To be removed in 3.10 Substitute `puremagic <https://pypi.org/project/puremagic/>`_, `filetype <https://pypi.org/project/filetype/>`_, `python-magic <https://pypi.org/project/python-magic/>`_ ossaudiodev ~~~~~~~~~~~ The `ossaudiodev <https://docs.python.org/3/library/ossaudiodev.html>`_ module provides support for Open Sound System, an interface to sound playback and capture devices. OSS was initially free software, but later support for newer sound devices and improvements were proprietary. Linux community abandoned OSS in favor of ALSA [1]_. Some operation systems like OpenBSD and NetBSD provide an incomplete [2]_ emulation of OSS. Module type C extension Deprecated in 3.8 To be removed in 3.10 Substitute **none** sndhdr ~~~~~~ The `sndhdr <https://docs.python.org/3/library/sndhdr.html>`_ module is similar to the `imghdr`_ module but for audio formats. It guesses file format, channels, frame rate, and sample widths from the first 512 bytes of a file or buffer. The module only supports AU, AIFF, HCOM, VOC, WAV, and other ancient formats. Module type pure Python (depends on `audioop`_ C extension for some operations) Deprecated in 3.8 To be removed in 3.10 Substitute `puremagic <https://pypi.org/project/puremagic/>`_, `filetype <https://pypi.org/project/filetype/>`_, `python-magic <https://pypi.org/project/python-magic/>`_ sunau ~~~~~ The `sunau <https://docs.python.org/3/library/sunhdr.html>`_ module provides support for Sun AU sound format. It's yet another old, obsolete file format. Module type pure Python (depends on `audioop`_ C extension for some operations) Deprecated in 3.8 To be removed in 3.10 Substitute **none** Networking modules ------------------ asynchat ~~~~~~~~ The `asynchat <https://docs.python.org/3/library/asynchat.html>`_ module is build on top of `asyncore`_ and has been deprecated since Python 3.6. Module type pure Python Deprecated in 3.6 Removed in 3.10 Substitute asyncio asyncore ~~~~~~~~ The `asyncore <https://docs.python.org/3/library/asyncore.html>`_ module was the first module for asynchronous socket service clients and servers. It has been replaced by asyncio and is deprecated since Python 3.6. The ``asyncore`` module is also used in stdlib tests. The tests for ``ftplib``, ``logging``, ``smptd``, ``smtplib``, and ``ssl`` are partly based on ``asyncore``. These tests must be updated to use asyncio or threading. Module type pure Python Deprecated in 3.6 Removed in 3.10 Substitute asyncio cgi ~~~ The `cgi <https://docs.python.org/3/library/cgi.html>`_ module is a support module for Common Gateway Interface (CGI) scripts. CGI is deemed as inefficient because every incoming request is handled in a new process. PEP 206 considers the module as *designed poorly and are now near-impossible to fix*. Several people proposed to either keep the cgi module for features like `cgi.parse_qs()` or move `cgi.escape()` to a different module. The functions `cgi.parse_qs` and `cgi.parse_qsl` have been deprecated for a while and are actually aliases for `urllib.parse.parse_qs` and `urllib.parse.parse_qsl`. The function `cgi.quote` has been deprecated in favor of `html.quote` with secure default values. Module type pure Python Deprecated in 3.8 To be removed in 3.10 Substitute **none** cgitb ~~~~~ The `cgitb <https://docs.python.org/3/library/cgitb.html>`_ module is a helper for the cgi module for configurable tracebacks. The ``cgitb`` module is not used by any major Python web framework (Django, Pyramid, Plone, Flask, CherryPy, or Bottle). Only Paste uses it in an optional debugging middleware. Module type pure Python Deprecated in 3.8 To be removed in 3.10 Substitute **none** smtpd ~~~~~ The `smtpd <https://docs.python.org/3/library/smtpd.html>`_ module provides a simple implementation of a SMTP mail server. The module documentation marks the module as deprecated and recommends ``aiosmtpd`` instead. The deprecation message was added in releases 3.4.7, 3.5.4, and 3.6.1. Module type pure Python Deprecated in **3.7** To be removed in 3.10 Substitute aiosmtpd nntplib ~~~~~~~ The `nntplib <https://docs.python.org/3/library/nntplib.html>`_ module implements the client side of the Network News Transfer Protocol (nntp). News groups used to be a dominant platform for online discussions. Over the last two decades, news has been slowly but steadily replaced with mailing lists and web-based discussion platforms. Twisted is also `planning <https://twistedmatrix.com/trac/ticket/9405>`_ to deprecate NNTP support and `pynnt <https://github.com/greenbender/pynntp>`_ hasn't seen any activity since 2014. This is a good indicator that the public interest in NNTP support is declining. The ``nntplib`` tests have been the cause of additional work in the recent past. Python only contains client side of NNTP. The tests connect to external news server. The servers are sometimes unavailble, too slow, or do not work correctly over IPv6. The situation causes flaky test runs on buildbots. Module type pure Python Deprecated in 3.8 To be removed in 3.10 Substitute **none** Operating system interface -------------------------- crypt ~~~~~ The `crypt <https://docs.python.org/3/library/crypt.html>`_ module implements password hashing based on ``crypt(3)`` function from ``libcrypt`` or ``libxcrypt`` on Unix-like platform. The algorithms are mostly old, of poor quality and insecure. Users are discouraged to use them. * The module is not available on Windows. Cross-platform application need an alternative implementation any way. * Only DES encryption is guarenteed to be available. DES has an extremely limited key space of 2**56. * MD5, salted SHA256, salted SHA512, and Blowfish are optional extension. SSHA256 and SSHA512 are glibc extensions. Blowfish (bcrypt) is the only algorithm that is still secure. However it's in glibc and therefore not commonly available on Linux. * Depending on the platform, the ``crypt`` module is not thread safe. Only implementations with ``crypt_r(3)`` are thread safe. * The module was never useful to interact with system user and password databases. On BSD, macOS, and Linux, all user authentication and password modification operations must go through PAM (pluggable authentication module), see `spwd`_ deprecation. Module type C extension + Python module Deprecated in 3.8 To be removed in 3.10 Substitute `bcrypt <https://pypi.org/project/bcrypt/>`_, `passlib <https://pypi.org/project/passlib/>`_, `argon2cffi <https://pypi.org/project/argon2-cffi/>`_, hashlib module (PBKDF2, scrypt) macpath ~~~~~~~ The `macpath <https://docs.python.org/3/library/macpath.html>`_ module provides Mac OS 9 implementation of os.path routines. Mac OS 9 is no longer supported Module type pure Python Deprecated in 3.7 Removed in 3.8 Substitute **none** nis ~~~ The `nis <https://docs.python.org/3/library/nis.html>`_ module provides NIS/YP support. Network Information Service / Yellow Pages is an old and deprecated directory service protocol developed by Sun Microsystems. It's designed successor NIS+ from 1992 never took off. For a long time, libc's Name Service Switch, LDAP, and Kerberos/GSSAPI are considered a more powerful and more secure replacement of NIS. Module type C extension Deprecated in 3.8 To be removed in 3.10 Substitute **none** spwd ~~~~ The `spwd <https://docs.python.org/3/library/spwd.html>`_ module provides direct access to Unix shadow password database using non-standard APIs. In general it's a bad idea to use the spwd. The spwd circumvents system security policies, it does not use the PAM stack, and is only compatible with local user accounts, because it ignores NSS. The use of the ``spwd`` module for access control must be consider a *security bug*, as it bypasses PAM's access control. Further more the ``spwd`` module uses the `shadow(3) <http://man7.org/linux/man-pages/man3/shadow.3.html>`_ APIs. Functions like ``getspnam(3)`` access the ``/etc/shadow`` file directly. This is dangerous and even forbidden for confined services on systems with a security engine like SELinux or AppArmor. Module type C extension Deprecated in 3.8 To be removed in 3.10 Substitute `python-pam <https://pypi.org/project/python-pam/>`_, `simpleplam <https://pypi.org/project/simplepam/>`_ Misc modules ------------ formatter ~~~~~~~~~ The `formatter <https://docs.python.org/3/library/formatter.html>`_ module is an old text formatting module which has been deprecated since Python 3.4. Module type pure Python Deprecated in 3.4 To be removed in 3.10 Substitute *n/a* imp ~~~ The `imp <https://docs.python.org/3/library/imp.html>`_ module is the predecessor of the `importlib <https://docs.python.org/3/library/importlib.html>`_ module. Most functions have been deprecated since Python 3.3 and the module since Python 3.4. Module type C extension Deprecated in 3.4 To be removed in 3.10 Substitute importlib msilib ~~~~~~ The `msilib <https://docs.python.org/3/library/msilib.html>`_ package is a Windows-only package. It supports the creation of Microsoft Installers (MSI). The package also exposes additional APIs to create cabinet files (CAB). The module is used to facilitate distutils to create MSI installers with ``bdist_msi`` command. In the past it was used to create CPython's official Windows installer, too. Microsoft is slowly moving away from MSI in favor of Windows 10 Apps (AppX) as new deployment model [3]_. Module type C extension + Python code Deprecated in 3.8 To be removed in 3.10 Substitute **none** parser ~~~~~~ The `parser <https://docs.python.org/3/library/parser.html>`_ module provides an interface to Python’s internal parser and byte-code compiler. The stdlib has superior ways to interact with the parse tree. From Python 2.5 onward, it's much more convenient to cut in at the Abstract Syntax Tree (AST) generation and compilation stage. The ``parser`` module causes additional work. It's C code that must be kept in sync with any change to Python's grammar and internal parser. Pablo wants to remove the parser module and promote lib2to3's pgen2 instead [6]_. Most importantly the presence of the ``parser`` module makes it harder to switch to something more powerful than a LL(1) parser [7]_. Since the ``parser`` module is documented as deprecated since Python 2.5 and a new parsing technology is planned for 3.9, the ``parser`` module is scheduled for removal in 3.9. Module type C extension Deprecated in 3.8, documented as deprecated since **2.5** To be removed in **3.9** Substitute ast, lib2to3.pgen2 pipes ~~~~~ The `pipes <https://docs.python.org/3/library/pipes.html>`_ module provides helpers to pipe the input of one command into the output of another command. The module is built on top of ``os.popen``. Users are encouraged to use the subprocess module instead. Module type pure Python Deprecated in 3.8 To be removed in 3.10 Substitute subprocess module Removed modules =============== fpectl ------ The `fpectl <https://docs.python.org/3.6/library/fpectl.html>`_ module was never built by default, its usage was discouraged and considered dangerous. It also required a configure flag that caused an ABI incompatibility. The module was removed in 3.7 by Nathaniel J. Smith in `bpo-29137 <https://bugs.python.org/issue29137>`_. Module type C extension + CAPI Deprecated in 3.7 Removed in 3.7 Substitute **none** Modules to keep =============== Some modules were originally proposed for deprecation. fileinput --------- The `fileinput <https://docs.python.org/3/library/fileinput.html>`_ module implements a helpers to iterate over a list of files from ``sys.argv``. The module predates the optparser and argparser module. The same functionality can be implemented with the argparser module. Several core developers expressed their interest to keep the module in the standard library, as it is handy for quick scripts. Module type pure Python lib2to3 ------- The `lib2to3 <https://docs.python.org/3/library/2to3.html>`_ package provides the ``2to3`` command to transpile Python 2 code to Python 3 code. The package is useful for other tasks besides porting code from Python 2 to 3. For example `black`_ uses it for code reformatting. Module type pure Python getopt ------ The `getopt <https://docs.python.org/3/library/getopt.html>`_ module mimics C's getopt() option parser. Although users are encouraged to use argparse instead, the getopt module is still widely used. The module is small, simple, and handy for C developers to write simple Python scripts. Module type pure Python Substitute argparse optparse -------- The `optparse <https://docs.python.org/3/library/optparse.html>`_ module is the predecessor of the argparse module. Although it has been deprecated for many years, it's still too widely used to remove it. Module type pure Python Deprecated in 3.2 Substitute argparse wave ---- The `wave <https://docs.python.org/3/library/wave.html>`_ module provides support for the WAV sound format. The module is not deprecated, because The WAV format is still relevant these days. The ``wave`` module is also used in education, e.g. to show kids how to make noise with a computer. The module uses one simple function from the `audioop`_ module to perform byte swapping between little and big endian formats. Before 24 bit WAV support was added, byte swap used to be implemented with the ``array`` module. To remove ``wave``'s dependency on the ``audioop``, the byte swap function could be either be moved to another module (e.g. ``operator``) or the ``array`` module could gain support for 24 bit (3 byte) arrays. Module type pure Python (depends on *byteswap* from `audioop`_ C extension) Deprecated in 3.8 To be removed in 3.10 Substitute *n/a* Future maintenance of removed modules ===================================== The main goal of the PEP is to reduce the burden and workload on the Python core developer team. Therefore removed modules will not be maintained by the core team as separate PyPI packages. However the removed code, tests and documentation may be moved into a new git repository, so community members have a place from which they can pick up and fork code. A first draft of a `legacylib <https://github.com/tiran/legacylib>`_ repository is available on my private Github account. The modules could be made available on PyPI. The Python core team will not publish or maintain the packages. It is my hope that members of the Python community will adopt, maintain, and perhaps improve the deprecated modules. It's my hope that some of the deprecated modules will be picked up and adopted by users that actually care about them. For example ``colorsys`` and ``imghdr`` are useful modules, but have limited feature set. A fork of ``imghdr`` can add new features and support for more image formats, without being constrained by Python's release cycle. Most of the modules are in pure Python and can be easily packaged. Some depend on a simple C module, e.g. `audioop`_ and `crypt`_. Since `audioop`_ does not depend on any external libraries, it can be shipped in as binary wheels with some effort. Other C modules can be replaced with ctypes or cffi. For example I created `legacycrypt <https://github.com/tiran/legacycrypt>`_ with ``_crypt`` extension reimplemented with a few lines of ctypes code. Discussions =========== * Elana Hashman and Nick Coghlan suggested to keep the *getopt* module. * Berker Peksag proposed to deprecate and removed *msilib*. * Brett Cannon recommended to delay active deprecation warnings and removal of modules like *imp* until Python 3.10. Version 3.8 will be released shortly before Python 2 reaches end of lifetime. A delay reduced churn for users that migrate from Python 2 to 3.8. * Brett also came up with the idea to keep lib2to3. The package is useful for other purposes, e.g. `black <https://pypi.org/project/black/>`_ uses it to reformat Python code. * At one point, distutils was mentioned in the same sentence as this PEP. To avoid lengthy discussion and delay of the PEP, I decided against dealing with distutils. Deprecation of the distutils package will be handled by another PEP. * Multiple people (Gregory P. Smith, David Beazley, Nick Coghlan, ...) convinced me to keep the `wave`_ module. [4]_ * Gregory P. Smith proposed to deprecate `nntplib`_. [4]_ * Andrew Svetlov mentioned the ``socketserver`` module is questionable. However it's used to implement ``http.server`` and ``xmlrpc.server``. The stdlib doesn't have a replacement for the servers, yet. Update history ============== Update 1 -------- * Deprecate `parser`_ module * Keep `fileinput`_ module * Elaborate why `crypt`_ and `spwd`_ are dangerous and bad * Improve sections for `cgitb`_, `colorsys`_, `nntplib`_, and `smtpd`_ modules * The `colorsys`_, `crypt`_, `imghdr`_, `sndhdr`_, and `spwd`_ sections now list suitable substitutions. * Mention that ``socketserver`` is going to stay for ``http.server`` and ``xmlrpc.server`` * The future maintenance section now states that the deprecated modules may be adopted by Python community members. References ========== .. [1] https://en.wikipedia.org/wiki/Open_Sound_System#Free,_proprietary,_free .. [2] https://man.openbsd.org/ossaudio .. [3] https://blogs.msmvps.com/installsite/blog/2015/05/03/the-future-of-windows-… .. [4] https://twitter.com/ChristianHeimes/status/1130257799475335169 .. [5] https://twitter.com/dabeaz/status/1130278844479545351 .. [6] https://mail.python.org/pipermail/python-dev/2019-May/157464.html .. [7] https://discuss.python.org/t/switch-pythons-parsing-tech-to-something-more-… Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

17 35

[PEP 558] thinking through locals() semantics
by Nathaniel Smith 02 Jun '19

02 Jun '19

First, I want to say: I'm very happy with PEP 558's changes to f_locals. It solves the weird threading bugs, and exposes the fundamental operations you need for debugging in a simple and clean way, while leaving a lot of implementation flexibility for future Python VMs. It's a huge improvement over what we had before. I'm not as sure about the locals() parts of the proposal. It might be fine, but there are some complex trade-offs here that I'm still trying to wrap my head around. The rest of this document is me thinking out loud to try to clarify these issues. ##### What are we trying to solve? There are two major questions, which are somewhat distinct: - What should the behavior of locals() be in CPython? - How much of that should be part of the language definition, vs CPython implementation details? The status quo is that for locals() inside function scope, the behavior is quite complex and subtle, and it's entirely implementation defined. In the current PEP draft, there are some small changes to the semantics, and also it promotes them becoming part of the official language semantics. I think the first question, about semantics, is the more important one. If we're promoting them to the language definition, the main effect is just to make it more important we get the semantics right. ##### What are the PEP's proposed semantics for locals()? They're kinda subtle. [Nick: please double-check this section, both for errors and because I think it includes some edge cases that the PEP currently doesn't mention.] For module/class scopes, locals() has always returned a mapping object which acts as a "source of truth" for the actual local environment – mutating the environment directly changes the mapping object, and vice-versa. That's not going to change. In function scopes, things are more complicated. The *local environment* is conceptually well-defined, and includes: - local variables (current source of truth: "fast locals" array) - closed-over variables (current source of truth: cell objects) - any arbitrary key/values written to frame.f_locals that don't correspond to local or closed-over variables, e.g. you can do frame.f_locals[object()] = 10, and then later read it out again. However, the mapping returned by locals() does not directly reflect this local environment. Instead, each function frame has a dict associated with it. locals() returns this dict. The dict always holds any non-local/non-closed-over variables, and also, in certain circumstances, we write a snapshot of local and closed-over variables back into the dict. Specifically, we write back: - Whenever locals() is called - Whenever exec() or eval() is called without passing an explicit locals argument - After every trace/profile event, if a Python-level tracing/profiling function is registered. (Note: in CPython, the use of Python-level tracing/profiling functions is extremely rare. It's more common in alternative implementations like PyPy. For example, the coverage package uses a C-level tracing function on CPython, which does not trigger locals updates, but on PyPy it uses a Python-level tracing function, which does trigger updates.) In addition, the PEP doesn't say, but I think that any writes to f_locals immediately update both the environment and the locals dict. These semantics have some surprising consequences. Most obviously, in function scope (unlike other scopes), mutating locals() does not affect the actual local environment: def f(): a = 1 locals()["a"] = 2 assert a == 1 The writeback rules can also produce surprising results: def f(): loc1 = locals() # Since it's a snapshot created at the time of the call # to locals(), it doesn't contain 'loc1': assert "loc1" not in loc1 loc2 = locals() # Now loc1 has changed: assert "loc1" in loc1 However, the results here are totally different if a Python-level tracing/profiling function is installed – in particular, the first assertion fails. The interaction between f_locals and and locals() is also subtle: def f(): a = 1 loc = locals() assert "loc" not in loc # Regular variable updates don't affect 'loc' a = 2 assert loc["a"] == 1 # But debugging updates do: sys._getframe().f_locals["a"] = 3 assert a == 3 assert loc["a"] == 3 # But it's not a full writeback assert "loc" not in loc # Mutating 'loc' doesn't affect f_locals: loc["a"] = 1 assert sys._getframe().f_locals["a"] == 1 # Except when it does: loc["b"] = 3 assert sys._getframe().f_locals["b"] == 3 Again, the results here are totally different if a Python-level tracing/profiling function is installed. And you can also hit these subtleties via 'exec' and 'eval': def f(): a = 1 loc = locals() assert "loc" not in loc # exec() triggers writeback, and then mutates the locals dict exec("a = 2; b = 3") # So now the current environment has been reflected into 'loc' assert "loc" in loc # Also loc["a"] has been changed to reflect the exec'ed assignments assert loc["a"] == 2 # But if we look at the actual environment, directly or via # f_locals, we can see that 'a' has not changed: assert a == 1 assert sys._getframe().f_locals["a"] == 1 # loc["b"] changed as well: assert loc["b"] == 3 # And this *does* show up in f_locals: assert sys._getframe().f_locals["b"] == 3 Of course, many of these edge cases are pretty obscure, so it's not clear how much they matter. But I think we can at least agree that this isn't the one obvious way to do it :-). ##### What's the landscape of possible semantics? I did some brainstorming, and came up with 4 sets of semantics that seem plausible enough to at least consider: - [PEP]: the semantics in the current PEP draft. - [PEP-minus-tracing]: same as [PEP], except dropping the writeback on Python-level trace/profile events. - [snapshot]: in function scope, each call to locals() returns a new, *static* snapshot of the local environment, removing all this writeback stuff. Something like: def locals(): frame = get_caller_frame() if is_function_scope(frame): # make a point-in-time copy of the "live" proxy object return dict(frame.f_locals) else: # in module/class scope, return the actual local environment return frame.f_locals - [proxy]: Simply return the .f_locals object, so in all contexts locals() returns a live mutable view of the actual environment: def locals(): return get_caller_frame().f_locals ##### How to evaluate our options? I can think of a lot of criteria that all-else-being-equal we would like Python to meet. (Of course, in practice they conflict.) Consistency across APIs: it's surprising if locals() and frame.f_locals do different things. This argues for [proxy]. Consistency across contexts: it's surprising if locals() has acts differently in module/class scope versus function scope. This argues for [proxy]. Consistent behavior when the environment shifts, or small maintenance changes are made: it's nice if code that works today keeps working tomorrow. On this criterion, I think [snapshot] > [proxy] > [PEP-minus-tracing] >>> [PEP]. [PEP] is particularly bad here because a very rare environmental change that almost no-one tests and mostly only happens when debugging (i.e., enabling tracing) causes a radical change in semantics. Simplicity of explaining to users: all else being equal, it's nice if our docs are short and clear and the language fits in your head. On this criterion, I think: [proxy] > [snapshot] > [PEP-minus-tracing] > [PEP]. As evidence that the current behavior is confusing, see: - Ned gets confused and writes a long blog post after he figures it out: https://nedbatchelder.com/blog/201211/tricky_locals.html - A linter that warns against mutating locals(): https://lgtm.com/rules/10030096/ Simplicity of implementation: "If the implementation is easy to explain, it may be a good idea." Since we need the proxy code anyway to implement f_locals, I think it's: [proxy] (free) > [snapshot] (one 'if' statement) > ([PEP] = [PEP-minus-tracing]). Impact on other interpreter implementations: all else being equal, we'd like to give new interpreters maximal freedom to do clever things. (And local variables are a place where language VMs often expend a lot of cleverness.) [proxy] and [snapshot] are both easily implemented in terms of f_locals, so they basically don't constrain alternative implementations at all. I'm not as sure about [PEP] and [PEP-minus-tracing]. I originally thought they must be horrible. On further thought, I'm not convinced they're *that* bad, since the need to support people doing silly stuff like frame.f_locals[object()] = 10 means that implementations will already need to sometimes attach something like a dict object to their function frames. But perhaps alternative implementations would like to disallow this, or are OK with making it really slow but care about locals() performance more. Anyway, it's definitely ([proxy] = [snapshot]) > ([PEP] = [PEP-minus-tracing]), but I'm not sure whether the '>' is large or small. Backwards compatibility: help(locals) says: NOTE: Whether or not updates to this dictionary will affect name lookups in the local scope and vice-versa is *implementation dependent* and not covered by any backwards compatibility guarantees. So that claims that there are ~no backwards compatibility issues here. I'm going to ignore that; no matter what the docs say, we still don't want to break everyone's code. And unfortunately, I can't think of any realistic way to do a gradual transition, with like deprecation warnings and all that (can anyone else?), so whatever we do will be a flag-day change. Of our four options, [PEP] is intuitively the closest to what CPython has traditionally done. But what exactly breaks under the different approaches? I'll split this off into its own section. ##### Backwards compatibility I'll split this into three parts: code that treats locals() as read-only, exec()/eval(), and code that mutates locals(). I believe (but haven't checked) that the majority of uses of locals() are in simple cases like: def f(): .... print("{a} {b}".format(**locals())) Luckily, this code remains totally fine under all four of our candidates. exec() and eval() are an interesting case. In Python 2, exec'ing some assignments actually *did* mutate the local environment, e.g. you could do: # Python 2 def f(): exec "a = 1" assert a == 1 In Python 3, this was changed, so now exec() inside a function cannot mutate the enclosing scope. We got some bug reports about this change, and there are a number of questions on stackoverflow about it, e.g.: - https://bugs.python.org/issue4831 - https://stackoverflow.com/questions/52217525/how-can-i-change-the-value-of-… - https://stackoverflow.com/questions/50995581/eval-exec-with-assigning-varia… In all released versions of Python, eval() was syntactically unable to rebind variables, so eval()'s interaction with the local environment was undefined. However, in 3.8, eval() *will* be able to rebind variables using the ':=' operator, so this interaction will become user-visible. Presumably we'll want eval() to match exec(). OK, with that background out of the way, let's look at our candidates. If we adopt [proxy], then that will mean exec()-inside-functions will go back to the Python 2 behavior, where executing assignments in the enclosing scope actually changes the enclosing scope. This will likely break some code out there that's relying on the Python 3 behavior, though I don't know how common that is. (I'm guessing not too common? Using the same variable inside and outside an 'exec' and trusting that they *won't* be the same seems like an unusual thing to do. But I don't know.) With [PEP] and [PEP-minus-tracing], exec() is totally unchanged. With [snapshot], there's technically a small difference: if you call locals() and then exec(), the exec() no longer triggers an implicit writeback to the dict that locals() returned. I think we can ignore this, and say that for all three of these, exec() is unlikely to produce backwards compatibility issues. OK, finally, let's talk about code that calls locals() and then mutates the return value. The main difference between our candidates is how they handle mutation, so this seems like the most important case to focus on. Conveniently, Mark Shannon and friends have statically analyzed a large corpus of Python code and made a list of cases where people do this: https://lgtm.com/rules/10030096/alerts/ Thanks! I haven't gone through the whole list, but I read through the first few in the hopes of getting a better sense of what kind of code does this in the real world and how it would be impacted by our different options. https://lgtm.com/projects/g/pydata/xarray/snapshot/a2ac6af744584c8afed3d56d… Current: raises if a Python-level trace/profile function is set [PEP]: raises if a Python-level trace/profile function is set [PEP-minus-tracing]: ok [snapshot]: ok [proxy]: always raises Comment: uses locals() to capture a bunch of passed in kwargs so it can pass them as **kwargs to another function, and treats them like a snapshot. The authors were clearly aware of the dangers, because this pattern appears multiple times in this file, and all the other places make an explicit copy of locals() before using it, but this place apparently got missed. Fix is trivial: just do that here too. https://github.com/swagger-api/swagger-codegen/blob/master/modules/swagger-… Current: ok [PEP]: ok [PEP-minus-tracing]: ok [snapshot]: ok [proxy]: ok Comment: this is inside a tool used to generate Python wrappers for REST APIs. The vast majority of entries in the lgtm database are from code generated by this tool. This was tricky to analyze, because it's complex templated code, and it does mutate locals(). But I'm pretty confident that the mutations end up just... not mattering, and all possible generated code works under all of our candidates. Which is lucky, because if we did have to fix this it wouldn't be trivial: fixing up the code generator itself wouldn't be too hard, but it'll take years for everyone to regenerate their old wrappers. https://lgtm.com/projects/g/saltstack/salt/snapshot/bb0950e5eafbb897c8e969e… Current: ok [PEP]: ok [PEP-minus-tracing]: ok [snapshot]: raises [proxy]: ok Comment: the use of locals() here is totally superfluous – it repeatedly reads and writes to locals()[mod], and in all cases this could be replaced by a simple variable, or any other dict. And it's careful not to assign to any name that matches an actual local variable, so it works fine with [proxy] too. But it does assume that multiple calls to locals() return the same dict, so [snapshot] breaks it. In this case the fix is trivial: just use a variable. https://lgtm.com/projects/g/materialsproject/pymatgen/snapshot/fd6900ed1040… Current: buggy if a trace/profile function is set [PEP]: buggy if a trace/profile function is set [PEP-minus-tracing]: ok [snapshot]: ok [proxy]: raises Comment: Another example of collecting kwargs into a dict. Actually this code is always buggy, because they seem to think that dict.pop("a", "b") removes both "a" and "b" from the dict... but it would raise an exception on [proxy], because they use one of those local variables after popping it from locals(). Fix is trivial: take an explicit snapshot of locals() before modifying it. ##### Conclusion so far [PEP-minus-tracing] seems to strictly dominate [PEP]. It's equal or better on all criteria, and actually *more* compatible with all the legacy code I looked at, even though it's technically less consistent with what CPython used to do. Unless someone points out some new argument, I think we can reject the writeback-when-tracing part of the PEP draft. Choosing between the remaining three is more of a judgement call. I'm leaning towards saying that on net, [snapshot] beats [PEP-minus-tracing]: it's dramatically simpler, and the backwards incompatibilities that we've found so far seem pretty minor, on par with what we do in every point release. (In fact, in 3/4 of the cases I looked at, [snapshot] is actually what users seemed to trying to use in the first place.) For [proxy] versus [snapshot], a lot depends on what we think of changing the semantics of exec(). [proxy] is definitely more consistent and elegant, and if we could go back in time I think it's what we'd have done from the start. Its compatibility is maybe a bit worse than [snapshot] on non-exec() cases, but this seems pretty minor overall (it often doesn't matter, and if it does just write dict(locals()) instead of locals(), like you would in non-function scope). But the change in exec() semantics is an actual language change, even though it may not affect much real code, so that's what stands out for me. I'd very much like to hear about any considerations I've missed, and any opinions on the "judgement call" part. -- Nathaniel J. Smith -- https://vorpus.org

10 26

PEP 595: Improving bugs.python.org
by Ezio Melotti 02 Jun '19

02 Jun '19

Hello, Berker and I have been working on a PEP that suggests we keep using and improving bugs.python.org and Roundup instead of switching to GitHub Issues as proposed by PEP 581. The PEP covers: * What are the advantages of Roundup over GitHub issues; * What features are missing in Roundup and how can we add them; * Issues with PEP 581; * Issues with the migration plan proposed by PEP 588; The rendered version of PEP 595 is available at https://www.python.org/dev/peps/pep-0595/ For reference, you can consult PEP 581 and 588 at https://www.python.org/dev/peps/pep-0581/ and https://www.python.org/dev/peps/pep-0588/ The full text of the PEP is include below. We are planning to update the PEP to include the feedback we receive and to update the status of features as we implement them (we also have a Google Summer of Code students working on it). Best Regards, Ezio Melotti ================ PEP: 595 Title: Improving bugs.python.org Author: Ezio Melotti <ezio.melotti(a)gmail.com>, Berker Peksag <berker.peksag(a)gmail.com> Status: Draft Type: Process Content-Type: text/x-rst Created: 12-May-2019 Abstract ======== This PEP proposes a list of improvements to make bugs.python.org more usable for contributors and core developers. This PEP also discusses why remaining on Roundup should be preferred over switching to GitHub Issues, as proposed by :pep:`581`. Motivation ========== On May 14th, 2019 :pep:`581` has been accepted [#]_ without much public discussion and without a clear consensus [#]_. The PEP contains factual errors and doesn't address some of the issues that the migration to GitHub Issues might present. Given the scope of the migration, the amount of work required, and how it will negatively affect the workflow during the transition phase, this decision should be re-evaluated. Roundup advantages over GitHub Issues ===================================== This section discusses reasons why Roundup should be preferred over GitHub Issues and Roundup features that are not available on GitHub Issues. * **Roundup is the status quo.** Roundup has been an integral part of the CPython workflow for years. It is a stable product that has been tested and customized to adapt to our needs as the workflow evolved. It is possible to gradually improve it and avoid the disruption that a switch to a different system would inevitabily bring to the workflow. * **Open-source and Python powered.** Roundup is an open-source project and is written in Python. By using it and supporting it, we also support the Python ecosystem. Several features developed for bpo have also been ported to upstream Roundup over the years. * **Fully customizable.** Roundup can be (and has been) fully customized to fit our needs. * **Finer-grained access control.** Roundup allows the creation of different roles with different permissions (e.g. create, view, edit, etc.) for each individual property, and users can have multiple roles. * **Flexible UI.** While Roundup UI might look dated, it is convenient and flexible. For example, on the issue page, each field (e.g. title, type, versions, status, linked files and PRs, etc.) have appropriate UI elements (input boxes, dropdowns, tables, etc.) that are easy to set and also provide a convenient way to get info about the issue at a glance. The number of fields, their values, and the UI element they use is also fully customizable. GitHub only provides labels. The issue list page presents the issues in a compact and easy to read table with separate columns for different fields. For comparison, Roundup lists 50 issues in a screen, whereas GitHub takes two screens to shows 25 issues. * **Advanced search.** Roundup provides an accurate way to search and filter by using any combination of issue fields. It is also possible to customize the number of results and the fields displayed in the table, and the sorting and grouping (up to two levels). bpo also provides predefined summaries (e.g. "Created by you", "Assigned to you", etc.) and allows the creation of custom search queries that can be conveniently accessed from the sidebar. * **Nosy list autocomplete.** The nosy list has an autocomplete feature that suggests maintainers and experts. The suggestions are automatically updated when the experts index [#]_ changes. * **Dependencies and Superseders.** Roundup allows to specify dependencies that must be addressed before the current issues can be closed and a superseder issue to easily mark duplicates [#]_. The list of dependencies can also be used to create meta-issues that references several other sub-issues [#]_. Improving Roundup ================= This section lists some of the issues mentioned by :pep:`581` and other desired features and discusses how they can be implemented by improving Roundup and/or our instance. * **REST API support.** A REST API will make integration with other services and the development of new tools and applications easiers. Upstream Roundup now supports a REST API. Updating the tracker will make the REST API available. * **GitHub login support.** This will allow users to login to bugs.python.org (bpo) without having to create a new account. It will also solve issues with confirmation emails being marked as spam, and provide two-factor authentication. A patch to add this functionality is already available and is being integrated at the time of writing [#]_. * **Markdown support and message preview and editing.** This feature will allow the use of Markdown in messages and the ability to preview the message before the submission and edit it afterward. This can be done, but it will take some work. Possible solutions have been proposed on the roundup-devel mailing list [#]_. * **"Remove me from nosy list" button.** Add a button on issue pages to remove self from the nosy list. This feature will be added during GSoC 2019. * **Mobile friendly theme.** Current theme of bugs.python.org looks dated and it doesn't work well with mobile browsers. A mobile-friendly theme that is more modern but still familiar will be added. * **Add PR link to BPO emails.** Currently bpo emails don't include links to the corresponding PRs. A patch [#]_ is available to change the content of the bpo emails from:: components: +Tkinter versions: +Python 3.4 pull_requests: +42 to:: components: +Tkinter versions: +Python 3.4 pull_request: https://github.com/python/cpython/pull/341 * **Python 3 support.** Using Python 3 will make maintenance easier. Upstream Roundup now supports Python 3. Updating the tracker will allow us to switch to Python 3. The instances will need to be updated as well. * **Use upstream Roundup.** We currently use a fork of Roundup with a few modifications, most notably the GitHub integration. If this is ported upstream, we can start using upstream Roundup without having to maintain our fork. PEP 581 issues ============== This section addresses some errors and inaccuracies found in :pep:`581`. The "Why GitHub?" section of PEP 581 lists features currently available on GitHub Issues but not on Roundup. Some of this features are currently supported: * "Ability to reply to issue and pull request conversations via email." * Being able to reply by email has been one of the core features of Roundup since the beginning. It is also possible to create new issues or close existing ones, set or modify fields, and add attachments. * "Email notifications containing metadata, integrated with Gmail, allowing systematic filtering of emails." * Emails sent by Roundup contains metadata that can be used for filtering. * "Additional privacy, such as offering the user a choice to hide an email address, while still allowing communication with the user through @-mentions." * Email addresses are hidden by default to users that are not registered. Registered users can see other users' addresses because we configured the tracker to show them. It can easily be changed if desired. Users can still be added to the nosy list by using their username even if their address is hidden. * "Ability to automatically close issues when a PR has been merged." * The GitHub integration of Roundup automatically closes issues when a commit that contains "fixes issue <id>" is merged. (Alternative spellings such as "closes" or "bug" are also supported.) See [#]_ for a recent example of this feature. * "Support for permalinks, allowing easy quoting and copying & pasting of source code." * Roundup has permalinks for issues, messages, attachments, etc. In addition, Roundup allows to easily rewrite broken URLs in messages (e.g. if the code hosting changes). * "Core developers, volunteers, and the PSF don't have to maintain the issue infrastructure/site, giving us more time and resources to focus on the development of Python." * While this is partially true, additional resources are required to write and maintain bots. In some cases, bots are required to workaround GitHub's lack of features rather than expanding. [#]_ was written specifically to workaround GitHub's email integration. Updating our bots to stay up-to-date with changes in the GitHub API has also maintenance cost. [#]_ took two days to be fixed. In addition, we will still need to maintain Roundup for bpo (even if it becomes read-only) and for the other trackers we currently host/maintain (Jython [#]_ and Roundup [#]_). The "Issues with Roundup / bpo" section of :pep:`581` lists some issues that have already been fixed: * "The upstream Roundup code is in Mercurial. Without any CI available, it puts heavy burden on the few existing maintainers in terms of reviewing, testing, and applying patches." * While Roundup uses Mercurial by default, there is a git clone available on GitHub [#]_. Roundup also has CI available [#]_ [#]_. * "There is no REST API available. There is an open issue in Roundup for adding REST API. Last activity was in 2016." * The REST API has been integrated and it's now available in Roundup. * "Users email addresses are exposed. There is no option to mask it." * Exposing addresses to registered and logged in users was a decision taken when our instance was set up. This has now been changed to make the email addresses hidden for regular users too (Developers and Coordinators can still see them). The "Email address"" column from the user listing page [#]_ has been removed too. * "It sends a number of unnecessary emails and notifications, and it is difficult, if not impossible, to configure." * This can be configured. * "Creating an account has been a hassle. There have been reports of people having trouble creating accounts or logging in." * The main issue is confirmation emails being marked as spam. Work has been done to resolve the issue. Migration considerations ======================== This section describes issues with the migrations that might not have been addressed by :pep:`581` and :pep:`588`. :pep:`588` suggests to add a button to migrate issues to GitHub only when someone wants to keep working on them. This approach has several issues: * bpo will need to be updated in order to add a button that, once pressed, creates a new issue on GitHub, copies over all the messages, attachments, and creates/adds label for the existing fields. Permissions will also need to be tweaked to make individual issues read-only once they are migrated, and to prevent users to create new accounts. * The issues will be split between two trackers; searching issues will take significant more effort. * The conversion from Roundup to GitHub is lossy, unless all the bpo fields are converted into labels or preserved somewhere else. * bpo converts a number of references into links, including issue, message, and PR IDs, changeset numbers, legacy SVN revision numbers, paths to files in the repo, files in tracebacks (detecting the correct branch), links to devguide pages and sections [#]_. This happens when messages are requested so it is possible to create the correct link (e.g. all the file links used to point to hg.python.org and now point to GitHub). If the links are hardcoded during the migration, it will be difficult (if not impossible) to change them later. If they aren't, they will either be lost, or a tool to generate the links and updating them will need to be written. * GitHub doesn't provide a way to set and preserve issue IDs if they are migrated automatically with the use of a button. (Some projects managed to preserve the IDs by contaacting the GitHub staff and migrating the issues *en masse*.) * On top of the work and changes required to migrate to GitHub issues, we will still need to keep running and maintaining Roundup, for both our instance (read-only) and for the Jython and Roundup trackers (read-write). In addition to the issues listed in the "Open issues" section of :pep:`588`, this issues will need to be addressed: * GitHub is properietary and there is risk of vendor lock-in. Their business model might change and they could shut down altogether. * Switching to GitHub Issues will likely increase the number of invalid reports and increase the triaging effort. This concern has been raised in the past in a Zulip topic [#]_. There have been already cases where people posted comments on PRs that required moderators to mark them as off-topic or disruptive, delete them altogether, and even lock the conversation [#]_. * Roundup sends weekly reports to python-dev with a summary that includes new issues, recent issues with no replies, recent issues waiting for review, most discussed issues, closed issues, and deltas for open/closed/total issue counts [#]_. The report provides an easy way to keep track of the tracker activity and to make sure that issues that require attention are noticed. The data collect by the weekly report is also use to generate statistics and graphs that can be used to gain new insights [#]_. * There are currently two mailing lists where Roundup posts new tracker issues and all messages respectively: new-bugs-announce [#]_ and python-bugs-list [#]_. A new system will need to be developed to preserve this functionality. These MLs offer additional ways to keep track of the tracker activity. References ========== .. [#] [Python-Dev] PEP 581 (Using GitHub issues for CPython) is accepted https://mail.python.org/pipermail/python-dev/2019-May/157399.html .. [#] [python-committers] [Python-Dev] PEP 581 (Using GitHub issues for CPython) is accepted https://mail.python.org/pipermail/python-committers/2019-May/006755.html .. [#] Experts Index -- Python Devguide https://devguide.python.org/experts/ .. [#] An example of superseded issues: "re.sub() replaces only several matches" https://bugs.python.org/issue12078 .. [#] An example of meta issue using dependencies to track sub-issues: "Meta-issue: support of the android platform"" https://bugs.python.org/issue26865 .. [#] Support logging in with GitHub https://github.com/python/bugs.python.org/issues/7 .. [#] Re: [Roundup-devel] PEP 581 and Google Summer of Code https://sourceforge.net/p/roundup/mailman/message/36667828/ .. [#] [Tracker-discuss] [issue624] bpo emails contain useless non-github pull_request number - users want a link to actual github PR https://mail.python.org/pipermail/tracker-discuss/2018-June/004547.html .. [#] The commit reported in msg342882 closes the issue (see the history below) https://bugs.python.org/issue36951#msg342882 .. [#] The cpython-emailer-webhook project https://github.com/berkerpeksag/cpython-emailer-webhook .. [#] A recent incident caused by GitHub https://github.com/python/bedevere/pull/163 .. [#] Jython issue tracker https://bugs.jython.org/ .. [#] Roundup issue tracker https://issues.roundup-tracker.org/ .. [#] GitHub clone of Roundup https://github.com/roundup-tracker/roundup .. [#] Travis-CI for Roundup https://travis-ci.org/roundup-tracker/roundup) and codecov .. [#] Codecov for Roundup https://codecov.io/gh/roundup-tracker/roundup/commits .. [#] User listing -- Python tracker https://bugs.python.org/user?@sort=username .. [#] Generating Special Links in a Comment -- Python Devguide https://devguide.python.org/triaging/#generating-special-links-in-a-comment .. [#] The New-bugs-announce mailing list https://mail.python.org/mailman/listinfo/new-bugs-announce .. [#] The Python-bugs-list mailing list https://mail.python.org/mailman/listinfo/python-bugs-list .. [#] An example of [Python-Dev] Summary of Python tracker Issues https://mail.python.org/pipermail/python-dev/2019-May/157483.html .. [#] Issues stats -- Python tracker https://bugs.python.org/issue?@template=stats .. [#] s/n ratio -- Python -- Zulip https://python.zulipchat.com/#narrow/stream/130206-pep581/topic/s.2Fn.20rat… .. [#] For example this and other related PRs: https://github.com/python/cpython/pull/9099 Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

7 12

RFC: PEP 587 "Python Initialization Configuration": 3rd version
by Victor Stinner 31 May '19

31 May '19

Hi, Thanks to the constructive discussions, I enhanced my PEP 587. I don't plan any further change, the PEP is now ready to review (and maybe even for pronouncement, hi Thomas! :-)). The Rationale now better explains all challenges and the complexity of the Python Initialization Configuration. The "Isolate Python" section is a short guide explaining how configure Python to embed it into an application. The "Path Configuration" section elaborates the most interesting part of the configuration: configure where Python looks for modules (sys.path). I added PyWideStringList_Insert() to allow to prepend a path in module_search_paths. The "Python Issues" section give a long list of issues solved directly or indirectly by this PEP. I'm open for bikeshedding on PyConfig fields names and added functions names ;-) I hesitate for "use_module_search_paths": maybe "module_search_paths_set" is a better name, as in "is module_search_paths set?". The purpose of this field is to allow to have an empty sys.path (ask PyConfig_Read() to not override it). IMHO an empty sys.path makes sense for some specific use cases, like executing Pyhon code without any external module. My PEP 587 proposes better names: Py_FrozenFlag becomes PyConfig.pathconfig_warnings and Py_DebugFlag becomes PyConfig.parser_debug. I also avoided double negation. For example, Py_DontWriteBytecodeFlag becomes write_bytecode. Changes between version 3 and version 2: * PyConfig: Add configure_c_stdio and parse_argv; rename _frozen to pathconfig_warnings. * Rename functions using bytes strings and wide strings. For example, Py_PreInitializeFromWideArgs() becomes Py_PreInitializeFromArgs(), and PyConfig_SetArgv() becomes PyConfig_SetBytesArgv(). * Add PyWideStringList_Insert() function. * New "Path configuration", "Isolate Python", "Python Issues" and "Version History" sections. * PyConfig_SetString() and PyConfig_SetBytesString() now requires the configuration as the first argument. * Rename Py_UnixMain() to Py_BytesMain() HTML version: https://www.python.org/dev/peps/pep-0587/ Full PEP text below. I know that the PEP is long, but well, it's a complex topic, and I chose to add many examples to make the API easier to understand. Victor --- PEP: 587 Title: Python Initialization Configuration Author: Victor Stinner <vstinner(a)redhat.com>, Nick Coghlan <ncoghlan(a)gmail.com> BDFL-Delegate: Thomas Wouters <thomas(a)python.org> Discussions-To: python-dev(a)python.org Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 27-Mar-2019 Python-Version: 3.8 Abstract ======== Add a new C API to configure the Python Initialization providing finer control on the whole configuration and better error reporting. It becomes possible to read the configuration and modify it before it is applied. It also becomes possible to completely override how Python computes the module search paths (``sys.path``). Building a customized Python which behaves as regular Python becomes easier using the new ``Py_RunMain()`` function. Moreover, command line arguments passed to ``PyConfig.argv`` are now parsed as the regular Python parses command line options, and ``PyConfig.xoptions`` are handled as ``-X opt`` command line options. This extracts a subset of the API design from the PEP 432 development and refactoring work that is now considered sufficiently stable to make public (allowing 3rd party embedding applications access to the same configuration APIs that the native CPython CLI is now using). Rationale ========= Python is highly configurable but its configuration evolved organically. The initialization configuration is scattered all around the code using different ways to set them: global configuration variables (ex: ``Py_IsolatedFlag``), environment variables (ex: ``PYTHONPATH``), command line arguments (ex: ``-b``), configuration files (ex: ``pyvenv.cfg``), function calls (ex: ``Py_SetProgramName()``). A straightforward and reliable way to configure Python is needed. Some configuration parameters are not accessible from the C API, or not easily. For example, there is no API to override the default values of ``sys.executable``. Some options like ``PYTHONPATH`` can only be set using an environment variable which has a side effect on Python child processes. Some options also depends on other options: see `Priority and Rules`_. Python 3.7 API does not provide a consistent view of the overall configuration. The C API of Python 3.7 Initialization takes ``wchar_t*`` strings as input whereas the Python filesystem encoding is set during the initialization which can lead to mojibake. Python 3.7 APIs like ``Py_Initialize()`` aborts the process on memory allocation failure which is not convenient when Python is embedded. Moreover, ``Py_Main()`` could exit directly the process rather than returning an exit code. Proposed new API reports the error or exit code to the caller which can decide how to handle it. Implementing the PEP 540 (UTF-8 Mode) and the new ``-X dev`` correctly was almost impossible in Python 3.6. The code base has been deeply reworked in Python 3.7 and then in Python 3.8 to read the configuration into a structure with no side effect. It becomes possible to clear the configuration (release memory) and read again the configuration if the encoding changed . It is required to implement properly the UTF-8 which changes the encoding using ``-X utf8`` command line option. Internally, bytes ``argv`` strings are decoded from the filesystem encoding. The ``-X dev`` changes the memory allocator (behaves as ``PYTHONMALLOC=debug``), whereas it was not possible to change the memory allocation *while* parsing the command line arguments. The new design of the internal implementation not only allowed to implement properly ``-X utf8`` and ``-X dev``, it also allows to change the Python behavior way more easily, especially for corner cases like that, and ensure that the configuration remains consistent: see `Priority and Rules`_. This PEP is a partial implementation of PEP 432 which is the overall design. New fields can be added later to ``PyConfig`` structure to finish the implementation of the PEP 432 (e.g. by adding a new partial initialization API which allows to configure Python using Python objects to finish the full initialization). However, those features are omitted from this PEP as even the native CPython CLI doesn't work that way - the public API proposal in this PEP is limited to features which have already been implemented and adopted as private APIs for us in the native CPython CLI. Python Initialization C API =========================== This PEP proposes to add the following new structures, functions and macros. New structures (4): * ``PyConfig`` * ``PyInitError`` * ``PyPreConfig`` * ``PyWideStringList`` New functions (17): * ``Py_PreInitialize(config)`` * ``Py_PreInitializeFromBytesArgs(config, argc, argv)`` * ``Py_PreInitializeFromArgs(config, argc, argv)`` * ``PyWideStringList_Append(list, item)`` * ``PyWideStringList_Insert(list, index, item)`` * ``PyConfig_SetString(config,config_str, str)`` * ``PyConfig_SetBytesString(config, config_str, str)`` * ``PyConfig_SetBytesArgv(config, argc, argv)`` * ``PyConfig_SetArgv(config, argc, argv)`` * ``PyConfig_Read(config)`` * ``PyConfig_Clear(config)`` * ``Py_InitializeFromConfig(config)`` * ``Py_InitializeFromBytesArgs(config, argc, argv)`` * ``Py_InitializeFromArgs(config, argc, argv)`` * ``Py_BytesMain(argc, argv)`` * ``Py_RunMain()`` * ``Py_ExitInitError(err)`` New macros (9): * ``PyPreConfig_INIT`` * ``PyConfig_INIT`` * ``Py_INIT_OK()`` * ``Py_INIT_ERR(MSG)`` * ``Py_INIT_NO_MEMORY()`` * ``Py_INIT_EXIT(EXITCODE)`` * ``Py_INIT_IS_ERROR(err)`` * ``Py_INIT_IS_EXIT(err)`` * ``Py_INIT_FAILED(err)`` This PEP also adds ``_PyRuntimeState.preconfig`` (``PyPreConfig`` type) and ``PyInterpreterState.config`` (``PyConfig`` type) fields to these internal structures. ``PyInterpreterState.config`` becomes the new reference configuration, replacing global configuration variables and other private variables. PyWideStringList ---------------- ``PyWideStringList`` is a list of ``wchar_t*`` strings. Example to initialize a string from C static array:: static wchar_t* argv[2] = { L"-c", L"pass", }; PyWideStringList config_argv = PyWideStringList_INIT; config_argv.length = Py_ARRAY_LENGTH(argv); config_argv.items = argv; ``PyWideStringList`` structure fields: * ``length`` (``Py_ssize_t``) * ``items`` (``wchar_t**``) Methods: * ``PyInitError PyWideStringList_Append(PyWideStringList *list, const wchar_t *item)``: Append *item* to *list*. * ``PyInitError PyWideStringList_Insert(PyWideStringList *list, Py_ssize_t index, const wchar_t *item)``: Insert *item* into *list* at *index*. If *index* is greater than *list* length, just append *item* to *list*. If *length* is non-zero, *items* must be non-NULL and all strings must be non-NULL. PyInitError ----------- ``PyInitError`` is a structure to store an error message or an exit code for the Python Initialization. For an error, it stores the C function name which created the error. Example:: PyInitError alloc(void **ptr, size_t size) { *ptr = PyMem_RawMalloc(size); if (*ptr == NULL) { return Py_INIT_NO_MEMORY(); } return Py_INIT_OK(); } int main(int argc, char **argv) { void *ptr; PyInitError err = alloc(&ptr, 16); if (Py_INIT_FAILED(err)) { Py_ExitInitError(err); } PyMem_Free(ptr); return 0; } ``PyInitError`` fields: * ``exitcode`` (``int``): argument passed to ``exit()``, only set by ``Py_INIT_EXIT()``. * ``err_msg`` (``const char*``): error message * private ``_func`` field: used by ``Py_INIT_ERR()`` to store the C function name which created the error. * private ``_type`` field: for internal usage only. Macro to create an error: * ``Py_INIT_OK()``: Success. * ``Py_INIT_ERR(err_msg)``: Initialization error with a message. * ``Py_INIT_NO_MEMORY()``: Memory allocation failure (out of memory). * ``Py_INIT_EXIT(exitcode)``: Exit Python with the specified exit code. Other macros and functions: * ``Py_INIT_IS_ERROR(err)``: Is the result an error? * ``Py_INIT_IS_EXIT(err)``: Is the result an exit? * ``Py_INIT_FAILED(err)``: Is the result an error or an exit? Similar to ``Py_INIT_IS_ERROR(err) || Py_INIT_IS_EXIT(err)``. * ``Py_ExitInitError(err)``: Call ``exit(exitcode)`` on Unix or ``ExitProcess(exitcode)`` if the result is an exit, call ``Py_FatalError(err_msg)`` if the result is an error. Must not be called if the result is a success. Pre-Initialization with PyPreConfig ----------------------------------- ``PyPreConfig`` structure is used to pre-initialize Python: * Set the memory allocator * Configure the LC_CTYPE locale * Set the UTF-8 mode Example using the pre-initialization to enable the UTF-8 Mode:: PyPreConfig preconfig = PyPreConfig_INIT; preconfig.utf8_mode = 1; PyInitError err = Py_PreInitialize(&preconfig); if (Py_INIT_FAILED(err)) { Py_ExitInitError(err); } /* at this point, Python will speak UTF-8 */ Py_Initialize(); /* ... use Python API here ... */ Py_Finalize(); Functions to pre-initialize Python: * ``PyInitError Py_PreInitialize(const PyPreConfig *config)`` * ``PyInitError Py_PreInitializeFromBytesArgs(const PyPreConfig *config, int argc, char **argv)`` * ``PyInitError Py_PreInitializeFromArgs(const PyPreConfig *config, int argc, wchar_t **argv)`` These functions can be called with *config* set to ``NULL``. If Python is initialized with command line arguments, the command line arguments must also be passed to pre-initialize Python, since they have an effect on the pre-configuration like encodings. For example, the ``-X utf8`` command line option enables the UTF-8 Mode. These functions can be called with *config* set to ``NULL``. The caller is responsible to handle error or exit using ``Py_INIT_FAILED()`` and ``Py_ExitInitError()``. ``PyPreConfig`` fields: * ``allocator`` (``char*``, default: ``NULL``): Name of the memory allocator (ex: ``"malloc"``). * ``coerce_c_locale`` (``int``, default: 0): If equals to 2, coerce the C locale; if equals to 1, read the LC_CTYPE locale to decide if it should be coerced. * ``coerce_c_locale_warn`` (``int``, default: 0): If non-zero, emit a warning if the C locale is coerced. * ``dev_mode`` (``int``, default: 0): See ``PyConfig.dev_mode``. * ``isolated`` (``int``, default: 0): See ``PyConfig.isolated``. * ``legacy_windows_fs_encoding`` (``int``, Windows only, default: 0): If non-zero, disable UTF-8 Mode, set the Python filesystem encoding to ``mbcs``, set the filesystem error handler to ``replace``. * ``use_environment`` (``int``, default: 1): See ``PyConfig.use_environment``. * ``utf8_mode`` (``int``, default: 0): If non-zero, enable the UTF-8 mode. ``PyPreConfig`` private field, for internal use only: * ``_config_version`` (``int``, default: config version): Configuration version, used for ABI compatibility. The C locale coercion (PEP 538) and the UTF-8 Mode (PEP 540) are disabled by default in ``PyPreConfig``. Set ``coerce_c_locale``, ``coerce_c_locale_warn`` and ``utf8_mode`` to ``-1`` to let Python enable them depending on the user configuration. In this case, it's safer to explicitly pre-initialize Python to ensure that encodings are configured before the Python initialization starts. Example to get the same encoding than regular Python:: PyPreConfig preconfig = PyPreConfig_INIT; preconfig.coerce_c_locale = -1; preconfig.coerce_c_locale_warn = -1; preconfig.utf8_mode = -1; PyInitError err = Py_PreInitialize(&preconfig); if (Py_INIT_FAILED(err)) { Py_ExitInitError(err); } Initialization with PyConfig ---------------------------- The ``PyConfig`` structure contains all parameters to configure Python. Example setting the program name:: PyInitError err; PyConfig config = PyConfig_INIT; err = PyConfig_SetString(&config.program_name, L"my_program"); if (_Py_INIT_FAILED(err)) { Py_ExitInitError(err); } err = Py_InitializeFromConfig(&config); PyConfig_Clear(&config); if (Py_INIT_FAILED(err)) { Py_ExitInitError(err); } ``PyConfig`` methods: * ``PyInitError PyConfig_SetString(PyConfig *config, wchar_t **config_str, const wchar_t *str)``: Copy the wide character string *str* into ``*config_str``. * ``PyInitError PyConfig_SetBytesString(PyConfig *config, wchar_t **config_str, const char *str)``: Decode *str* using ``Py_DecodeLocale()`` and set the result into ``*config_str``. Pre-initialize Python if needed to ensure that encodings are properly configured. * ``PyInitError PyConfig_SetArgv(PyConfig *config, int argc, wchar_t **argv)``: Set command line arguments from wide character strings. * ``PyInitError PyConfig_SetBytesArgv(PyConfig *config, int argc, char **argv)``: Set command line arguments: decode bytes using ``Py_DecodeLocale()``. Pre-initialize Python if needed to ensure that encodings are properly configured. * ``PyInitError PyConfig_Read(PyConfig *config)``: Read all Python configuration. Fields which are already set are left unchanged. * ``void PyConfig_Clear(PyConfig *config)``: Release configuration memory. Functions to initialize Python: * ``PyInitError Py_InitializeFromConfig(const PyConfig *config)``: Initialize Python from *config* configuration. *config* can be ``NULL``. The caller of these methods and functions is responsible to handle failure or exit using ``Py_INIT_FAILED()`` and ``Py_ExitInitError()``. ``PyConfig`` fields: * ``argv`` (``PyWideStringList``, default: empty): Command line arguments, ``sys.argv``. It is parsed and updated by default, set ``parse_argv`` to 0 to avoid that. * ``base_exec_prefix`` (``wchar_t*``, default: ``NULL``): ``sys.base_exec_prefix``. * ``base_prefix`` (``wchar_t*``, default: ``NULL``): ``sys.base_prefix``. * ``buffered_stdio`` (``int``, default: 1): If equals to 0, enable unbuffered mode, make stdout and stderr streams to be unbuffered. * ``bytes_warning`` (``int``, default: 0): If equals to 1, issue a warning when comparing ``bytes`` or ``bytearray`` with ``str``, or comparing ``bytes`` with ``int``. If equal or greater to 2, raise a ``BytesWarning`` exception. * ``check_hash_pycs_mode`` (``wchar_t*``, default: ``"default"``): ``--check-hash-based-pycs`` command line option value (see PEP 552). * ``configure_c_stdio`` (``int``, default: 1): If non-zero, configure C standard streams (``stdio``, ``stdout``, ``stdout``). For example, set their mode to ``O_BINARY`` on Windows. * ``dev_mode`` (``int``, default: 0): Development mode * ``dll_path`` (``wchar_t*``, Windows only, default: ``NULL``): Windows DLL path. * ``dump_refs`` (``int``, default: 0): If non-zero, dump all objects which are still alive at exit * ``exec_prefix`` (``wchar_t*``, default: ``NULL``): ``sys.exec_prefix``. * ``executable`` (``wchar_t*``, default: ``NULL``): ``sys.executable``. * ``faulthandler`` (``int``, default: 0): If non-zero, call ``faulthandler.enable()``. * ``filesystem_encoding`` (``wchar_t*``, default: ``NULL``): Filesystem encoding, ``sys.getfilesystemencoding()``. * ``filesystem_errors`` (``wchar_t*``, default: ``NULL``): Filesystem encoding errors, ``sys.getfilesystemencodeerrors()``. * ``use_hash_seed`` (``int``, default: 0), ``hash_seed`` (``unsigned long``, default: 0): Randomized hash function seed. * ``home`` (``wchar_t*``, default: ``NULL``): Python home directory. * ``import_time`` (``int``, default: 0): If non-zero, profile import time. * ``inspect`` (``int``, default: 0): Enter interactive mode after executing a script or a command. * ``install_signal_handlers`` (``int``, default: 1): Install signal handlers? * ``interactive`` (``int``, default: 0): Interactive mode. * ``legacy_windows_stdio`` (``int``, Windows only, default: 0): If non-zero, use ``io.FileIO`` instead of ``WindowsConsoleIO`` for ``sys.stdin``, ``sys.stdout`` and ``sys.stderr``. * ``malloc_stats`` (``int``, default: 0): If non-zero, dump memory allocation statistics at exit. * ``module_search_path_env`` (``wchar_t*``, default: ``NULL``): ``PYTHONPATH`` environment variale value. * ``use_module_search_paths`` (``int``, default: 0), ``module_search_paths`` (``PyWideStringList``, default: empty): ``sys.path``. * ``optimization_level`` (``int``, default: 0): Compilation optimization level. * ``parse_argv`` (``int``, default: 1): If non-zero, parse ``argv`` command line arguments and update ``argv``. * ``parser_debug`` (``int``, default: 0): If non-zero, turn on parser debugging output (for expert only, depending on compilation options). * ``pathconfig_warnings`` (``int``, default: 1): If equal to 0, suppress warnings when computing the path configuration. * ``prefix`` (``wchar_t*``, default: ``NULL``): ``sys.prefix``. * ``program_name`` (``wchar_t*``, default: ``NULL``): Program name. * ``program`` (``wchar_t*``, default: ``NULL``): ``argv[0]`` or an empty string. * ``pycache_prefix`` (``wchar_t*``, default: ``NULL``): ``.pyc`` cache prefix. * ``quiet`` (``int``, default: 0): Quiet mode. For example, don't display the copyright and version messages even in interactive mode. * ``run_command`` (``wchar_t*``, default: ``NULL``): ``-c COMMAND`` argument. * ``run_filename`` (``wchar_t*``), default: ``NULL``: ``python3 SCRIPT`` argument. * ``run_module`` (``wchar_t*``, default: ``NULL``): ``python3 -m MODULE`` argument. * ``show_alloc_count`` (``int``, default: 0): Show allocation counts at exit? * ``show_ref_count`` (``int``, default: 0): Show total reference count at exit? * ``site_import`` (``int``, default: 1): Import the ``site`` module at startup? * ``skip_source_first_line`` (``int``, default: 0): Skip the first line of the source? * ``stdio_encoding`` (``wchar_t*``, default: ``NULL``), ``stdio_errors`` (``wchar_t*``, default: ``NULL``): Encoding and encoding errors of ``sys.stdin``, ``sys.stdout`` and ``sys.stderr``. * ``tracemalloc`` (``int``, default: 0): If non-zero, call ``tracemalloc.start(value)``. * ``user_site_directory`` (``int``, default: 1): If non-zero, add user site directory to ``sys.path``. * ``verbose`` (``int``, default: 0): If non-zero, enable verbose mode. * ``warnoptions`` (``PyWideStringList``, default: empty): Options of the ``warnings`` module to build warnings filters. * ``write_bytecode`` (``int``, default: 1): If non-zero, write ``.pyc`` files. * ``xoptions`` (``PyWideStringList``, default: empty): ``sys._xoptions``. ``PyConfig`` private fields, for internal use only: * ``_config_version`` (``int``, default: config version): Configuration version, used for ABI compatibility. * ``_install_importlib`` (``int``, default: 1): Install importlib? * ``_init_main`` (``int``, default: 1): If equal to 0, stop Python initialization before the "main" phase (see PEP 432). By default, the ``argv`` arguments are parsed as regular Python command line arguments and ``argv`` is updated to strip parsed Python arguments: see `Command Line Arguments`_. Set ``parse_argv`` to 0 to avoid parsing and updating ``argv``. If ``argv`` is empty, an empty string is added to ensure that ``sys.argv`` always exists and is never empty. The ``xoptions`` options are parsed to set other options: see `-X Options`_. More complete example modifying the configuration before calling ``PyConfig_Read()``, and then modify the read configuration:: PyInitError init_python(const char *program_name) { PyInitError err; PyConfig config = PyConfig_INIT; /* Set the program name before reading the configuraton (decode byte string from the locale encoding) */ err = PyConfig_SetBytesString(&config.program_name, program_name); if (_Py_INIT_FAILED(err)) { goto fail; } /* Read all configuration at once */ err = PyConfig_Read(&config); if (_Py_INIT_FAILED(err)) { goto fail; } /* Append our custom search path to sys.path */ err = PyWideStringList_Append(&config.module_search_paths, L"/path/to/more/modules"); if (_Py_INIT_FAILED(err)) { goto fail; } /* Override executable computed by PyConfig_Read() */ err = PyConfig_SetString(&config, &config.executable, L"my_executable"); if (_Py_INIT_FAILED(err)) { goto fail; } err = Py_InitializeFromConfig(&config); /* Py_InitializeFromConfig() copied config which must now be cleared to release memory */ PyConfig_Clear(&config); return err; fail: PyConfig_Clear(&config); Py_ExitInitError(err); } .. note:: ``PyConfig`` does not have any field for extra inittab functions: ``PyImport_AppendInittab()`` and ``PyImport_ExtendInittab()`` functions are still relevant (and can be called before Python initialization). Initialization with constant PyConfig ------------------------------------- When no ``PyConfig`` method is used but only ``Py_InitializeFromConfig()``, the caller is responsible for managing ``PyConfig`` memory. In that case, constant strings and constant string lists can be used to avoid dynamically allocated memory. It can be used for most simple configurations. Example of Python initialization enabling the isolated mode:: PyConfig config = PyConfig_INIT; config.isolated = 1; PyInitError err = Py_InitializeFromConfig(&config); if (Py_INIT_FAILED(err)) { Py_ExitInitError(err); } /* ... use Python API here ... */ Py_Finalize(); ``PyConfig_Clear()`` is not needed in this example since ``config`` does not contain any dynamically allocated string: ``Py_InitializeFromConfig`` is responsible to fill other fields and manage the memory. For convenience, two other functions are provided for constant ``PyConfig``: * ``PyInitError Py_InitializeFromArgs(const PyConfig *config, int argc, wchar_t **argv)`` * ``PyInitError Py_InitializeFromBytesArgs(const PyConfig *config, int argc, char **argv)`` They be called with *config* set to ``NULL``. The caller of these functions is responsible to handle failure or exit using ``Py_INIT_FAILED()`` and ``Py_ExitInitError()``. Path Configuration ------------------ ``PyConfig`` contains multiple fields for the path configuration: * Path configuration input fields: * ``home`` * ``module_search_path_env`` * ``pathconfig_warnings`` * Path configuration output fields: * ``dll_path`` (Windows only) * ``exec_prefix`` * ``executable`` * ``prefix`` * ``use_module_search_paths``, ``module_search_paths`` Set ``pathconfig_warnings`` to 0 to suppress warnings when computing the path configuration. It is possible to completely ignore the function computing the default path configuration by setting explicitly all path configuration output fields listed above. A string is considered as set even if it's an empty string. ``module_search_paths`` is considered as set if ``use_module_search_paths`` is set to 1. In this case, path configuration input fields are ignored as well. If ``base_prefix`` or ``base_exec_prefix`` fields are not set, they inherit their value from ``prefix`` and ``exec_prefix`` respectively. If ``site_import`` is non-zero, ``sys.path`` can be modified by the ``site`` module. For example, if ``user_site_directory`` is non-zero, the user site directory is added to ``sys.path`` (if it exists). Isolate Python -------------- The default configuration is designed to behave as a regular Python. To embed Python into an application, it's possible to tune the configuration to better isolated the embedded Python from the system: * Set ``isolated`` to 1 to ignore environment variables and not prepend the current directory to ``sys.path``. * Set the `Path Configuration`_ ("output fields") to ignore the function computing the default path configuration. Py_BytesMain() -------------- Python 3.7 provides a high-level ``Py_Main()`` function which requires to pass command line arguments as ``wchar_t*`` strings. It is non-trivial to use the correct encoding to decode bytes. Python has its own set of issues with C locale coercion and UTF-8 Mode. This PEP adds a new ``Py_BytesMain()`` function which takes command line arguments as bytes:: int Py_BytesMain(int argc, char **argv) Py_RunMain() ------------ The new ``Py_RunMain()`` function executes the command (``PyConfig.run_command``), the script (``PyConfig.run_filename``) or the module (``PyConfig.run_module``) specified on the command line or in the configuration, and then finalizes Python. It returns an exit status that can be passed to the ``exit()`` function. Example of customized Python in isolated mode:: #include <Python.h> int main(int argc, char *argv[]) { PyConfig config = PyConfig_INIT; config.isolated = 1; PyInitError err = Py_InitializeFromBytesArgs(&config, argc, argv); if (Py_INIT_FAILED(err)) { Py_ExitInitError(err); } /* put more configuration code here if needed */ return Py_RunMain(); } The example is a basic implementation of the "System Python Executable" discussed in PEP 432. Memory allocations and Py_DecodeLocale() ---------------------------------------- Python memory allocation functions like ``PyMem_RawMalloc()`` must not be used before Python pre-initialization, whereas calling directly ``malloc()`` and ``free()`` is always safe. For ``PyPreConfig`` and constant ``PyConfig``, the caller is responsible to manage dynamically allocated memory; constant strings and constant string lists can be used to avoid memory allocations. Dynamic ``PyConfig`` requires to call ``PyConfig_Clear()`` to release memory. ``Py_DecodeLocale()`` must not be called before the pre-initialization. Backwards Compatibility ======================= This PEP only adds a new API: it leaves the existing API unchanged and has no impact on the backwards compatibility. The implementation ensures that the existing API is compatible with the new API. For example, ``PyConfig`` uses the value of global configuration variables as default values. Annex: Python Configuration =========================== Priority and Rules ------------------ Priority of configuration parameters, highest to lowest: * ``PyConfig`` * ``PyPreConfig`` * Configuration files * Command line options * Environment variables * Global configuration variables Priority of warning options, highest to lowest: * ``PyConfig.warnoptions`` * ``PyConfig.dev_mode`` (add ``"default"``) * ``PYTHONWARNINGS`` environment variables * ``-W WARNOPTION`` command line argument * ``PyConfig.bytes_warning`` (add ``"error::BytesWarning"`` if greater than 1, or add ``"default::BytesWarning``) Rules on ``PyConfig`` parameters: * If ``isolated`` is non-zero, ``use_environment`` and ``user_site_directory`` are set to 0. * If ``legacy_windows_fs_encoding`` is non-zero, ``utf8_mode`` is set to 0. * If ``dev_mode`` is non-zero, ``allocator`` is set to ``"debug"``, ``faulthandler`` is set to 1, and ``"default"`` filter is added to ``warnoptions``. But the ``PYTHONMALLOC`` environment variable has the priority over ``dev_mode`` to set the memory allocator. * If ``base_prefix`` is not set, it inherits ``prefix`` value. * If ``base_exec_prefix`` is not set, it inherits ``exec_prefix`` value. * If the ``python._pth`` configuration file is present, ``isolated`` is set to 1 and ``site_import`` is set to 0; but ``site_import`` is set to 1 if ``python._pth`` contains ``import site``. Rules on ``PyConfig`` and ``PyPreConfig`` parameters: * If ``PyPreConfig.legacy_windows_fs_encoding`` is non-zero, set ``PyConfig.utf8_mode`` to 0, set ``PyConfig.filesystem_encoding`` to ``mbcs``, and set ``PyConfig.filesystem_errors`` to ``replace``. Configuration Files ------------------- Python configuration files: * ``pyvenv.cfg`` * ``python._pth`` (Windows only) * ``pybuilddir.txt`` (Unix only) Global Configuration Variables ------------------------------ Global configuration variables mapped to ``PyPreConfig`` fields: ======================================== ================================ Variable Field ======================================== ================================ ``Py_IgnoreEnvironmentFlag`` ``use_environment`` (NOT) ``Py_IsolatedFlag`` ``isolated`` ``Py_LegacyWindowsFSEncodingFlag`` ``legacy_windows_fs_encoding`` ``Py_UTF8Mode`` ``utf8_mode`` ======================================== ================================ (NOT) means that the ``PyPreConfig`` value is the oposite of the global configuration variable value. Global configuration variables mapped to ``PyConfig`` fields: ======================================== ================================ Variable Field ======================================== ================================ ``Py_BytesWarningFlag`` ``bytes_warning`` ``Py_DebugFlag`` ``parser_debug`` ``Py_DontWriteBytecodeFlag`` ``write_bytecode`` (NOT) ``Py_FileSystemDefaultEncodeErrors`` ``filesystem_errors`` ``Py_FileSystemDefaultEncoding`` ``filesystem_encoding`` ``Py_FrozenFlag`` ``pathconfig_warnings`` (NOT) ``Py_HasFileSystemDefaultEncoding`` ``filesystem_encoding`` ``Py_HashRandomizationFlag`` ``use_hash_seed``, ``hash_seed`` ``Py_IgnoreEnvironmentFlag`` ``use_environment`` (NOT) ``Py_InspectFlag`` ``inspect`` ``Py_InteractiveFlag`` ``interactive`` ``Py_IsolatedFlag`` ``isolated`` ``Py_LegacyWindowsStdioFlag`` ``legacy_windows_stdio`` ``Py_NoSiteFlag`` ``site_import`` (NOT) ``Py_NoUserSiteDirectory`` ``user_site_directory`` (NOT) ``Py_OptimizeFlag`` ``optimization_level`` ``Py_QuietFlag`` ``quiet`` ``Py_UnbufferedStdioFlag`` ``buffered_stdio`` (NOT) ``Py_VerboseFlag`` ``verbose`` ``_Py_HasFileSystemDefaultEncodeErrors`` ``filesystem_errors`` ======================================== ================================ (NOT) means that the ``PyConfig`` value is the oposite of the global configuration variable value. ``Py_LegacyWindowsFSEncodingFlag`` and ``Py_LegacyWindowsStdioFlag`` are only available on Windows. Command Line Arguments ---------------------- Usage:: python3 [options] python3 [options] -c COMMAND python3 [options] -m MODULE python3 [options] SCRIPT Command line options mapped to pseudo-action on ``PyPreConfig`` fields: ================================ ================================ Option ``PyConfig`` field ================================ ================================ ``-E`` ``use_environment = 0`` ``-I`` ``isolated = 1`` ``-X dev`` ``dev_mode = 1`` ``-X utf8`` ``utf8_mode = 1`` ``-X utf8=VALUE`` ``utf8_mode = VALUE`` ================================ ================================ Command line options mapped to pseudo-action on ``PyConfig`` fields: ================================ ================================ Option ``PyConfig`` field ================================ ================================ ``-b`` ``bytes_warning++`` ``-B`` ``write_bytecode = 0`` ``-c COMMAND`` ``run_command = COMMAND`` ``--check-hash-based-pycs=MODE`` ``_check_hash_pycs_mode = MODE`` ``-d`` ``parser_debug++`` ``-E`` ``use_environment = 0`` ``-i`` ``inspect++`` and ``interactive++`` ``-I`` ``isolated = 1`` ``-m MODULE`` ``run_module = MODULE`` ``-O`` ``optimization_level++`` ``-q`` ``quiet++`` ``-R`` ``use_hash_seed = 0`` ``-s`` ``user_site_directory = 0`` ``-S`` ``site_import`` ``-t`` ignored (kept for backwards compatibility) ``-u`` ``buffered_stdio = 0`` ``-v`` ``verbose++`` ``-W WARNING`` add ``WARNING`` to ``warnoptions`` ``-x`` ``skip_source_first_line = 1`` ``-X OPTION`` add ``OPTION`` to ``xoptions`` ================================ ================================ ``-h``, ``-?`` and ``-V`` options are handled without ``PyConfig``. -X Options ---------- -X options mapped to pseudo-action on ``PyConfig`` fields: ================================ ================================ Option ``PyConfig`` field ================================ ================================ ``-X dev`` ``dev_mode = 1`` ``-X faulthandler`` ``faulthandler = 1`` ``-X importtime`` ``import_time = 1`` ``-X pycache_prefix=PREFIX`` ``pycache_prefix = PREFIX`` ``-X showalloccount`` ``show_alloc_count = 1`` ``-X showrefcount`` ``show_ref_count = 1`` ``-X tracemalloc=N`` ``tracemalloc = N`` ================================ ================================ Environment Variables --------------------- Environment variables mapped to ``PyPreConfig`` fields: ================================= ============================================= Variable ``PyPreConfig`` field ================================= ============================================= ``PYTHONCOERCECLOCALE`` ``coerce_c_locale``, ``coerce_c_locale_warn`` ``PYTHONDEVMODE`` ``dev_mode`` ``PYTHONLEGACYWINDOWSFSENCODING`` ``legacy_windows_fs_encoding`` ``PYTHONMALLOC`` ``allocator`` ``PYTHONUTF8`` ``utf8_mode`` ================================= ============================================= Environment variables mapped to ``PyConfig`` fields: ================================= ==================================== Variable ``PyConfig`` field ================================= ==================================== ``PYTHONDEBUG`` ``parser_debug`` ``PYTHONDEVMODE`` ``dev_mode`` ``PYTHONDONTWRITEBYTECODE`` ``write_bytecode`` ``PYTHONDUMPREFS`` ``dump_refs`` ``PYTHONEXECUTABLE`` ``program_name`` ``PYTHONFAULTHANDLER`` ``faulthandler`` ``PYTHONHASHSEED`` ``use_hash_seed``, ``hash_seed`` ``PYTHONHOME`` ``home`` ``PYTHONINSPECT`` ``inspect`` ``PYTHONIOENCODING`` ``stdio_encoding``, ``stdio_errors`` ``PYTHONLEGACYWINDOWSSTDIO`` ``legacy_windows_stdio`` ``PYTHONMALLOCSTATS`` ``malloc_stats`` ``PYTHONNOUSERSITE`` ``user_site_directory`` ``PYTHONOPTIMIZE`` ``optimization_level`` ``PYTHONPATH`` ``module_search_path_env`` ``PYTHONPROFILEIMPORTTIME`` ``import_time`` ``PYTHONPYCACHEPREFIX,`` ``pycache_prefix`` ``PYTHONTRACEMALLOC`` ``tracemalloc`` ``PYTHONUNBUFFERED`` ``buffered_stdio`` ``PYTHONVERBOSE`` ``verbose`` ``PYTHONWARNINGS`` ``warnoptions`` ================================= ==================================== ``PYTHONLEGACYWINDOWSFSENCODING`` and ``PYTHONLEGACYWINDOWSSTDIO`` are specific to Windows. Annex: Python 3.7 API ===================== Python 3.7 has 4 functions in its C API to initialize and finalize Python: * ``Py_Initialize()``, ``Py_InitializeEx()``: initialize Python * ``Py_Finalize()``, ``Py_FinalizeEx()``: finalize Python Python 3.7 can be configured using `Global Configuration Variables`_, `Environment Variables`_, and the following functions: * ``PyImport_AppendInittab()`` * ``PyImport_ExtendInittab()`` * ``PyMem_SetAllocator()`` * ``PyMem_SetupDebugHooks()`` * ``PyObject_SetArenaAllocator()`` * ``Py_SetPath()`` * ``Py_SetProgramName()`` * ``Py_SetPythonHome()`` * ``Py_SetStandardStreamEncoding()`` * ``PySys_AddWarnOption()`` * ``PySys_AddXOption()`` * ``PySys_ResetWarnOptions()`` There is also a high-level ``Py_Main()`` function. Python Issues ============= Issues that will be fixed by this PEP, directly or indirectly: * `bpo-1195571 <https://bugs.python.org/issue1195571>`_: "simple callback system for Py_FatalError" * `bpo-11320 <https://bugs.python.org/issue11320>`_: "Usage of API method Py_SetPath causes errors in Py_Initialize() (Posix ony)" * `bpo-13533 <https://bugs.python.org/issue13533>`_: "Would like Py_Initialize to play friendly with host app" * `bpo-14956 <https://bugs.python.org/issue14956>`_: "custom PYTHONPATH may break apps embedding Python" * `bpo-19983 <https://bugs.python.org/issue19983>`_: "When interrupted during startup, Python should not call abort() but exit()" * `bpo-22213 <https://bugs.python.org/issue22213>`_: "Make pyvenv style virtual environments easier to configure when embedding Python". This PEP more or * `bpo-22257 <https://bugs.python.org/issue22257>`_: "PEP 432: Redesign the interpreter startup sequence" * `bpo-29778 <https://bugs.python.org/issue29778>`_: "_Py_CheckPython3 uses uninitialized dllpath when embedder sets module path with Py_SetPath" * `bpo-30560 <https://bugs.python.org/issue30560>`_: "Add Py_SetFatalErrorAbortFunc: Allow embedding program to handle fatal errors". * `bpo-31745 <https://bugs.python.org/issue31745>`_: "Overloading "Py_GetPath" does not work" * `bpo-32573 <https://bugs.python.org/issue32573>`_: "All sys attributes (.argv, ...) should exist in embedded environments". * `bpo-34725 <https://bugs.python.org/issue34725>`_: "Py_GetProgramFullPath() odd behaviour in Windows" * `bpo-36204 <https://bugs.python.org/issue36204>`_: "Deprecate calling Py_Main() after Py_Initialize()? Add Py_InitializeFromArgv()?" * `bpo-33135 <https://bugs.python.org/issue33135>`_: "Define field prefixes for the various config structs". The PEP now defines well how warnings options are handled. Issues of the PEP implementation: * `bpo-16961 <https://bugs.python.org/issue16961>`_: "No regression tests for -E and individual environment vars" * `bpo-20361 <https://bugs.python.org/issue20361>`_: "-W command line options and PYTHONWARNINGS environmental variable should not override -b / -bb command line options" * `bpo-26122 <https://bugs.python.org/issue26122>`_: "Isolated mode doesn't ignore PYTHONHASHSEED" * `bpo-29818 <https://bugs.python.org/issue29818>`_: "Py_SetStandardStreamEncoding leads to a memory error in debug mode" * `bpo-31845 <https://bugs.python.org/issue31845>`_: "PYTHONDONTWRITEBYTECODE and PYTHONOPTIMIZE have no effect" * `bpo-32030 <https://bugs.python.org/issue32030>`_: "PEP 432: Rewrite Py_Main()" * `bpo-32124 <https://bugs.python.org/issue32124>`_: "Document functions safe to be called before Py_Initialize()" * `bpo-33042 <https://bugs.python.org/issue33042>`_: "New 3.7 startup sequence crashes PyInstaller" * `bpo-33932 <https://bugs.python.org/issue33932>`_: "Calling Py_Initialize() twice now triggers a fatal error (Python 3.7)" * `bpo-34008 <https://bugs.python.org/issue34008>`_: "Do we support calling Py_Main() after Py_Initialize()?" * `bpo-34170 <https://bugs.python.org/issue34170>`_: "Py_Initialize(): computing path configuration must not have side effect (PEP 432)" * `bpo-34589 <https://bugs.python.org/issue34589>`_: "Py_Initialize() and Py_Main() should not enable C locale coercion" * `bpo-34639 <https://bugs.python.org/issue34639>`_: "PYTHONCOERCECLOCALE is ignored when using -E or -I option" * `bpo-36142 <https://bugs.python.org/issue36142>`_: "Add a new _PyPreConfig step to Python initialization to setup memory allocator and encodings" * `bpo-36202 <https://bugs.python.org/issue36202>`_: "Calling Py_DecodeLocale() before _PyPreConfig_Write() can produce mojibake" * `bpo-36301 <https://bugs.python.org/issue36301>`_: "Add _Py_PreInitialize() function" * `bpo-36443 <https://bugs.python.org/issue36443>`_: "Disable coerce_c_locale and utf8_mode by default in _PyPreConfig?" * `bpo-36444 <https://bugs.python.org/issue36444>`_: "Python initialization: remove _PyMainInterpreterConfig" * `bpo-36471 <https://bugs.python.org/issue36471>`_: "PEP 432, PEP 587: Add _Py_RunMain()" * `bpo-36763 <https://bugs.python.org/issue36763>`_: "PEP 587: Rework initialization API to prepare second version of the PEP" * `bpo-36775 <https://bugs.python.org/issue36775>`_: "Rework filesystem codec implementation" * `bpo-36900 <https://bugs.python.org/issue36900>`_: "Use _PyCoreConfig rather than global configuration variables" Issues related to this PEP: * `bpo-12598 <https://bugs.python.org/issue12598>`_: "Move sys variable initialization from import.c to sysmodule.c" * `bpo-15577 <https://bugs.python.org/issue15577>`_: "Real argc and argv in embedded interpreter" * `bpo-16202 <https://bugs.python.org/issue16202>`_: "sys.path[0] security issues" * `bpo-18309 <https://bugs.python.org/issue18309>`_: "Make python slightly more relocatable" * `bpo-25631 <https://bugs.python.org/issue25631>`_: "Segmentation fault with invalid Unicode command-line arguments in embedded Python" * `bpo-26007 <https://bugs.python.org/issue26007>`_: "Support embedding the standard library in an executable" * `bpo-31210 <https://bugs.python.org/issue31210>`_: "Can not import modules if sys.prefix contains DELIM". * `bpo-31349 <https://bugs.python.org/issue31349>`_: "Embedded initialization ignores Py_SetProgramName()" * `bpo-33919 <https://bugs.python.org/issue33919>`_: "Expose _PyCoreConfig structure to Python" * `bpo-35173 <https://bugs.python.org/issue35173>`_: "Re-use already existing functionality to allow Python 2.7.x (both embedded and standalone) to locate the module path according to the shared library" Version History =============== * Version 3: * ``PyConfig``: Add ``configure_c_stdio`` and ``parse_argv``, rename ``_frozen`` to ``pathconfig_warnings``. * Rename functions using bytes strings and wide character strings. For example, ``Py_PreInitializeFromWideArgs`` becomes ``Py_PreInitializeFromArgs``, and ``PyConfig_SetArgv`` becomes ``PyConfig_SetBytesArgv``. * Add ``PyWideStringList_Insert()`` function. * New "Path configuration", "Isolate Python", "Python Issues" and "Version History" sections. * ``PyConfig_SetString()`` and ``PyConfig_SetBytesString()`` now requires the configuration as the first argument. * Rename ``Py_UnixMain()`` to ``Py_BytesMain()`` * Version 2: Add ``PyConfig`` methods (ex: ``PyConfig_Read()``), add ``PyWideStringList_Append()``, rename ``PyWideCharList`` to ``PyWideStringList``. * Version 1: Initial version. Copyright ========= This document has been placed in the public domain.

5 11

Summary of Python tracker Issues
by Python tracker 31 May '19

31 May '19

ACTIVITY SUMMARY (2019-05-24 - 2019-05-31) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 7028 (-39) closed 41839 (+118) total 48867 (+79) Open issues with patches: 2820 Issues opened (50) ================== #26423: Integer overflow in wrap_lenfunc() on 64-bit build of Windows https://bugs.python.org/issue26423 reopened by vstinner #34725: Py_GetProgramFullPath() odd behaviour in Windows https://bugs.python.org/issue34725 reopened by vstinner #35753: Importing call from unittest.mock directly causes ValueError https://bugs.python.org/issue35753 reopened by pablogsal #36889: Merge StreamWriter and StreamReader into just asyncio.Stream https://bugs.python.org/issue36889 reopened by koobs #36991: zipfile: AttributeError on extract https://bugs.python.org/issue36991 reopened by berker.peksag #37038: Make idlelib/run.py runnable. https://bugs.python.org/issue37038 opened by terry.reedy #37039: IDLE: Improve zoomheight doc and behavior. https://bugs.python.org/issue37039 opened by terry.reedy #37040: checking for membership in itertools.count enters infinite loo https://bugs.python.org/issue37040 opened by Dan Rose #37041: IDLE: path browser unusable on some displays https://bugs.python.org/issue37041 opened by aroberge #37042: wait_for(coro, timeout=0) memleak https://bugs.python.org/issue37042 opened by ixje #37043: Buildbots fail when new files are added https://bugs.python.org/issue37043 opened by jaraco #37044: Build/test artifacts not ignored for framework build https://bugs.python.org/issue37044 opened by jaraco #37048: ssl module: QUIC support for HTTP/3 https://bugs.python.org/issue37048 opened by christian.heimes #37052: Add examples for mocking async for and async context manager i https://bugs.python.org/issue37052 opened by xtreak #37055: Numerous warnings with blake2 module https://bugs.python.org/issue37055 opened by remi.lapeyre #37060: import ctypes fails with a statically linked interpreter due t https://bugs.python.org/issue37060 opened by indygreg #37062: `AutoNumber` class in enum documentation: support *args in con https://bugs.python.org/issue37062 opened by rrt #37063: Incorrect application of func.__defaults__ by inspect's signat https://bugs.python.org/issue37063 opened by bup #37064: Feature request: option to keep/add flags to pathfix. https://bugs.python.org/issue37064 opened by pkopkan #37068: Emit SyntaxWarning for f-strings without expressions ? https://bugs.python.org/issue37068 opened by mbussonn #37069: regrtest: log unraisable exceptions and uncaught thread except https://bugs.python.org/issue37069 opened by vstinner #37071: HTMLParser mistakenly inventing new tags while parsing https://bugs.python.org/issue37071 opened by htran #37073: clarify functions docs in IO modules and Bytes Objects https://bugs.python.org/issue37073 opened by Windson Yang #37074: os.stat() does not work for NUL and CON https://bugs.python.org/issue37074 opened by serhiy.storchaka #37075: Error message improvement for AsyncMock https://bugs.python.org/issue37075 opened by xtreak #37077: Threading: add builtin TID attribute to Thread objects for AIX https://bugs.python.org/issue37077 opened by Michael.Felt #37080: Cannot compile Python3.7.3 on Alt-F (ARM) https://bugs.python.org/issue37080 opened by Jarl Gullberg #37081: Test with OpenSSL 1.1.1c https://bugs.python.org/issue37081 opened by christian.heimes #37082: Assignment expression operator (walrus) not in built-in help() https://bugs.python.org/issue37082 opened by mbussonn #37083: Document TYPE_COMMENT in documentation reference for compound https://bugs.python.org/issue37083 opened by pablogsal #37084: _ctypes not failing, can't find reason https://bugs.python.org/issue37084 opened by Brian Spratke #37085: Expose additional socket constants for CAN_BCM flags https://bugs.python.org/issue37085 opened by karlding #37086: time.sleep error message misleading https://bugs.python.org/issue37086 opened by Justin Fay #37087: Adding native id support for openbsd https://bugs.python.org/issue37087 opened by David Carlier #37088: Add a way to schedule a function to be called from the main th https://bugs.python.org/issue37088 opened by yselivanov #37089: `import Lib.os` works on windows (but shouldn't) https://bugs.python.org/issue37089 opened by Anthony Sottile #37090: test_gdb's test_pycfunction should test all calling convention https://bugs.python.org/issue37090 opened by petr.viktorin #37091: subprocess - uncaught PermissionError in send_signal can cause https://bugs.python.org/issue37091 opened by wesinator #37093: http.client aborts header parsing upon encountering non-ASCII https://bugs.python.org/issue37093 opened by tburke #37095: [Feature Request]: Add zstd support in tarfile https://bugs.python.org/issue37095 opened by evan0greenup #37096: Add large-file tests for modules using sendfile(2) https://bugs.python.org/issue37096 opened by giampaolo.rodola #37097: python_is_optimized() false negatives https://bugs.python.org/issue37097 opened by jdemeyer #37098: test_memfd_create() test failure https://bugs.python.org/issue37098 opened by jdemeyer #37100: test_coroutine.test_unawaited_warning_when_module_broken fails https://bugs.python.org/issue37100 opened by xtreak #37102: Automatically dedent docstring constants by default https://bugs.python.org/issue37102 opened by gregory.p.smith #37103: Undo deprecation of optparse https://bugs.python.org/issue37103 opened by andersk #37105: Add deprecated-remove information on stream doc https://bugs.python.org/issue37105 opened by eamanu #37107: ensurepip --upgrade doesn't change the version of pip used by https://bugs.python.org/issue37107 opened by cjw296 #37110: Clarify hashability of custom class instances https://bugs.python.org/issue37110 opened by cito #37116: Use PEP 570 syntax for positional-only parameters https://bugs.python.org/issue37116 opened by serhiy.storchaka Most recent 15 issues with no replies (15) ========================================== #37116: Use PEP 570 syntax for positional-only parameters https://bugs.python.org/issue37116 #37110: Clarify hashability of custom class instances https://bugs.python.org/issue37110 #37107: ensurepip --upgrade doesn't change the version of pip used by https://bugs.python.org/issue37107 #37105: Add deprecated-remove information on stream doc https://bugs.python.org/issue37105 #37102: Automatically dedent docstring constants by default https://bugs.python.org/issue37102 #37097: python_is_optimized() false negatives https://bugs.python.org/issue37097 #37096: Add large-file tests for modules using sendfile(2) https://bugs.python.org/issue37096 #37095: [Feature Request]: Add zstd support in tarfile https://bugs.python.org/issue37095 #37093: http.client aborts header parsing upon encountering non-ASCII https://bugs.python.org/issue37093 #37090: test_gdb's test_pycfunction should test all calling convention https://bugs.python.org/issue37090 #37087: Adding native id support for openbsd https://bugs.python.org/issue37087 #37085: Expose additional socket constants for CAN_BCM flags https://bugs.python.org/issue37085 #37084: _ctypes not failing, can't find reason https://bugs.python.org/issue37084 #37080: Cannot compile Python3.7.3 on Alt-F (ARM) https://bugs.python.org/issue37080 #37077: Threading: add builtin TID attribute to Thread objects for AIX https://bugs.python.org/issue37077 Most recent 15 issues waiting for review (15) ============================================= #37116: Use PEP 570 syntax for positional-only parameters https://bugs.python.org/issue37116 #37105: Add deprecated-remove information on stream doc https://bugs.python.org/issue37105 #37098: test_memfd_create() test failure https://bugs.python.org/issue37098 #37096: Add large-file tests for modules using sendfile(2) https://bugs.python.org/issue37096 #37090: test_gdb's test_pycfunction should test all calling convention https://bugs.python.org/issue37090 #37088: Add a way to schedule a function to be called from the main th https://bugs.python.org/issue37088 #37085: Expose additional socket constants for CAN_BCM flags https://bugs.python.org/issue37085 #37081: Test with OpenSSL 1.1.1c https://bugs.python.org/issue37081 #37077: Threading: add builtin TID attribute to Thread objects for AIX https://bugs.python.org/issue37077 #37075: Error message improvement for AsyncMock https://bugs.python.org/issue37075 #37064: Feature request: option to keep/add flags to pathfix. https://bugs.python.org/issue37064 #37039: IDLE: Improve zoomheight doc and behavior. https://bugs.python.org/issue37039 #37038: Make idlelib/run.py runnable. https://bugs.python.org/issue37038 #37034: Argument Clinic omits name of keyword-only parameter on _PyArg https://bugs.python.org/issue37034 #37030: Lib/cmd.py: Hide undocumented commands in help and completenam https://bugs.python.org/issue37030 Top 10 most discussed issues (10) ================================= #37039: IDLE: Improve zoomheight doc and behavior. https://bugs.python.org/issue37039 20 msgs #33725: Python crashes on macOS after fork with no exec https://bugs.python.org/issue33725 16 msgs #36839: Support the buffer protocol in code objects https://bugs.python.org/issue36839 12 msgs #36974: Implement PEP 590 https://bugs.python.org/issue36974 12 msgs #35545: asyncio.base_events.create_connection doesn't handle scoped IP https://bugs.python.org/issue35545 9 msgs #36709: Asyncio SSL keep-alive connections raise errors after loop clo https://bugs.python.org/issue36709 9 msgs #36889: Merge StreamWriter and StreamReader into just asyncio.Stream https://bugs.python.org/issue36889 8 msgs #37011: pdb: restore original tracing function instead of sys.settrace https://bugs.python.org/issue37011 8 msgs #26423: Integer overflow in wrap_lenfunc() on 64-bit build of Windows https://bugs.python.org/issue26423 6 msgs #28866: Type cache is not correctly invalidated on a class defining mr https://bugs.python.org/issue28866 6 msgs Issues closed (115) =================== #5028: tokenize.generate_tokens doesn't always return logical line https://bugs.python.org/issue5028 closed by cheryl.sabella #11320: Usage of API method Py_SetPath causes errors in Py_Initialize( https://bugs.python.org/issue11320 closed by vstinner #13533: Would like Py_Initialize to play friendly with host app https://bugs.python.org/issue13533 closed by vstinner #14656: Add a macro for unreachable code https://bugs.python.org/issue14656 closed by pablogsal #14956: custom PYTHONPATH may break apps embedding Python https://bugs.python.org/issue14956 closed by vstinner #19569: Use __attribute__((deprecated)) to warn usage of deprecated fu https://bugs.python.org/issue19569 closed by vstinner #19983: When interrupted during startup, Python should not call abort( https://bugs.python.org/issue19983 closed by vstinner #22102: Zipfile generates Zipfile error in zip with 0 total number of https://bugs.python.org/issue22102 closed by cheryl.sabella #22454: Adding the opposite function of shlex.split() https://bugs.python.org/issue22454 closed by berker.peksag #22640: Add silent mode for py_compile https://bugs.python.org/issue22640 closed by berker.peksag #24564: shutil.copytree fails when copying NFS to NFS https://bugs.python.org/issue24564 closed by giampaolo.rodola #24732: 3.5.0b3 Windows accept() on unready non-blocking socket raises https://bugs.python.org/issue24732 closed by vstinner #25735: math.factorial doc should mention integer return type https://bugs.python.org/issue25735 closed by cheryl.sabella #27639: UserList.__getitem__ doesn't account for slices https://bugs.python.org/issue27639 closed by SilentGhost #27987: obmalloc's 8-byte alignment causes undefined behavior https://bugs.python.org/issue27987 closed by inada.naoki #29143: Logger should ignore propagate property for disabled handlers. https://bugs.python.org/issue29143 closed by mariocj89 #29262: Provide a way to check for *real* typing.Union instances https://bugs.python.org/issue29262 closed by levkivskyi #29696: Use namedtuple in string.Formatter.parse iterator response https://bugs.python.org/issue29696 closed by rhettinger #29778: _Py_CheckPython3 uses uninitialized dllpath when embedder sets https://bugs.python.org/issue29778 closed by vstinner #29883: asyncio: Windows Proactor Event Loop UDP Support https://bugs.python.org/issue29883 closed by asvetlov #30006: Deadlocks in `concurrent.futures.ProcessPoolExecutor` https://bugs.python.org/issue30006 closed by pitrou #30560: Py_SetFatalErrorAbortFunc: Allow embedding program to handle f https://bugs.python.org/issue30560 closed by vstinner #30969: Docs should say that `x is z or x == z` is used for `x in y` i https://bugs.python.org/issue30969 closed by cheryl.sabella #31745: Overloading "Py_GetPath" does not work https://bugs.python.org/issue31745 closed by vstinner #32299: unittest.mock.patch.dict.__enter__ should return the dict https://bugs.python.org/issue32299 closed by cheryl.sabella #32388: Remove cross-version binary compatibility https://bugs.python.org/issue32388 closed by pitrou #32941: mmap should expose madvise() https://bugs.python.org/issue32941 closed by pitrou #33006: docstring of filter function is incorrect https://bugs.python.org/issue33006 closed by ned.deily #33135: Define field prefixes for the various config structs https://bugs.python.org/issue33135 closed by vstinner #33407: Implement Py_DEPRECATED() macro for Visual Studio https://bugs.python.org/issue33407 closed by vstinner #33918: Hooking into pause/resume of iterators/coroutines https://bugs.python.org/issue33918 closed by asvetlov #33919: Expose _PyCoreConfig structure to Python https://bugs.python.org/issue33919 closed by vstinner #34148: Fatal read error on socket transport https://bugs.python.org/issue34148 closed by asvetlov #34613: asyncio.StreamReader initialization documentation incorrectly https://bugs.python.org/issue34613 closed by asvetlov #34626: PEP 384's PyType_Spec and PyType_Slot are not documented https://bugs.python.org/issue34626 closed by petr.viktorin #34632: Port importlib_metadata to Python 3.8 https://bugs.python.org/issue34632 closed by barry #34655: Support sendfile in asyncio streams API https://bugs.python.org/issue34655 closed by asvetlov #34791: xml package does not obey sys.flags.ignore_environment https://bugs.python.org/issue34791 closed by vstinner #34818: test.test_ssl.ThreadedTests.test_tls1_3 fails in 2.7 with Attr https://bugs.python.org/issue34818 closed by cheryl.sabella #34975: start_tls() difficult when using asyncio.start_server() https://bugs.python.org/issue34975 closed by asvetlov #34993: asyncio.streams.FlowControlMixin should be part of the API https://bugs.python.org/issue34993 closed by asvetlov #35246: asyncio.create_subprocess_exec doesn't accept pathlib.Path lik https://bugs.python.org/issue35246 closed by asvetlov #35279: asyncio uses too many threads by default https://bugs.python.org/issue35279 closed by inada.naoki #35397: Undeprecate and document urllib.parse.unwrap https://bugs.python.org/issue35397 closed by cheryl.sabella #35925: test_httplib test_nntplib test_ssl fail on ARMv7 Debian buster https://bugs.python.org/issue35925 closed by ned.deily #36204: Deprecate calling Py_Main() after Py_Initialize()? Add Py_Init https://bugs.python.org/issue36204 closed by vstinner #36271: '_io.TextIOWrapper' object has no attribute 'mode' https://bugs.python.org/issue36271 closed by berker.peksag #36540: PEP 570: Python Positional-Only Parameters https://bugs.python.org/issue36540 closed by pablogsal #36610: os.sendfile can return EINVAL on Solaris https://bugs.python.org/issue36610 closed by giampaolo.rodola #36686: Docs: asyncio.loop.subprocess_exec documentation is confusing, https://bugs.python.org/issue36686 closed by sbstp #36721: Add pkg-config python-3.8-embed and --embed to python3.8-confi https://bugs.python.org/issue36721 closed by vstinner #36739: "4.6. Defining Functions" should mention nonlocal https://bugs.python.org/issue36739 closed by Mariatta #36772: Let lru_cache be used as a decorator with no arguments https://bugs.python.org/issue36772 closed by rhettinger #36782: Add tests for the datetime C API https://bugs.python.org/issue36782 closed by vstinner #36794: asyncio.Lock documentation in Py3.8 lacks parts presented in d https://bugs.python.org/issue36794 closed by asvetlov #36809: Crash for test test_importlib https://bugs.python.org/issue36809 closed by vstinner #36829: Add sys.unraisablehook() to custom how "unraisable exceptions" https://bugs.python.org/issue36829 closed by vstinner #36840: Add stream.abort() async method https://bugs.python.org/issue36840 closed by asvetlov #36842: Implement PEP 578 https://bugs.python.org/issue36842 closed by steve.dower #36843: AIX build fails with failure to get random numbers https://bugs.python.org/issue36843 closed by vstinner #36856: faulthandler._stack_overflow doesn't work on x86-linux with KP https://bugs.python.org/issue36856 closed by vstinner #36891: Additional startup plugin for vendors https://bugs.python.org/issue36891 closed by inada.naoki #36922: Implement Py_TPFLAGS_METHOD_DESCRIPTOR https://bugs.python.org/issue36922 closed by petr.viktorin #36933: sys.set_coroutine_wrapper documented as to be removed in 3.8 ( https://bugs.python.org/issue36933 closed by asvetlov #36969: pdb.do_args: display keyword-only and positional only argument https://bugs.python.org/issue36969 closed by pablogsal #36983: typing.__all__ has drifted from actual contents https://bugs.python.org/issue36983 closed by gvanrossum #36996: unittest.mock.patch decorator doesn't work with async function https://bugs.python.org/issue36996 closed by asvetlov #36999: Expose the coroutine object in asyncio task objects https://bugs.python.org/issue36999 closed by asvetlov #37001: symtable.symtable doesn't accept bytes which leads to a mismat https://bugs.python.org/issue37001 closed by dino.viehland #37003: ast unparse does not support f-string new debug format. https://bugs.python.org/issue37003 closed by mbussonn #37007: Implement socket.if_{nametoindex,indextoname} for Windows https://bugs.python.org/issue37007 closed by steve.dower #37015: Fix asyncio mock warnings https://bugs.python.org/issue37015 closed by asvetlov #37017: Use LOAD_METHOD optimization in CallMethod C API functions https://bugs.python.org/issue37017 closed by inada.naoki #37021: Can _random.getrandbits() be converted to METH_FASTCALL? https://bugs.python.org/issue37021 closed by pablogsal #37023: test_gdb failed on AMD64 Debian PGO 3.x https://bugs.python.org/issue37023 closed by steve.dower #37027: Return a safe proxy over a socket from get_extra_info('socket' https://bugs.python.org/issue37027 closed by asvetlov #37028: Implement asyncio repl https://bugs.python.org/issue37028 closed by yselivanov #37032: Add CodeType.replace() method https://bugs.python.org/issue37032 closed by vstinner #37035: Don't log OSError exceptions in asyncio transports https://bugs.python.org/issue37035 closed by asvetlov #37045: Implement PEP 591: add Final qualifier and @final decorator to https://bugs.python.org/issue37045 closed by levkivskyi #37046: Implement PEP 586: add Literal type constructor to typing https://bugs.python.org/issue37046 closed by levkivskyi #37047: Refactor AsyncMock setup logic in create_autospec https://bugs.python.org/issue37047 closed by yselivanov #37049: Implement PEP 589: add TypedDict to typing https://bugs.python.org/issue37049 closed by levkivskyi #37050: Remove expr_text from ast node FormattedValue https://bugs.python.org/issue37050 closed by eric.smith #37051: Glossary item "hashable" incorrect https://bugs.python.org/issue37051 closed by rhettinger #37053: Tools/parser/unparse.py does not handle u"bar" correctly https://bugs.python.org/issue37053 closed by pablogsal #37054: Ignored exceptions in test_memoryio https://bugs.python.org/issue37054 closed by vstinner #37056: test_tools is failing on several buildbots https://bugs.python.org/issue37056 closed by levkivskyi #37057: suspicious.py sphinx extension access non-existing attribute. https://bugs.python.org/issue37057 closed by mbussonn #37058: Implement PEP 544: add Protocol to typing https://bugs.python.org/issue37058 closed by levkivskyi #37059: spam https://bugs.python.org/issue37059 closed by Mariatta #37061: The strangest glitch I have ever seen - incorrect indenterror, https://bugs.python.org/issue37061 closed by eric.smith #37065: File and lineno is not reported for syntax error in f-string https://bugs.python.org/issue37065 closed by SilentGhost #37066: os.execl opening a new bash shell doesn't work if initfile/rcf https://bugs.python.org/issue37066 closed by xtreak #37067: os.execl doesn't allow for empty string in mac https://bugs.python.org/issue37067 closed by ned.deily #37070: Clean up f-string debug handling, including potential memory l https://bugs.python.org/issue37070 closed by eric.smith #37072: PyNode_Compile() crashes in Python 3.8. https://bugs.python.org/issue37072 closed by gvanrossum #37076: _thread.start_new_thread(): call sys.unraisablehook() to handl https://bugs.python.org/issue37076 closed by vstinner #37078: spam https://bugs.python.org/issue37078 closed by xtreak #37079: PEM cadata causes ssl.SSLError: nested asn1 error https://bugs.python.org/issue37079 closed by christian.heimes #37092: LoggerAdapter doesn't have fatal() like Logger https://bugs.python.org/issue37092 closed by vinay.sajip #37094: Provide an example for TestCase.skipTest in unittest doc https://bugs.python.org/issue37094 closed by zach.ware #37099: test_inspect generates DeprecationWarning https://bugs.python.org/issue37099 closed by xtreak #37101: Filterer.filter can be rewritten using built-ins just as effic https://bugs.python.org/issue37101 closed by dwvisser #37104: logging.Logger.disabled is not documented https://bugs.python.org/issue37104 closed by vinay.sajip #37106: python re.escape doesn't escape some special characters. https://bugs.python.org/issue37106 closed by eric.smith #37108: Positional-only arguments break super() https://bugs.python.org/issue37108 closed by pablogsal #37109: Inacurrate documentation regarding return type of math.factori https://bugs.python.org/issue37109 closed by Antony.Lee #37111: Logging - Inconsistent behaviour when handling unicode https://bugs.python.org/issue37111 closed by vinay.sajip #37112: Error in compilig the AST for functions with optional position https://bugs.python.org/issue37112 closed by pablogsal #37113: 'ß'.upper() should return 'ẞ' https://bugs.python.org/issue37113 closed by SilentGhost #37114: lstrip remove extra characters in the presence of a matching n https://bugs.python.org/issue37114 closed by christian.heimes #37115: Support annotations in positional-only arguments https://bugs.python.org/issue37115 closed by pablogsal #1230540: Add threading.excepthook() to handle uncaught exceptions raise https://bugs.python.org/issue1230540 closed by vstinner #1195571: simple callback system for Py_FatalError https://bugs.python.org/issue1195571 closed by vstinner

1 0

Should I postpone PEP 558 (locals() semantics) to Python 3.9?
by Nick Coghlan 31 May '19

31 May '19

Hi folks, The reference implementation for PEP 558 (my attempt to fix the interaction between tracing functions and closure variables) is currently segfaulting somewhere deep in the garbage collector, and I've found that there's an issue with the PyEval_GetLocals() API returning a borrowed reference that means I need to tweak the proposed C API a bit such that PyEval_GetLocals() returns the proxy at function scope, and we add a new PyEval_GetPyLocals() that matches the locals() builtin. I don't *want* to postpone this to Python 3.9, but there turned out to be more remaining work than I thought there was to get this ready for inclusion in beta 1. I'll try to get the C API design details sorted today, but the segfault is mystifying me, and prevents the option of putting the core implementation in place for b1, and tidying up the documentation and comments for b2. Cheers, Nick.

4 4

PEP 590: vectorcall without tp_call
by Jeroen Demeyer 31 May '19

31 May '19

Hello, I have one implementation question about vectorcall which is not specified in PEP 590: what should happen if a type implements vectorcall (i.e. _Py_TPFLAGS_HAVE_VECTORCALL is set) but doesn't set tp_call (i.e. tp_call == NULL)? I see the following possibilities: 1. Ignore this problem/assume that it won't happen. This would be bad, since callable(obj) would be False even though obj() would succeed. 2. Raise SystemError. 3. Automatically set tp_call to PyVectorcall_Call. I would vote for 3 since it's the most user-friendly option. There is also no way how it could be wrong: it ensures that tp_call and vectorcall are consistent. Any opinions? Jeroen.

5 7

PEP 558: Defined semantics for locals()
by Nick Coghlan 30 May '19

30 May '19

Hi folks, After a couple of years hiatus, I finally got back to working on the reference implementation for PEP 558, my proposal to resolve some weird interactions between closures and trace functions in CPython by formally defining the expected semantics of the locals() builtin at function scope, and then ensuring that CPython adheres to those clarified semantics. The full text of the PEP is included below, and the rendered version is available at https://www.python.org/dev/peps/pep-0558/ The gist of the PEP is that: 1. The behaviour when no trace functions are installed and no frame introspection APIs are invoked is fine, so nothing should change in that regard (except that we elevate that behaviour from "the way CPython happens to work" to "the way the language and library reference says that Python implementations are supposed to work", and make sure CPython continues to behave that way even when a trace hook *is* installed) 2. If you get hold of a frame object in CPython (or another implementation that emulates the CPython frame API), whether via a trace hook or via a frame introspection API, then writing to the returned mapping will update the actual function locals and/or closure reference immediately, rather than relying on the FastToLocals/LocalsToFast APIs 3. The LocalsToFast C API changes to always produce RuntimeError (since there's no way for us to make it actually work correctly and consistently in the presence of closures, and the trace hook implementation won't need it any more given the write-through proxy behaviour on frame objects' "f_locals" attribute) The reference implementation still isn't quite done yet, but it's far enough along that I'm happy with the semantics and C API updates proposed in the current version of the PEP. Cheers, Nick. P.S. I'm away this weekend, so I expect the reference implementation to be done late next week, and I'd be submitting the PEP to Nathaniel for formal pronouncement at that point. However, I'm posting this thread now so that there's more time for discussion prior to the 3.8b1 deadline. ============== PEP: 558 Title: Defined semantics for locals() Author: Nick Coghlan <ncoghlan(a)gmail.com> BDFL-Delegate: Nathaniel J. Smith Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 2017-09-08 Python-Version: 3.8 Post-History: 2017-09-08, 2019-05-22 Abstract ======== The semantics of the ``locals()`` builtin have historically been underspecified and hence implementation dependent. This PEP proposes formally standardising on the behaviour of the CPython 3.6 reference implementation for most execution scopes, with some adjustments to the behaviour at function scope to make it more predictable and independent of the presence or absence of tracing functions. Rationale ========= While the precise semantics of the ``locals()`` builtin are nominally undefined, in practice, many Python programs depend on it behaving exactly as it behaves in CPython (at least when no tracing functions are installed). Other implementations such as PyPy are currently replicating that behaviour, up to and including replication of local variable mutation bugs that can arise when a trace hook is installed [1]_. While this PEP considers CPython's current behaviour when no trace hooks are installed to be acceptable (and largely desirable), it considers the current behaviour when trace hooks are installed to be problematic, as it causes bugs like [1]_ *without* even reliably enabling the desired functionality of allowing debuggers like ``pdb`` to mutate local variables [3]_. Proposal ======== The expected semantics of the ``locals()`` builtin change based on the current execution scope. For this purpose, the defined scopes of execution are: * module scope: top-level module code, as well as any other code executed using ``exec()`` or ``eval()`` with a single namespace * class scope: code in the body of a ``class`` statement, as well as any other code executed using ``exec()`` or ``eval()`` with separate local and global namespaces * function scope: code in the body of a ``def`` or ``async def`` statement We also allow interpreters to define two "modes" of execution, with only the first mode being considered part of the language specification itself: * regular operation: the way the interpreter behaves by default * tracing mode: the way the interpreter behaves when a trace hook has been registered in one or more threads via an implementation dependent mechanism like ``sys.settrace`` ([4]_) in CPython's ``sys`` module or ``PyEval_SetTrace`` ([5]_) in CPython's C API For regular operation, this PEP proposes elevating the current behaviour of the CPython reference implementation to become part of the language specification. For tracing mode, this PEP proposes changes to CPython's behaviour at function scope that bring the ``locals()`` builtin semantics closer to those used in regular operation, while also making the related frame API semantics clearer and easier for interactive debuggers to rely on. The proposed tracing mode changes also affect the semantics of frame object references obtained through other means, such as via a traceback, or via the ``sys._getframe()`` API. New ``locals()`` documentation ------------------------------ The heart of this proposal is to revise the documentation for the ``locals()`` builtin to read as follows: Return a dictionary representing the current local symbol table, with variable names as the keys, and their currently bound references as the values. This will always be the same dictionary for a given runtime execution frame. At module scope, as well as when using ``exec()`` or ``eval()`` with a single namespace, this function returns the same namespace as ``globals()``. At class scope, it returns the namespace that will be passed to the metaclass constructor. When using ``exec()`` or ``eval()`` with separate local and global namespaces, it returns the local namespace passed in to the function call. At function scope (including for generators and coroutines), it returns a dynamic snapshot of the function's local variables and any nonlocal cell references. In this case, changes made via the snapshot are *not* written back to the corresponding local variables or nonlocal cell references, and any such changes to the snapshot will be overwritten if the snapshot is subsequently refreshed (e.g. by another call to ``locals()``). CPython implementation detail: the dynamic snapshot for the current frame will be implicitly refreshed before each call to the trace function when a trace function is active. For reference, the current documentation of this builtin reads as follows: Update and return a dictionary representing the current local symbol table. Free variables are returned by locals() when it is called in function blocks, but not in class blocks. Note: The contents of this dictionary should not be modified; changes may not affect the values of local and free variables used by the interpreter. (In other words: the status quo is that the semantics and behaviour of ``locals()`` are currently formally implementation defined, whereas the proposed state after this PEP is that the only implementation defined behaviour will be that encountered at function scope when a tracing function is defined, with the behaviour in all other cases being defined by the language and library references) Module scope ------------ At module scope, as well as when using ``exec()`` or ``eval()`` with a single namespace, ``locals()`` must return the same object as ``globals()``, which must be the actual execution namespace (available as ``inspect.currentframe().f_locals`` in implementations that provide access to frame objects). Variable assignments during subsequent code execution in the same scope must dynamically change the contents of the returned mapping, and changes to the returned mapping must change the values bound to local variable names in the execution environment. The semantics at module scope are required to be the same in both tracing mode (if provided by the implementation) and in regular operation. To capture this expectation as part of the language specification, the following paragraph will be added to the documentation for ``locals()``: At module scope, as well as when using ``exec()`` or ``eval()`` with a single namespace, this function returns the same namespace as ``globals()``. This part of the proposal does not require any changes to the reference implementation - it is standardisation of the current behaviour. Class scope ----------- At class scope, as well as when using ``exec()`` or ``eval()`` with separate global and local namespaces, ``locals()`` must return the specified local namespace (which may be supplied by the metaclass ``__prepare__`` method in the case of classes). As for module scope, this must be a direct reference to the actual execution namespace (available as ``inspect.currentframe().f_locals`` in implementations that provide access to frame objects). Variable assignments during subsequent code execution in the same scope must change the contents of the returned mapping, and changes to the returned mapping must change the values bound to local variable names in the execution environment. The mapping returned by ``locals()`` will *not* be used as the actual class namespace underlying the defined class (the class creation process will copy the contents to a fresh dictionary that is only accessible by going through the class machinery). For nested classes defined inside a function, any nonlocal cells referenced from the class scope are *not* included in the ``locals()`` mapping. The semantics at class scope are required to be the same in both tracing mode (if provided by the implementation) and in regular operation. To capture this expectation as part of the language specification, the following two paragraphs will be added to the documentation for ``locals()``: When using ``exec()`` or ``eval()`` with separate local and global namespaces, [this function] returns the given local namespace. At class scope, it returns the namespace that will be passed to the metaclass constructor. This part of the proposal does not require any changes to the reference implementation - it is standardisation of the current behaviour. Function scope -------------- At function scope, interpreter implementations are granted significant freedom to optimise local variable access, and hence are NOT required to permit arbitrary modification of local and nonlocal variable bindings through the mapping returned from ``locals()``. Historically, this leniency has been described in the language specification with the words "The contents of this dictionary should not be modified; changes may not affect the values of local and free variables used by the interpreter." This PEP proposes to change that text to instead say: At function scope (including for generators and coroutines), [this function] returns a dynamic snapshot of the function's local variables and any nonlocal cell references. In this case, changes made via the snapshot are *not* written back to the corresponding local variables or nonlocal cell references, and any such changes to the snapshot will be overwritten if the snapshot is subsequently refreshed (e.g. by another call to ``locals()``). CPython implementation detail: the dynamic snapshot for the currently executing frame will be implicitly refreshed before each call to the trace function when a trace function is active. This part of the proposal *does* require changes to the CPython reference implementation, as while it accurately describes the behaviour in regular operation, the "write back" strategy currently used to support namespace changes from trace functions doesn't comply with it (and also causes the quirky behavioural problems mentioned in the Rationale). CPython Implementation Changes ============================== The current cause of CPython's tracing mode quirks (both the side effects from simply installing a tracing function and the fact that writing values back to function locals only works for the specific function being traced) is the way that locals mutation support for trace hooks is currently implemented: the ``PyFrame_LocalsToFast`` function. When a trace function is installed, CPython currently does the following for function frames (those where the code object uses "fast locals" semantics): 1. Calls ``PyFrame_FastToLocals`` to update the dynamic snapshot 2. Calls the trace hook (with tracing of the hook itself disabled) 3. Calls ``PyFrame_LocalsToFast`` to capture any changes made to the dynamic snapshot This approach is problematic for a few different reasons: * Even if the trace function doesn't mutate the snapshot, the final step resets any cell references back to the state they were in before the trace function was called (this is the root cause of the bug report in [1]_) * If the trace function *does* mutate the snapshot, but then does something that causes the snapshot to be refreshed, those changes are lost (this is one aspect of the bug report in [3]_) * If the trace function attempts to mutate the local variables of a frame other than the one being traced (e.g. ``frame.f_back.f_locals``), those changes will almost certainly be lost (this is another aspect of the bug report in [3]_) * If a ``locals()`` reference is passed to another function, and *that* function mutates the snapshot namespace, then those changes *may* be written back to the execution frame *if* a trace hook is installed The proposed resolution to this problem is to take advantage of the fact that whereas functions typically access their *own* namespace using the language defined ``locals()`` builtin, trace functions necessarily use the implementation dependent ``frame.f_locals`` interface, as a frame reference is what gets passed to hook implementations. Instead of being a direct reference to the dynamic snapshot returned by ``locals()``, ``frame.f_locals`` will be updated to instead return a dedicated proxy type (implemented as a private subclass of the existing ``types.MappingProxyType``) that has two internal attributes not exposed as part of either the Python or public C API: * *mapping*: the dynamic snapshot that is returned by the ``locals()`` builtin * *frame*: the underlying frame that the snapshot is for ``__setitem__`` and ``__delitem__`` operations on the proxy will affect not only the dynamic snapshot, but *also* the corresponding fast local or cell reference on the underlying frame. The ``locals()`` builtin will be made aware of this proxy type, and continue to return a reference to the dynamic snapshot rather than to the write-through proxy. At the C API layer, ``PyEval_GetLocals()`` will implement the same semantics as the Python level ``locals()`` builtin, and a new ``PyFrame_GetPyLocals(frame)`` accessor API will be provided to allow the function level proxy bypass logic to be encapsulated entirely inside the frame implementation. The C level equivalent of accessing ``pyframe.f_locals`` in Python will be a new ``PyFrame_GetLocalsAttr(frame)`` API. Like the Python level descriptor, the new API will implicitly refresh the dynamic snapshot at function scope before returning a reference to the write-through proxy. The ``PyFrame_LocalsToFast()`` function will be changed to always emit ``RuntimeError``, explaining that it is no longer a supported operation, and affected code should be updated to rely on the write-through tracing mode proxy instead. Design Discussion ================= Ensuring ``locals()`` returns a shared snapshot at function scope ----------------------------------------------------------------- The ``locals()`` builtin is a required part of the language, and in the reference implementation it has historically returned a mutable mapping with the following characteristics: * each call to ``locals()`` returns the *same* mapping * for namespaces where ``locals()`` returns a reference to something other than the actual local execution namespace, each call to ``locals()`` updates the mapping with the current state of the local variables and any referenced nonlocal cells * changes to the returned mapping *usually* aren't written back to the local variable bindings or the nonlocal cell references, but write backs can be triggered by doing one of the following: * installing a Python level trace hook (write backs then happen whenever the trace hook is called) * running a function level wildcard import (requires bytecode injection in Py3) * running an ``exec`` statement in the function's scope (Py2 only, since ``exec`` became an ordinary builtin in Python 3) The proposal in this PEP aims to retain the first two properties (to maintain backwards compatibility with as much code as possible) while ensuring that simply installing a trace hook can't enable rebinding of function locals via the ``locals()`` builtin (whereas enabling rebinding via ``frame.f_locals`` inside the tracehook implementation is fully intended). Keeping ``locals()`` as a dynamic snapshot at function scope ------------------------------------------------------------ It would theoretically be possible to change the semantics of the ``locals()`` builtin to return the write-through proxy at function scope, rather than continuing to return a dynamic snapshot. This PEP doesn't (and won't) propose this as it's a backwards incompatible change in practice, even though code that relies on the current behaviour is technically operating in an undefined area of the language specification. Consider the following code snippet:: def example(): x = 1 locals()["x"] = 2 print(x) Even with a trace hook installed, that function will consistently print ``1`` on the current reference interpreter implementation:: >>> example() 1 >>> import sys >>> def basic_hook(*args): ... return basic_hook ... >>> sys.settrace(basic_hook) >>> example() 1 Similarly, ``locals()`` can be passed to the ``exec()`` and ``eval()`` builtins at function scope without risking unexpected rebinding of local variables. Provoking the reference interpreter into incorrectly mutating the local variable state requires a more complex setup where a nested function closes over a variable being rebound in the outer function, and due to the use of either threads, generators, or coroutines, it's possible for a trace function to start running for the nested function before the rebinding operation in the outer function, but finish running after the rebinding operation has taken place (in which case the rebinding will be reverted, which is the bug reported in [1]_). In addition to preserving the de facto semantics which have been in place since PEP 227 introduced nested scopes in Python 2.1, the other benefit of restricting the write-through proxy support to the implementation-defined frame object API is that it means that only interpreter implementations which emulate the full frame API need to offer the write-through capability at all, and that JIT-compiled implementations only need to enable it when a frame introspection API is invoked, or a trace hook is installed, not whenever ``locals()`` is accessed at function scope. What happens with the default args for ``eval()`` and ``exec()``? ----------------------------------------------------------------- These are formally defined as inheriting ``globals()`` and ``locals()`` from the calling scope by default. There isn't any need for the PEP to change these defaults, so it doesn't. Changing the frame API semantics in regular operation ----------------------------------------------------- Earlier versions of this PEP proposed having the semantics of the frame ``f_locals`` attribute depend on whether or not a tracing hook was currently installed - only providing the write-through proxy behaviour when a tracing hook was active, and otherwise behaving the same as the ``locals()`` builtin. That was adopted as the original design proposal for a couple of key reasons, one pragmatic and one more philosophical: * Object allocations and method wrappers aren't free, and tracing functions aren't the only operations that access frame locals from outside the function. Restricting the changes to tracing mode meant that the additional memory and execution time overhead of these changes would as close to zero in regular operation as we can possibly make them. * "Don't change what isn't broken": the current tracing mode problems are caused by a requirement that's specific to tracing mode (support for external rebinding of function local variable references), so it made sense to also restrict any related fixes to tracing mode However, actually attempting to implement and document that dynamic approach highlighted the fact that it makes for a really subtle runtime state dependent behaviour distinction in how ``frame.f_locals`` works, and creates several new edge cases around how ``f_locals`` behaves as trace functions are added and removed. Accordingly, the design was switched to the current one, where ``frame.f_locals`` is always a write-through proxy, and ``locals()`` is always a dynamic snapshot, which is both simpler to implement and easier to explain. Regardless of how the CPython reference implementation chooses to handle this, optimising compilers and interpreters also remain free to impose additional restrictions on debuggers, by making local variable mutation through frame objects an opt-in behaviour that may disable some optimisations (just as the emulation of CPython's frame API is already an opt-in flag in some Python implementations). Historical semantics at function scope -------------------------------------- The current semantics of mutating ``locals()`` and ``frame.f_locals`` in CPython are rather quirky due to historical implementation details: * actual execution uses the fast locals array for local variable bindings and cell references for nonlocal variables * there's a ``PyFrame_FastToLocals`` operation that populates the frame's ``f_locals`` attribute based on the current state of the fast locals array and any referenced cells. This exists for three reasons: * allowing trace functions to read the state of local variables * allowing traceback processors to read the state of local variables * allowing ``locals()`` to read the state of local variables * a direct reference to ``frame.f_locals`` is returned from ``locals()``, so if you hand out multiple concurrent references, then all those references will be to the exact same dictionary * the two common calls to the reverse operation, ``PyFrame_LocalsToFast``, were removed in the migration to Python 3: ``exec`` is no longer a statement (and hence can no longer affect function local namespaces), and the compiler now disallows the use of ``from module import *`` operations at function scope * however, two obscure calling paths remain: ``PyFrame_LocalsToFast`` is called as part of returning from a trace function (which allows debuggers to make changes to the local variable state), and you can also still inject the ``IMPORT_STAR`` opcode when creating a function directly from a code object rather than via the compiler This proposal deliberately *doesn't* formalise these semantics as is, since they only make sense in terms of the historical evolution of the language and the reference implementation, rather than being deliberately designed. Implementation ============== The reference implementation update is in development as a draft pull request on GitHub ([6]_). Acknowledgements ================ Thanks to Nathaniel J. Smith for proposing the write-through proxy idea in [1]_ and pointing out some critical design flaws in earlier iterations of the PEP that attempted to avoid introducing such a proxy. References ========== .. [1] Broken local variable assignment given threads + trace hook + closure (https://bugs.python.org/issue30744) .. [2] Clarify the required behaviour of ``locals()`` (https://bugs.python.org/issue17960) .. [3] Updating function local variables from pdb is unreliable (https://bugs.python.org/issue9633) .. [4] CPython's Python API for installing trace hooks (https://docs.python.org/dev/library/sys.html#sys.settrace) .. [5] CPython's C API for installing trace hooks (https://docs.python.org/3/c-api/init.html#c.PyEval_SetTrace) .. [6] PEP 558 reference implementation (https://github.com/python/cpython/pull/3640/files) Copyright ========= This document has been placed in the public domain. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

13 30

Per opcode cache for LOAD_GLOBAL
by Inada Naoki 29 May '19

29 May '19

Hi, all. Yury implemented per opcode cache for LOAD_GLOBAL, LOAD_ATTR, and LOAD_METHOD. [1] I update the patch for current master branch, but only for LOAD_GLOBAL for now. [2] It sped up LOAD_GLOBAL about 40%. [3] It is attractive optimization. Now 3.8b1 will be coming soon, but the pull request is not reviewed well yet. For example, should we add some switch to disable the cache? May I merge it before beta1 and polish (or revert) it by rc1? Or should I postpone it for 3.9? Regards, [1]: https://bugs.python.org/issue26219 [2]: https://github.com/python/cpython/pull/12884 [3]: https://github.com/methane/sandbox/tree/master/2019/opcache_load_global#opc… -- Inada Naoki <songofacandy(a)gmail.com>

2 1

Re: [Python-Dev] Overloading comparison operator for lists
by Montana Burr 29 May '19

29 May '19

Ok, now I'm mildly curious to knpw: What is the justification for causing list == 3 to evaluate to False, besides the obvious "a list cannot equal a number"? On Tue, May 28, 2019 at 9:34 PM Montana Burr <montana.burr(a)gmail.com> wrote: > Ok, now I'm mildly curious to knpw: > > What is the justification for causing list == 3 to evaluate to False, > besides the obvious "a list cannot equal a number"? > > On Tue, May 28, 2019 at 7:52 PM Rob Cliffe <rob.cliffe(a)btinternet.com> > wrote: > >> >> >> On 27/05/2019 04:52:17, Montana Burr wrote: >> > NumPy arrays have this awesome feature, where array == 3 does an >> > element-wise comparison and returns a list. For example: >> > >> > np.array([1,2,3,4,5])==3 >> > >> > returns >> > >> > [False,False,True,False,False] >> > >> > It would be cool if Python had similar functionality for lists. >> > >> Well, it does have: >> >>> [x==3 for x in [1,2,3,4,5]] >> [False, False, True, False, False] >> >> This is IMHO much more intuitive than your construct overloading "==". >> It is also more flexible (any operation can be performed on x, not just >> an equality comparison). >> So sorry, but I can see no justification for changing the Python >> language to do something, which can already be done, to be done in a >> more obscure way. >> Also >> [1,2,3,4,5]==3 >> is already legal syntax (it evaluates to False, since the operands are >> not equal), so you are proposing a code-breaking change. >> >> Apologies if someone has already pointed all this out, as is very likely >> (I have only just (re-)joined python-ideas and may have missed the >> relevant posts). >> Best wishes >> Rob Cliffe >> > > > -- > > WebRep > Overall rating > > -- WebRep Overall rating

3 2