Python-Dev
Threads by month
- ----- 2025 -----
- February
- January
- ----- 2024 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2023 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2022 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2021 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2020 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2019 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2018 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2017 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2016 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2015 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2014 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2013 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2012 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2011 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2010 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2009 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2008 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2007 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2006 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2005 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2004 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2003 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2002 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2001 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2000 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 1999 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
May 2019
- 105 participants
- 74 discussions
Hi,
I have updated the PEP with feedback from discussions. The highlights are:
* Deprecate parser module
* Keep fileinput module
* Elaborate why crypt and spwd are dangerous and bad
* Improve sections for cgitb, colorsys, nntplib, and smtpd modules
* The colorsys, crypt, imghdr, sndhdr, and spwd sections now list suitable substitutions.
* Mention that socketserver is going to stay for http.server and xmlrpc.server
* The future maintenance section now states that the deprecated modules may be …
[View More]adopted by Python community members.
https://github.com/python/peps/compare/7799178a...2d536899?diff=unified#dif…
I'll be traveling the next couple of days and will only have limited opportunities to respond on feedback.
Christian
-------------------------------------------------------------------
PEP: 594
Title: Removing dead batteries from the standard library
Author: Christian Heimes <christian(a)python.org>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 20-May-2019
Post-History: 21-May-2019
Abstract
========
This PEP proposed a list of standard library modules to be removed from the
standard library. The modules are mostly historic data formats and APIs that
have been superseded a long time ago, e.g. Mac OS 9 and Commodore.
Rationale
=========
Back in the early days of Python, the interpreter came with a large set of
useful modules. This was often refrained to as "batteries included"
philosophy and was one of the corner stones to Python's success story.
Users didn't have to figure out how to download and install separate
packages in order to write a simple web server or parse email.
Times have changed. The introduction of the cheese shop (PyPI), setuptools,
and later pip, it became simple and straight forward to download and install
packages. Nowadays Python has a rich and vibrant ecosystem of third party
packages. It's pretty much standard to either install packages from PyPI or
use one of the many Python or Linux distributions.
On the other hand, Python's standard library is piling up cruft, unnecessary
duplication of functionality, and dispensable features. This is undesirable
for several reasons.
* Any additional module increases the maintenance cost for the Python core
development team. The team has limited resources, reduced maintenance cost
frees development time for other improvements.
* Modules in the standard library are generally favored and seen as the
de-facto solution for a problem. A majority of users only pick 3rd party
modules to replace a stdlib module, when they have a compelling reason, e.g.
lxml instead of `xml`. The removal of an unmaintained stdlib module
increases the chances of a community contributed module to become widely
used.
* A lean and mean standard library benefits platforms with limited resources
like devices with just a few hundred kilobyte of storage (e.g. BBC
Micro:bit). Python on mobile platforms like BeeWare or WebAssembly
(e.g. pyodide) also benefit from reduced download size.
The modules in the PEP have been selected for deprecation because their
removal is either least controversial or most beneficial. For example
least controversial are 30 years old multimedia formats like ``sunau``
audio format, which was used on SPARC and NeXT workstations in the late
1980ties. The ``crypt`` module has fundamental flaws that are better solved
outside the standard library.
This PEP also designates some modules as not scheduled for removal. Some
modules have been deprecated for several releases or seem unnecessary at
first glance. However it is beneficial to keep the modules in the standard
library, mostly for environments where installing a package from PyPI is not
an option. This can be cooperate environments or class rooms where external
code is not permitted without legal approval.
* The usage of FTP is declining, but some files are still provided over
the FTP protocol or hosters offer FTP to upload content. Therefore
``ftplib`` is going to stay.
* The ``optparse`` and ``getopt`` module are widely used. They are mature
modules with very low maintenance overhead.
* According to David Beazley [5]_ the ``wave`` module is easy to teach to
kids and can make crazy sounds. Making a computer generate crazy sounds is
powerful and highly motivating exercise for a 9yo aspiring developer. It's
a fun battery to keep.
Deprecation schedule
====================
3.8
---
This PEP targets Python 3.8. Version 3.8.0 final is scheduled to be released
a few months before Python 2.7 will reach its end of lifetime. We expect that
Python 3.8 will be targeted by users that migrate to Python 3 in 2019 and
2020. To reduce churn and to allow a smooth transition from Python 2,
Python 3.8 will neither raise `DeprecationWarning` nor remove any
modules that have been scheduled for removal. Instead deprecated modules will
just be *documented* as deprecated. Optionally modules may emit a
`PendingDeprecationWarning`.
All deprecated modules will also undergo a feature freeze. No additional
features should be added. Bug should still be fixed.
3.9
---
Starting with Python 3.9, deprecated modules will start issuing
`DeprecationWarning`. The `parser`_ module is removed and potentially
replaced with a new module.
All other deprecated modules are fully supported and will receive security
updates until Python 3.9 reaches its end of lifetime. Python 3.9.0 will
be released about 18 months after 3.8.0 (April 2021?) and most likely
be supported for 5 years after the release. The estimated EOL of Python 3.9
is in 2026.
3.10
----
In 3.10 all deprecated modules will be removed from the CPython repository
together with tests, documentation, and autoconf rules.
PEP acceptance process
======================
3.8.0b1 is scheduled to be release shortly after the PEP is officially
submitted. Since it's improbable that the PEP will pass all stages of the
PEP process in time, I propose a two step acceptance process that is
analogous Python's two release deprecation process.
The first *provisionally accepted* phase targets Python 3.8.0b1. In the first
phase no code is changes or removed. Modules are only documented as
deprecated. The only exception is the `parser`_ module. It has been
documented as deprecated since Python 2.5 and is scheduled for removal for
3.9 to make place for a more advanced parser.
The final decision, which modules will be removed and how the removed code
is preserved, can be delayed for another year.
Deprecated modules
==================
The modules are grouped as data encoding, multimedia, network, OS interface,
and misc modules. The majority of modules are for old data formats or
old APIs. Some others are rarely useful and have better replacements on
PyPI, e.g. Pillow for image processing or NumPy-based projects to deal with
audio processing.
.. csv-table:: Table 1: Proposed modules deprecations
:header: "Module", "Deprecated in", "To be removed", "Replacement"
:widths: 1, 1, 1, 2
aifc,3.8,3.10,\-
asynchat,3.8,3.10,asyncio
asyncore,3.8,3.10,asyncio
audioop,3.8,3.10,\-
binhex,3.8,3.10,\-
cgi,3.8,3.10,\-
cgitb,3.8,3.10,\-
chunk,3.8,3.10,\-
colorsys,3.8,3.10,"colormath, colour, colorspacious, Pillow"
crypt,3.8,3.10,"bcrypt, argon2cffi, hashlib, passlib"
fileinput,\-,**keep**,argparse
formatter,3.4,3.10,\-
fpectl,**3.7**,**3.7**,\-
getopt,**3.2**,**keep**,"argparse, optparse"
imghdr,3.8,3.10,"filetype, puremagic, python-magic"
imp,**3.4**,3.10,importlib
lib2to3,\-,**keep**,
macpath,**3.7**,**3.8**,\-
msilib,3.8,3.10,\-
nntplib,3.8,3.10,\-
nis,3.8,3.10,\-
optparse,\-,**keep**,argparse
ossaudiodev,3.8,3.10,\-
parser,**2.5**,**3.9**,"ast, lib2to3.pgen2"
pipes,3.8,3.10,subprocess
smtpd,"**3.4.7**, **3.5.4**",3.10,aiosmtpd
sndhdr,3.8,3.10,"filetype, puremagic, python-magic"
spwd,3.8,3.10,"python-pam, simplepam"
sunau,3.8,3.10,\-
uu,3.8,3.10,\-
wave,\-,**keep**,
xdrlib,3.8,3.10,\-
Data encoding modules
---------------------
binhex
~~~~~~
The `binhex <https://docs.python.org/3/library/binhex.html>`_ module encodes
and decodes Apple Macintosh binhex4 data. It was originally developed for
TSR-80. In the 1980s and early 1990s it was used on classic Mac OS 9 to
encode binary email attachments.
Module type
pure Python
Deprecated in
3.8
To be removed in
3.10
Substitute
**none**
uu
~~
The `uu <https://docs.python.org/3/library/uu.html>`_ module provides
uuencode format, an old binary encoding format for email from 1980. The uu
format has been replaced by MIME. The uu codec is provided by the binascii
module.
Module type
pure Python
Deprecated in
3.8
To be removed in
3.10
Substitute
**none**
xdrlib
~~~~~~
The `xdrlib <https://docs.python.org/3/library/xdrlib.html>`_ module supports
the Sun External Data Representation Standard. XDR is an old binary
serialization format from 1987. These days it's rarely used outside
specialized domains like NFS.
Module type
pure Python
Deprecated in
3.8
To be removed in
3.10
Substitute
**none**
Multimedia modules
------------------
aifc
~~~~
The `aifc <https://docs.python.org/3/library/aifc.html>`_ module provides
support for reading and writing AIFF and AIFF-C files. The Audio Interchange
File Format is an old audio format from 1988 based on Amiga IFF. It was most
commonly used on the Apple Macintosh. These days only few specialized
application use AIFF.
Module type
pure Python (depends on `audioop`_ C extension)
Deprecated in
3.8
To be removed in
3.10
Substitute
**none**
audioop
~~~~~~~
The `audioop <https://docs.python.org/3/library/audioop.html>`_ module
contains helper functions to manipulate raw audio data and adaptive
differential pulse-code modulated audio data. The module is implemented in
C without any additional dependencies. The `aifc`_, `sunau`_, and `wave`_
module depend on `audioop`_ for some operations. The byteswap operation in
the `wave`_ module can be substituted with little work.
Module type
C extension
Deprecated in
3.8
To be removed in
3.10
Substitute
**none**
colorsys
~~~~~~~~
The `colorsys <https://docs.python.org/3/library/colorsys.html>`_ module
defines color conversion functions between RGB, YIQ, HSL, and HSV coordinate
systems.
The PyPI packages *colormath*, *colour*, and *colorspacious* provide more and
advanced features. The Pillow library is better suited to transform images
between color systems.
Module type
pure Python
Deprecated in
3.8
To be removed in
3.10
Substitute
`colormath <https://pypi.org/project/colormath/>`_,
`colour <https://pypi.org/project/colour/>`_
`colorspacious <https://pypi.org/project/colorspacious/>`_,
`Pillow <https://pypi.org/project/Pillow/>`_
chunk
~~~~~
The `chunk <https://docs.python.org/3/library/chunk.html>`_ module provides
support for reading and writing Electronic Arts' Interchange File Format.
IFF is an old audio file format originally introduced for Commodore and
Amiga. The format is no longer relevant.
Module type
pure Python
Deprecated in
3.8
To be removed in
3.10
Substitute
**none**
imghdr
~~~~~~
The `imghdr <https://docs.python.org/3/library/imghdr.html>`_ module is a
simple tool to guess the image file format from the first 32 bytes
of a file or buffer. It supports only a limited amount of formats and
neither returns resolution nor color depth.
Module type
pure Python
Deprecated in
3.8
To be removed in
3.10
Substitute
`puremagic <https://pypi.org/project/puremagic/>`_,
`filetype <https://pypi.org/project/filetype/>`_,
`python-magic <https://pypi.org/project/python-magic/>`_
ossaudiodev
~~~~~~~~~~~
The `ossaudiodev <https://docs.python.org/3/library/ossaudiodev.html>`_
module provides support for Open Sound System, an interface to sound
playback and capture devices. OSS was initially free software, but later
support for newer sound devices and improvements were proprietary. Linux
community abandoned OSS in favor of ALSA [1]_. Some operation systems like
OpenBSD and NetBSD provide an incomplete [2]_ emulation of OSS.
Module type
C extension
Deprecated in
3.8
To be removed in
3.10
Substitute
**none**
sndhdr
~~~~~~
The `sndhdr <https://docs.python.org/3/library/sndhdr.html>`_ module is
similar to the `imghdr`_ module but for audio formats. It guesses file
format, channels, frame rate, and sample widths from the first 512 bytes of
a file or buffer. The module only supports AU, AIFF, HCOM, VOC, WAV, and
other ancient formats.
Module type
pure Python (depends on `audioop`_ C extension for some operations)
Deprecated in
3.8
To be removed in
3.10
Substitute
`puremagic <https://pypi.org/project/puremagic/>`_,
`filetype <https://pypi.org/project/filetype/>`_,
`python-magic <https://pypi.org/project/python-magic/>`_
sunau
~~~~~
The `sunau <https://docs.python.org/3/library/sunhdr.html>`_ module provides
support for Sun AU sound format. It's yet another old, obsolete file format.
Module type
pure Python (depends on `audioop`_ C extension for some operations)
Deprecated in
3.8
To be removed in
3.10
Substitute
**none**
Networking modules
------------------
asynchat
~~~~~~~~
The `asynchat <https://docs.python.org/3/library/asynchat.html>`_ module
is build on top of `asyncore`_ and has been deprecated since Python 3.6.
Module type
pure Python
Deprecated in
3.6
Removed in
3.10
Substitute
asyncio
asyncore
~~~~~~~~
The `asyncore <https://docs.python.org/3/library/asyncore.html>`_ module was
the first module for asynchronous socket service clients and servers. It
has been replaced by asyncio and is deprecated since Python 3.6.
The ``asyncore`` module is also used in stdlib tests. The tests for
``ftplib``, ``logging``, ``smptd``, ``smtplib``, and ``ssl`` are partly
based on ``asyncore``. These tests must be updated to use asyncio or
threading.
Module type
pure Python
Deprecated in
3.6
Removed in
3.10
Substitute
asyncio
cgi
~~~
The `cgi <https://docs.python.org/3/library/cgi.html>`_ module is a support
module for Common Gateway Interface (CGI) scripts. CGI is deemed as
inefficient because every incoming request is handled in a new process. PEP
206 considers the module as *designed poorly and are now near-impossible
to fix*.
Several people proposed to either keep the cgi module for features like
`cgi.parse_qs()` or move `cgi.escape()` to a different module. The
functions `cgi.parse_qs` and `cgi.parse_qsl` have been
deprecated for a while and are actually aliases for
`urllib.parse.parse_qs` and `urllib.parse.parse_qsl`. The
function `cgi.quote` has been deprecated in favor of `html.quote`
with secure default values.
Module type
pure Python
Deprecated in
3.8
To be removed in
3.10
Substitute
**none**
cgitb
~~~~~
The `cgitb <https://docs.python.org/3/library/cgitb.html>`_ module is a
helper for the cgi module for configurable tracebacks.
The ``cgitb`` module is not used by any major Python web framework (Django,
Pyramid, Plone, Flask, CherryPy, or Bottle). Only Paste uses it in an
optional debugging middleware.
Module type
pure Python
Deprecated in
3.8
To be removed in
3.10
Substitute
**none**
smtpd
~~~~~
The `smtpd <https://docs.python.org/3/library/smtpd.html>`_ module provides
a simple implementation of a SMTP mail server. The module documentation
marks the module as deprecated and recommends ``aiosmtpd`` instead. The
deprecation message was added in releases 3.4.7, 3.5.4, and 3.6.1.
Module type
pure Python
Deprecated in
**3.7**
To be removed in
3.10
Substitute
aiosmtpd
nntplib
~~~~~~~
The `nntplib <https://docs.python.org/3/library/nntplib.html>`_ module
implements the client side of the Network News Transfer Protocol (nntp). News
groups used to be a dominant platform for online discussions. Over the last
two decades, news has been slowly but steadily replaced with mailing lists
and web-based discussion platforms. Twisted is also
`planning <https://twistedmatrix.com/trac/ticket/9405>`_ to deprecate NNTP
support and `pynnt <https://github.com/greenbender/pynntp>`_ hasn't seen any
activity since 2014. This is a good indicator that the public interest in
NNTP support is declining.
The ``nntplib`` tests have been the cause of additional work in the recent
past. Python only contains client side of NNTP. The tests connect to
external news server. The servers are sometimes unavailble, too slow, or do
not work correctly over IPv6. The situation causes flaky test runs on
buildbots.
Module type
pure Python
Deprecated in
3.8
To be removed in
3.10
Substitute
**none**
Operating system interface
--------------------------
crypt
~~~~~
The `crypt <https://docs.python.org/3/library/crypt.html>`_ module implements
password hashing based on ``crypt(3)`` function from ``libcrypt`` or
``libxcrypt`` on Unix-like platform. The algorithms are mostly old, of poor
quality and insecure. Users are discouraged to use them.
* The module is not available on Windows. Cross-platform application need
an alternative implementation any way.
* Only DES encryption is guarenteed to be available. DES has an extremely
limited key space of 2**56.
* MD5, salted SHA256, salted SHA512, and Blowfish are optional extension.
SSHA256 and SSHA512 are glibc extensions. Blowfish (bcrypt) is the only
algorithm that is still secure. However it's in glibc and therefore not
commonly available on Linux.
* Depending on the platform, the ``crypt`` module is not thread safe. Only
implementations with ``crypt_r(3)`` are thread safe.
* The module was never useful to interact with system user and password
databases. On BSD, macOS, and Linux, all user authentication and
password modification operations must go through PAM (pluggable
authentication module), see `spwd`_ deprecation.
Module type
C extension + Python module
Deprecated in
3.8
To be removed in
3.10
Substitute
`bcrypt <https://pypi.org/project/bcrypt/>`_,
`passlib <https://pypi.org/project/passlib/>`_,
`argon2cffi <https://pypi.org/project/argon2-cffi/>`_,
hashlib module (PBKDF2, scrypt)
macpath
~~~~~~~
The `macpath <https://docs.python.org/3/library/macpath.html>`_ module
provides Mac OS 9 implementation of os.path routines. Mac OS 9 is no longer
supported
Module type
pure Python
Deprecated in
3.7
Removed in
3.8
Substitute
**none**
nis
~~~
The `nis <https://docs.python.org/3/library/nis.html>`_ module provides
NIS/YP support. Network Information Service / Yellow Pages is an old and
deprecated directory service protocol developed by Sun Microsystems. It's
designed successor NIS+ from 1992 never took off. For a long time, libc's
Name Service Switch, LDAP, and Kerberos/GSSAPI are considered a more powerful
and more secure replacement of NIS.
Module type
C extension
Deprecated in
3.8
To be removed in
3.10
Substitute
**none**
spwd
~~~~
The `spwd <https://docs.python.org/3/library/spwd.html>`_ module provides
direct access to Unix shadow password database using non-standard APIs.
In general it's a bad idea to use the spwd. The spwd circumvents system
security policies, it does not use the PAM stack, and is only compatible
with local user accounts, because it ignores NSS. The use of the ``spwd``
module for access control must be consider a *security bug*, as it bypasses
PAM's access control.
Further more the ``spwd`` module uses the
`shadow(3) <http://man7.org/linux/man-pages/man3/shadow.3.html>`_ APIs.
Functions like ``getspnam(3)`` access the ``/etc/shadow`` file directly. This
is dangerous and even forbidden for confined services on systems with a
security engine like SELinux or AppArmor.
Module type
C extension
Deprecated in
3.8
To be removed in
3.10
Substitute
`python-pam <https://pypi.org/project/python-pam/>`_,
`simpleplam <https://pypi.org/project/simplepam/>`_
Misc modules
------------
formatter
~~~~~~~~~
The `formatter <https://docs.python.org/3/library/formatter.html>`_ module
is an old text formatting module which has been deprecated since Python 3.4.
Module type
pure Python
Deprecated in
3.4
To be removed in
3.10
Substitute
*n/a*
imp
~~~
The `imp <https://docs.python.org/3/library/imp.html>`_ module is the
predecessor of the
`importlib <https://docs.python.org/3/library/importlib.html>`_ module. Most
functions have been deprecated since Python 3.3 and the module since
Python 3.4.
Module type
C extension
Deprecated in
3.4
To be removed in
3.10
Substitute
importlib
msilib
~~~~~~
The `msilib <https://docs.python.org/3/library/msilib.html>`_ package is a
Windows-only package. It supports the creation of Microsoft Installers (MSI).
The package also exposes additional APIs to create cabinet files (CAB). The
module is used to facilitate distutils to create MSI installers with
``bdist_msi`` command. In the past it was used to create CPython's official
Windows installer, too.
Microsoft is slowly moving away from MSI in favor of Windows 10 Apps (AppX)
as new deployment model [3]_.
Module type
C extension + Python code
Deprecated in
3.8
To be removed in
3.10
Substitute
**none**
parser
~~~~~~
The `parser <https://docs.python.org/3/library/parser.html>`_ module provides
an interface to Python’s internal parser and byte-code compiler. The stdlib
has superior ways to interact with the parse tree. From Python 2.5 onward,
it's much more convenient to cut in at the Abstract Syntax Tree (AST)
generation and compilation stage.
The ``parser`` module causes additional work. It's C code that must be
kept in sync with any change to Python's grammar and internal parser.
Pablo wants to remove the parser module and promote lib2to3's pgen2 instead
[6]_.
Most importantly the presence of the ``parser`` module makes it harder to
switch to something more powerful than a LL(1) parser [7]_. Since the
``parser`` module is documented as deprecated since Python 2.5 and a new
parsing technology is planned for 3.9, the ``parser`` module is scheduled for
removal in 3.9.
Module type
C extension
Deprecated in
3.8, documented as deprecated since **2.5**
To be removed in
**3.9**
Substitute
ast, lib2to3.pgen2
pipes
~~~~~
The `pipes <https://docs.python.org/3/library/pipes.html>`_ module provides
helpers to pipe the input of one command into the output of another command.
The module is built on top of ``os.popen``. Users are encouraged to use
the subprocess module instead.
Module type
pure Python
Deprecated in
3.8
To be removed in
3.10
Substitute
subprocess module
Removed modules
===============
fpectl
------
The `fpectl <https://docs.python.org/3.6/library/fpectl.html>`_ module was
never built by default, its usage was discouraged and considered dangerous.
It also required a configure flag that caused an ABI incompatibility. The
module was removed in 3.7 by Nathaniel J. Smith in
`bpo-29137 <https://bugs.python.org/issue29137>`_.
Module type
C extension + CAPI
Deprecated in
3.7
Removed in
3.7
Substitute
**none**
Modules to keep
===============
Some modules were originally proposed for deprecation.
fileinput
---------
The `fileinput <https://docs.python.org/3/library/fileinput.html>`_ module
implements a helpers to iterate over a list of files from ``sys.argv``. The
module predates the optparser and argparser module. The same functionality
can be implemented with the argparser module.
Several core developers expressed their interest to keep the module in the
standard library, as it is handy for quick scripts.
Module type
pure Python
lib2to3
-------
The `lib2to3 <https://docs.python.org/3/library/2to3.html>`_ package provides
the ``2to3`` command to transpile Python 2 code to Python 3 code.
The package is useful for other tasks besides porting code from Python 2 to
3. For example `black`_ uses it for code reformatting.
Module type
pure Python
getopt
------
The `getopt <https://docs.python.org/3/library/getopt.html>`_ module mimics
C's getopt() option parser.
Although users are encouraged to use argparse instead, the getopt module is
still widely used. The module is small, simple, and handy for C developers
to write simple Python scripts.
Module type
pure Python
Substitute
argparse
optparse
--------
The `optparse <https://docs.python.org/3/library/optparse.html>`_ module is
the predecessor of the argparse module.
Although it has been deprecated for many years, it's still too widely used
to remove it.
Module type
pure Python
Deprecated in
3.2
Substitute
argparse
wave
----
The `wave <https://docs.python.org/3/library/wave.html>`_ module provides
support for the WAV sound format.
The module is not deprecated, because The WAV format is still relevant these
days. The ``wave`` module is also used in education, e.g. to show kids how
to make noise with a computer.
The module uses one simple function from the `audioop`_ module to perform
byte swapping between little and big endian formats. Before 24 bit WAV
support was added, byte swap used to be implemented with the ``array``
module. To remove ``wave``'s dependency on the ``audioop``, the byte swap
function could be either be moved to another module (e.g. ``operator``) or
the ``array`` module could gain support for 24 bit (3 byte) arrays.
Module type
pure Python (depends on *byteswap* from `audioop`_ C extension)
Deprecated in
3.8
To be removed in
3.10
Substitute
*n/a*
Future maintenance of removed modules
=====================================
The main goal of the PEP is to reduce the burden and workload on the Python
core developer team. Therefore removed modules will not be maintained by
the core team as separate PyPI packages. However the removed code, tests and
documentation may be moved into a new git repository, so community members
have a place from which they can pick up and fork code.
A first draft of a `legacylib <https://github.com/tiran/legacylib>`_
repository is available on my private Github account. The modules could be
made available on PyPI. The Python core team will not publish or maintain
the packages. It is my hope that members of the Python community will
adopt, maintain, and perhaps improve the deprecated modules.
It's my hope that some of the deprecated modules will be picked up and
adopted by users that actually care about them. For example ``colorsys`` and
``imghdr`` are useful modules, but have limited feature set. A fork of
``imghdr`` can add new features and support for more image formats, without
being constrained by Python's release cycle.
Most of the modules are in pure Python and can be easily packaged. Some
depend on a simple C module, e.g. `audioop`_ and `crypt`_. Since `audioop`_
does not depend on any external libraries, it can be shipped in as binary
wheels with some effort. Other C modules can be replaced with ctypes or cffi.
For example I created `legacycrypt <https://github.com/tiran/legacycrypt>`_
with ``_crypt`` extension reimplemented with a few lines of ctypes code.
Discussions
===========
* Elana Hashman and Nick Coghlan suggested to keep the *getopt* module.
* Berker Peksag proposed to deprecate and removed *msilib*.
* Brett Cannon recommended to delay active deprecation warnings and removal
of modules like *imp* until Python 3.10. Version 3.8 will be released
shortly before Python 2 reaches end of lifetime. A delay reduced churn for
users that migrate from Python 2 to 3.8.
* Brett also came up with the idea to keep lib2to3. The package is useful
for other purposes, e.g. `black <https://pypi.org/project/black/>`_ uses
it to reformat Python code.
* At one point, distutils was mentioned in the same sentence as this PEP.
To avoid lengthy discussion and delay of the PEP, I decided against dealing
with distutils. Deprecation of the distutils package will be handled by
another PEP.
* Multiple people (Gregory P. Smith, David Beazley, Nick Coghlan, ...)
convinced me to keep the `wave`_ module. [4]_
* Gregory P. Smith proposed to deprecate `nntplib`_. [4]_
* Andrew Svetlov mentioned the ``socketserver`` module is questionable.
However it's used to implement ``http.server`` and ``xmlrpc.server``. The
stdlib doesn't have a replacement for the servers, yet.
Update history
==============
Update 1
--------
* Deprecate `parser`_ module
* Keep `fileinput`_ module
* Elaborate why `crypt`_ and `spwd`_ are dangerous and bad
* Improve sections for `cgitb`_, `colorsys`_, `nntplib`_, and `smtpd`_ modules
* The `colorsys`_, `crypt`_, `imghdr`_, `sndhdr`_, and `spwd`_ sections now
list suitable substitutions.
* Mention that ``socketserver`` is going to stay for ``http.server`` and
``xmlrpc.server``
* The future maintenance section now states that the deprecated modules
may be adopted by Python community members.
References
==========
.. [1] https://en.wikipedia.org/wiki/Open_Sound_System#Free,_proprietary,_free
.. [2] https://man.openbsd.org/ossaudio
.. [3] https://blogs.msmvps.com/installsite/blog/2015/05/03/the-future-of-windows-…
.. [4] https://twitter.com/ChristianHeimes/status/1130257799475335169
.. [5] https://twitter.com/dabeaz/status/1130278844479545351
.. [6] https://mail.python.org/pipermail/python-dev/2019-May/157464.html
.. [7] https://discuss.python.org/t/switch-pythons-parsing-tech-to-something-more-…
Copyright
=========
This document has been placed in the public domain.
..
Local Variables:
mode: indented-text
indent-tabs-mode: nil
sentence-end-double-space: t
fill-column: 70
coding: utf-8
End:
[View Less]
17
35
First, I want to say: I'm very happy with PEP 558's changes to
f_locals. It solves the weird threading bugs, and exposes the
fundamental operations you need for debugging in a simple and clean
way, while leaving a lot of implementation flexibility for future
Python VMs. It's a huge improvement over what we had before.
I'm not as sure about the locals() parts of the proposal. It might be
fine, but there are some complex trade-offs here that I'm still trying
to wrap my head around. The rest of …
[View More]this document is me thinking out
loud to try to clarify these issues.
##### What are we trying to solve?
There are two major questions, which are somewhat distinct:
- What should the behavior of locals() be in CPython?
- How much of that should be part of the language definition, vs
CPython implementation details?
The status quo is that for locals() inside function scope, the
behavior is quite complex and subtle, and it's entirely implementation
defined. In the current PEP draft, there are some small changes to the
semantics, and also it promotes them becoming part of the official
language semantics.
I think the first question, about semantics, is the more important
one. If we're promoting them to the language definition, the main
effect is just to make it more important we get the semantics right.
##### What are the PEP's proposed semantics for locals()?
They're kinda subtle. [Nick: please double-check this section, both
for errors and because I think it includes some edge cases that the
PEP currently doesn't mention.]
For module/class scopes, locals() has always returned a mapping object
which acts as a "source of truth" for the actual local environment –
mutating the environment directly changes the mapping object, and
vice-versa. That's not going to change.
In function scopes, things are more complicated. The *local
environment* is conceptually well-defined, and includes:
- local variables (current source of truth: "fast locals" array)
- closed-over variables (current source of truth: cell objects)
- any arbitrary key/values written to frame.f_locals that don't
correspond to local or closed-over variables, e.g. you can do
frame.f_locals[object()] = 10, and then later read it out again.
However, the mapping returned by locals() does not directly reflect
this local environment. Instead, each function frame has a dict
associated with it. locals() returns this dict. The dict always holds
any non-local/non-closed-over variables, and also, in certain
circumstances, we write a snapshot of local and closed-over variables
back into the dict.
Specifically, we write back:
- Whenever locals() is called
- Whenever exec() or eval() is called without passing an explicit
locals argument
- After every trace/profile event, if a Python-level tracing/profiling
function is registered.
(Note: in CPython, the use of Python-level tracing/profiling functions
is extremely rare. It's more common in alternative implementations
like PyPy. For example, the coverage package uses a C-level tracing
function on CPython, which does not trigger locals updates, but on
PyPy it uses a Python-level tracing function, which does trigger
updates.)
In addition, the PEP doesn't say, but I think that any writes to
f_locals immediately update both the environment and the locals dict.
These semantics have some surprising consequences. Most obviously, in
function scope (unlike other scopes), mutating locals() does not
affect the actual local environment:
def f():
a = 1
locals()["a"] = 2
assert a == 1
The writeback rules can also produce surprising results:
def f():
loc1 = locals()
# Since it's a snapshot created at the time of the call
# to locals(), it doesn't contain 'loc1':
assert "loc1" not in loc1
loc2 = locals()
# Now loc1 has changed:
assert "loc1" in loc1
However, the results here are totally different if a Python-level
tracing/profiling function is installed – in particular, the first
assertion fails.
The interaction between f_locals and and locals() is also subtle:
def f():
a = 1
loc = locals()
assert "loc" not in loc
# Regular variable updates don't affect 'loc'
a = 2
assert loc["a"] == 1
# But debugging updates do:
sys._getframe().f_locals["a"] = 3
assert a == 3
assert loc["a"] == 3
# But it's not a full writeback
assert "loc" not in loc
# Mutating 'loc' doesn't affect f_locals:
loc["a"] = 1
assert sys._getframe().f_locals["a"] == 1
# Except when it does:
loc["b"] = 3
assert sys._getframe().f_locals["b"] == 3
Again, the results here are totally different if a Python-level
tracing/profiling function is installed.
And you can also hit these subtleties via 'exec' and 'eval':
def f():
a = 1
loc = locals()
assert "loc" not in loc
# exec() triggers writeback, and then mutates the locals dict
exec("a = 2; b = 3")
# So now the current environment has been reflected into 'loc'
assert "loc" in loc
# Also loc["a"] has been changed to reflect the exec'ed assignments
assert loc["a"] == 2
# But if we look at the actual environment, directly or via
# f_locals, we can see that 'a' has not changed:
assert a == 1
assert sys._getframe().f_locals["a"] == 1
# loc["b"] changed as well:
assert loc["b"] == 3
# And this *does* show up in f_locals:
assert sys._getframe().f_locals["b"] == 3
Of course, many of these edge cases are pretty obscure, so it's not
clear how much they matter. But I think we can at least agree that
this isn't the one obvious way to do it :-).
##### What's the landscape of possible semantics?
I did some brainstorming, and came up with 4 sets of semantics that
seem plausible enough to at least consider:
- [PEP]: the semantics in the current PEP draft.
- [PEP-minus-tracing]: same as [PEP], except dropping the writeback on
Python-level trace/profile events.
- [snapshot]: in function scope, each call to locals() returns a new,
*static* snapshot of the local environment, removing all this
writeback stuff. Something like:
def locals():
frame = get_caller_frame()
if is_function_scope(frame):
# make a point-in-time copy of the "live" proxy object
return dict(frame.f_locals)
else:
# in module/class scope, return the actual local environment
return frame.f_locals
- [proxy]: Simply return the .f_locals object, so in all contexts
locals() returns a live mutable view of the actual environment:
def locals():
return get_caller_frame().f_locals
##### How to evaluate our options?
I can think of a lot of criteria that all-else-being-equal we would
like Python to meet. (Of course, in practice they conflict.)
Consistency across APIs: it's surprising if locals() and
frame.f_locals do different things. This argues for [proxy].
Consistency across contexts: it's surprising if locals() has acts
differently in module/class scope versus function scope. This argues
for [proxy].
Consistent behavior when the environment shifts, or small maintenance
changes are made: it's nice if code that works today keeps working
tomorrow. On this criterion, I think [snapshot] > [proxy] >
[PEP-minus-tracing] >>> [PEP]. [PEP] is particularly bad here because
a very rare environmental change that almost no-one tests and mostly
only happens when debugging (i.e., enabling tracing) causes a radical
change in semantics.
Simplicity of explaining to users: all else being equal, it's nice if
our docs are short and clear and the language fits in your head. On
this criterion, I think: [proxy] > [snapshot] > [PEP-minus-tracing] >
[PEP]. As evidence that the current behavior is confusing, see:
- Ned gets confused and writes a long blog post after he figures it
out: https://nedbatchelder.com/blog/201211/tricky_locals.html
- A linter that warns against mutating locals():
https://lgtm.com/rules/10030096/
Simplicity of implementation: "If the implementation is easy to
explain, it may be a good idea." Since we need the proxy code anyway
to implement f_locals, I think it's: [proxy] (free) > [snapshot] (one
'if' statement) > ([PEP] = [PEP-minus-tracing]).
Impact on other interpreter implementations: all else being equal,
we'd like to give new interpreters maximal freedom to do clever
things. (And local variables are a place where language VMs often
expend a lot of cleverness.) [proxy] and [snapshot] are both easily
implemented in terms of f_locals, so they basically don't constrain
alternative implementations at all. I'm not as sure about [PEP] and
[PEP-minus-tracing]. I originally thought they must be horrible. On
further thought, I'm not convinced they're *that* bad, since the need
to support people doing silly stuff like frame.f_locals[object()] = 10
means that implementations will already need to sometimes attach
something like a dict object to their function frames. But perhaps
alternative implementations would like to disallow this, or are OK
with making it really slow but care about locals() performance more.
Anyway, it's definitely ([proxy] = [snapshot]) > ([PEP] =
[PEP-minus-tracing]), but I'm not sure whether the '>' is large or
small.
Backwards compatibility: help(locals) says:
NOTE: Whether or not updates to this dictionary will affect
name lookups in the local scope and vice-versa is
*implementation dependent* and not covered by any backwards
compatibility guarantees.
So that claims that there are ~no backwards compatibility issues here.
I'm going to ignore that; no matter what the docs say, we still don't
want to break everyone's code. And unfortunately, I can't think of any
realistic way to do a gradual transition, with like deprecation
warnings and all that (can anyone else?), so whatever we do will be a
flag-day change.
Of our four options, [PEP] is intuitively the closest to what CPython
has traditionally done. But what exactly breaks under the different
approaches? I'll split this off into its own section.
##### Backwards compatibility
I'll split this into three parts: code that treats locals() as
read-only, exec()/eval(), and code that mutates locals().
I believe (but haven't checked) that the majority of uses of locals()
are in simple cases like:
def f():
....
print("{a} {b}".format(**locals()))
Luckily, this code remains totally fine under all four of our candidates.
exec() and eval() are an interesting case. In Python 2, exec'ing some
assignments actually *did* mutate the local environment, e.g. you
could do:
# Python 2
def f():
exec "a = 1"
assert a == 1
In Python 3, this was changed, so now exec() inside a function cannot
mutate the enclosing scope. We got some bug reports about this change,
and there are a number of questions on stackoverflow about it, e.g.:
- https://bugs.python.org/issue4831
- https://stackoverflow.com/questions/52217525/how-can-i-change-the-value-of-…
- https://stackoverflow.com/questions/50995581/eval-exec-with-assigning-varia…
In all released versions of Python, eval() was syntactically unable to
rebind variables, so eval()'s interaction with the local environment
was undefined. However, in 3.8, eval() *will* be able to rebind
variables using the ':=' operator, so this interaction will become
user-visible. Presumably we'll want eval() to match exec().
OK, with that background out of the way, let's look at our candidates.
If we adopt [proxy], then that will mean exec()-inside-functions will
go back to the Python 2 behavior, where executing assignments in the
enclosing scope actually changes the enclosing scope. This will likely
break some code out there that's relying on the Python 3 behavior,
though I don't know how common that is. (I'm guessing not too common?
Using the same variable inside and outside an 'exec' and trusting that
they *won't* be the same seems like an unusual thing to do. But I
don't know.)
With [PEP] and [PEP-minus-tracing], exec() is totally unchanged. With
[snapshot], there's technically a small difference: if you call
locals() and then exec(), the exec() no longer triggers an implicit
writeback to the dict that locals() returned. I think we can ignore
this, and say that for all three of these, exec() is unlikely to
produce backwards compatibility issues.
OK, finally, let's talk about code that calls locals() and then
mutates the return value. The main difference between our candidates
is how they handle mutation, so this seems like the most important
case to focus on.
Conveniently, Mark Shannon and friends have statically analyzed a
large corpus of Python code and made a list of cases where people do
this: https://lgtm.com/rules/10030096/alerts/
Thanks! I haven't gone through the whole list, but I read through the
first few in the hopes of getting a better sense of what kind of code
does this in the real world and how it would be impacted by our
different options.
https://lgtm.com/projects/g/pydata/xarray/snapshot/a2ac6af744584c8afed3d56d…
Current: raises if a Python-level trace/profile function is set
[PEP]: raises if a Python-level trace/profile function is set
[PEP-minus-tracing]: ok
[snapshot]: ok
[proxy]: always raises
Comment: uses locals() to capture a bunch of passed in kwargs so it
can pass them as **kwargs to another function, and treats them like a
snapshot. The authors were clearly aware of the dangers, because this
pattern appears multiple times in this file, and all the other places
make an explicit copy of locals() before using it, but this place
apparently got missed. Fix is trivial: just do that here too.
https://github.com/swagger-api/swagger-codegen/blob/master/modules/swagger-…
Current: ok
[PEP]: ok
[PEP-minus-tracing]: ok
[snapshot]: ok
[proxy]: ok
Comment: this is inside a tool used to generate Python wrappers for
REST APIs. The vast majority of entries in the lgtm database are from
code generated by this tool. This was tricky to analyze, because it's
complex templated code, and it does mutate locals(). But I'm pretty
confident that the mutations end up just... not mattering, and all
possible generated code works under all of our candidates. Which is
lucky, because if we did have to fix this it wouldn't be trivial:
fixing up the code generator itself wouldn't be too hard, but it'll
take years for everyone to regenerate their old wrappers.
https://lgtm.com/projects/g/saltstack/salt/snapshot/bb0950e5eafbb897c8e969e…
Current: ok
[PEP]: ok
[PEP-minus-tracing]: ok
[snapshot]: raises
[proxy]: ok
Comment: the use of locals() here is totally superfluous – it
repeatedly reads and writes to locals()[mod], and in all cases this
could be replaced by a simple variable, or any other dict. And it's
careful not to assign to any name that matches an actual local
variable, so it works fine with [proxy] too. But it does assume that
multiple calls to locals() return the same dict, so [snapshot] breaks
it. In this case the fix is trivial: just use a variable.
https://lgtm.com/projects/g/materialsproject/pymatgen/snapshot/fd6900ed1040…
Current: buggy if a trace/profile function is set
[PEP]: buggy if a trace/profile function is set
[PEP-minus-tracing]: ok
[snapshot]: ok
[proxy]: raises
Comment: Another example of collecting kwargs into a dict. Actually
this code is always buggy, because they seem to think that
dict.pop("a", "b") removes both "a" and "b" from the dict... but it
would raise an exception on [proxy], because they use one of those
local variables after popping it from locals(). Fix is trivial: take
an explicit snapshot of locals() before modifying it.
##### Conclusion so far
[PEP-minus-tracing] seems to strictly dominate [PEP]. It's equal or
better on all criteria, and actually *more* compatible with all the
legacy code I looked at, even though it's technically less consistent
with what CPython used to do. Unless someone points out some new
argument, I think we can reject the writeback-when-tracing part of the
PEP draft.
Choosing between the remaining three is more of a judgement call.
I'm leaning towards saying that on net, [snapshot] beats
[PEP-minus-tracing]: it's dramatically simpler, and the backwards
incompatibilities that we've found so far seem pretty minor, on par
with what we do in every point release. (In fact, in 3/4 of the cases
I looked at, [snapshot] is actually what users seemed to trying to use
in the first place.)
For [proxy] versus [snapshot], a lot depends on what we think of
changing the semantics of exec(). [proxy] is definitely more
consistent and elegant, and if we could go back in time I think it's
what we'd have done from the start. Its compatibility is maybe a bit
worse than [snapshot] on non-exec() cases, but this seems pretty minor
overall (it often doesn't matter, and if it does just write
dict(locals()) instead of locals(), like you would in non-function
scope). But the change in exec() semantics is an actual language
change, even though it may not affect much real code, so that's what
stands out for me.
I'd very much like to hear about any considerations I've missed, and
any opinions on the "judgement call" part.
--
Nathaniel J. Smith -- https://vorpus.org
[View Less]
10
26
Hello,
Berker and I have been working on a PEP that suggests we keep using
and improving bugs.python.org and Roundup instead of switching to
GitHub Issues as proposed by PEP 581.
The PEP covers:
* What are the advantages of Roundup over GitHub issues;
* What features are missing in Roundup and how can we add them;
* Issues with PEP 581;
* Issues with the migration plan proposed by PEP 588;
The rendered version of PEP 595 is available at
https://www.python.org/dev/peps/pep-0595/
For …
[View More]reference, you can consult PEP 581 and 588 at
https://www.python.org/dev/peps/pep-0581/ and
https://www.python.org/dev/peps/pep-0588/
The full text of the PEP is include below. We are planning to update
the PEP to include the feedback we receive and to update the status of
features as we implement them (we also have a Google Summer of Code
students working on it).
Best Regards,
Ezio Melotti
================
PEP: 595
Title: Improving bugs.python.org
Author: Ezio Melotti <ezio.melotti(a)gmail.com>, Berker Peksag
<berker.peksag(a)gmail.com>
Status: Draft
Type: Process
Content-Type: text/x-rst
Created: 12-May-2019
Abstract
========
This PEP proposes a list of improvements to make bugs.python.org
more usable for contributors and core developers. This PEP also
discusses why remaining on Roundup should be preferred over
switching to GitHub Issues, as proposed by :pep:`581`.
Motivation
==========
On May 14th, 2019 :pep:`581` has been accepted [#]_ without much
public discussion and without a clear consensus [#]_. The PEP
contains factual errors and doesn't address some of the
issues that the migration to GitHub Issues might present.
Given the scope of the migration, the amount of work required,
and how it will negatively affect the workflow during the
transition phase, this decision should be re-evaluated.
Roundup advantages over GitHub Issues
=====================================
This section discusses reasons why Roundup should be preferred
over GitHub Issues and Roundup features that are not available
on GitHub Issues.
* **Roundup is the status quo.** Roundup has been an integral
part of the CPython workflow for years. It is a stable product
that has been tested and customized to adapt to our needs as the
workflow evolved.
It is possible to gradually improve it and avoid the disruption
that a switch to a different system would inevitabily bring to
the workflow.
* **Open-source and Python powered.** Roundup is an open-source
project and is written in Python. By using it and supporting
it, we also support the Python ecosystem. Several features
developed for bpo have also been ported to upstream Roundup
over the years.
* **Fully customizable.** Roundup can be (and has been) fully
customized to fit our needs.
* **Finer-grained access control.** Roundup allows the creation
of different roles with different permissions (e.g. create,
view, edit, etc.) for each individual property, and users can
have multiple roles.
* **Flexible UI.** While Roundup UI might look dated, it is
convenient and flexible.
For example, on the issue page, each field (e.g. title, type,
versions, status, linked files and PRs, etc.) have appropriate
UI elements (input boxes, dropdowns, tables, etc.) that are
easy to set and also provide a convenient way to get info about
the issue at a glance. The number of fields, their values, and
the UI element they use is also fully customizable.
GitHub only provides labels.
The issue list page presents the issues in a compact and easy
to read table with separate columns for different fields. For
comparison, Roundup lists 50 issues in a screen, whereas GitHub
takes two screens to shows 25 issues.
* **Advanced search.** Roundup provides an accurate way to search
and filter by using any combination of issue fields.
It is also possible to customize the number of results and the
fields displayed in the table, and the sorting and grouping
(up to two levels).
bpo also provides predefined summaries (e.g. "Created by you",
"Assigned to you", etc.) and allows the creation of custom
search queries that can be conveniently accessed from the sidebar.
* **Nosy list autocomplete.** The nosy list has an autocomplete
feature that suggests maintainers and experts. The suggestions
are automatically updated when the experts index [#]_ changes.
* **Dependencies and Superseders.** Roundup allows to specify
dependencies that must be addressed before the current issues
can be closed and a superseder issue to easily mark duplicates
[#]_. The list of dependencies can also be used to create
meta-issues that references several other sub-issues [#]_.
Improving Roundup
=================
This section lists some of the issues mentioned by :pep:`581`
and other desired features and discusses how they can be implemented
by improving Roundup and/or our instance.
* **REST API support.** A REST API will make integration with other
services and the development of new tools and applications easiers.
Upstream Roundup now supports a REST API. Updating the tracker will
make the REST API available.
* **GitHub login support.** This will allow users to login
to bugs.python.org (bpo) without having to create a new account.
It will also solve issues with confirmation emails being marked
as spam, and provide two-factor authentication.
A patch to add this functionality is already available and is
being integrated at the time of writing [#]_.
* **Markdown support and message preview and editing.** This feature
will allow the use of Markdown in messages and the ability to
preview the message before the submission and edit it afterward.
This can be done, but it will take some work. Possible solutions
have been proposed on the roundup-devel mailing list [#]_.
* **"Remove me from nosy list" button.** Add a button on issue pages
to remove self from the nosy list.
This feature will be added during GSoC 2019.
* **Mobile friendly theme.** Current theme of bugs.python.org looks
dated and it doesn't work well with mobile browsers.
A mobile-friendly theme that is more modern but still familiar
will be added.
* **Add PR link to BPO emails.** Currently bpo emails don't include
links to the corresponding PRs.
A patch [#]_ is available to change the content of the bpo emails
from::
components: +Tkinter
versions: +Python 3.4
pull_requests: +42
to::
components: +Tkinter
versions: +Python 3.4
pull_request: https://github.com/python/cpython/pull/341
* **Python 3 support.** Using Python 3 will make maintenance easier.
Upstream Roundup now supports Python 3. Updating the tracker will
allow us to switch to Python 3. The instances will need to be
updated as well.
* **Use upstream Roundup.** We currently use a fork of Roundup with
a few modifications, most notably the GitHub integration. If this
is ported upstream, we can start using upstream Roundup without
having to maintain our fork.
PEP 581 issues
==============
This section addresses some errors and inaccuracies found in :pep:`581`.
The "Why GitHub?" section of PEP 581 lists features currently
available on GitHub Issues but not on Roundup. Some of this features
are currently supported:
* "Ability to reply to issue and pull request conversations via email."
* Being able to reply by email has been one of the core features of
Roundup since the beginning. It is also possible to create new
issues or close existing ones, set or modify fields, and add
attachments.
* "Email notifications containing metadata, integrated with Gmail,
allowing systematic filtering of emails."
* Emails sent by Roundup contains metadata that can be used for
filtering.
* "Additional privacy, such as offering the user a choice to hide an
email address, while still allowing communication with the user
through @-mentions."
* Email addresses are hidden by default to users that are not
registered. Registered users can see other users' addresses
because we configured the tracker to show them. It can easily
be changed if desired. Users can still be added to the nosy
list by using their username even if their address is hidden.
* "Ability to automatically close issues when a PR has been merged."
* The GitHub integration of Roundup automatically closes issues
when a commit that contains "fixes issue <id>" is merged.
(Alternative spellings such as "closes" or "bug" are also supported.)
See [#]_ for a recent example of this feature.
* "Support for permalinks, allowing easy quoting and copying &
pasting of source code."
* Roundup has permalinks for issues, messages, attachments, etc.
In addition, Roundup allows to easily rewrite broken URLs in
messages (e.g. if the code hosting changes).
* "Core developers, volunteers, and the PSF don't have to maintain the
issue infrastructure/site, giving us more time and resources to focus
on the development of Python."
* While this is partially true, additional resources are required to
write and maintain bots.
In some cases, bots are required to workaround GitHub's lack of
features rather than expanding. [#]_ was written
specifically to workaround GitHub's email integration.
Updating our bots to stay up-to-date with changes in the GitHub API
has also maintenance cost. [#]_ took two days to be fixed.
In addition, we will still need to maintain Roundup for bpo (even
if it becomes read-only) and for the other trackers
we currently host/maintain (Jython [#]_ and Roundup [#]_).
The "Issues with Roundup / bpo" section of :pep:`581` lists some issues
that have already been fixed:
* "The upstream Roundup code is in Mercurial. Without any CI available,
it puts heavy burden on the few existing maintainers in terms of
reviewing, testing, and applying patches."
* While Roundup uses Mercurial by default, there is a git clone
available on GitHub [#]_. Roundup also has CI available [#]_ [#]_.
* "There is no REST API available. There is an open issue in Roundup for
adding REST API. Last activity was in 2016."
* The REST API has been integrated and it's now available in Roundup.
* "Users email addresses are exposed. There is no option to mask it."
* Exposing addresses to registered and logged in users was a decision
taken when our instance was set up.
This has now been changed to make the email addresses hidden for
regular users too (Developers and Coordinators can still see them).
The "Email address"" column from the user listing page [#]_ has been
removed too.
* "It sends a number of unnecessary emails and notifications, and it is
difficult, if not impossible, to configure."
* This can be configured.
* "Creating an account has been a hassle. There have been reports of people
having trouble creating accounts or logging in."
* The main issue is confirmation emails being marked as spam. Work has
been done to resolve the issue.
Migration considerations
========================
This section describes issues with the migrations that might not
have been addressed by :pep:`581` and :pep:`588`.
:pep:`588` suggests to add a button to migrate issues to GitHub
only when someone wants to keep working on them. This approach
has several issues:
* bpo will need to be updated in order to add a button that,
once pressed, creates a new issue on GitHub, copies over all
the messages, attachments, and creates/adds label for the
existing fields. Permissions will also need to be tweaked
to make individual issues read-only once they are migrated,
and to prevent users to create new accounts.
* The issues will be split between two trackers; searching issues
will take significant more effort.
* The conversion from Roundup to GitHub is lossy, unless all
the bpo fields are converted into labels or preserved somewhere
else.
* bpo converts a number of references into links, including
issue, message, and PR IDs, changeset numbers, legacy SVN
revision numbers, paths to files in the repo, files in
tracebacks (detecting the correct branch), links to devguide
pages and sections [#]_. This happens when messages are
requested so it is possible to create the correct link (e.g.
all the file links used to point to hg.python.org and now
point to GitHub).
If the links are hardcoded during the migration, it will be
difficult (if not impossible) to change them later. If they
aren't, they will either be lost, or a tool to generate the
links and updating them will need to be written.
* GitHub doesn't provide a way to set and preserve issue IDs
if they are migrated automatically with the use of a button.
(Some projects managed to preserve the IDs by contaacting
the GitHub staff and migrating the issues *en masse*.)
* On top of the work and changes required to migrate to GitHub
issues, we will still need to keep running and maintaining
Roundup, for both our instance (read-only) and for the Jython
and Roundup trackers (read-write).
In addition to the issues listed in the "Open issues" section of
:pep:`588`, this issues will need to be addressed:
* GitHub is properietary and there is risk of vendor lock-in.
Their business model might change and they could shut down
altogether.
* Switching to GitHub Issues will likely increase the number of
invalid reports and increase the triaging effort. This concern
has been raised in the past in a Zulip topic [#]_.
There have been already cases where people posted comments on
PRs that required moderators to mark them as off-topic or
disruptive, delete them altogether, and even lock the
conversation [#]_.
* Roundup sends weekly reports to python-dev with a summary that
includes new issues, recent issues with no replies, recent
issues waiting for review, most discussed issues, closed issues,
and deltas for open/closed/total issue counts [#]_. The report
provides an easy way to keep track of the tracker activity and
to make sure that issues that require attention are noticed.
The data collect by the weekly report is also use to generate
statistics and graphs that can be used to gain new insights [#]_.
* There are currently two mailing lists where Roundup posts new
tracker issues and all messages respectively: new-bugs-announce
[#]_ and python-bugs-list [#]_. A new system will need to be
developed to preserve this functionality. These MLs offer
additional ways to keep track of the tracker activity.
References
==========
.. [#] [Python-Dev] PEP 581 (Using GitHub issues for CPython) is accepted
https://mail.python.org/pipermail/python-dev/2019-May/157399.html
.. [#] [python-committers] [Python-Dev] PEP 581 (Using GitHub issues
for CPython) is accepted
https://mail.python.org/pipermail/python-committers/2019-May/006755.html
.. [#] Experts Index -- Python Devguide
https://devguide.python.org/experts/
.. [#] An example of superseded issues:
"re.sub() replaces only several matches"
https://bugs.python.org/issue12078
.. [#] An example of meta issue using dependencies to track sub-issues:
"Meta-issue: support of the android platform""
https://bugs.python.org/issue26865
.. [#] Support logging in with GitHub
https://github.com/python/bugs.python.org/issues/7
.. [#] Re: [Roundup-devel] PEP 581 and Google Summer of Code
https://sourceforge.net/p/roundup/mailman/message/36667828/
.. [#] [Tracker-discuss] [issue624] bpo emails contain useless non-github
pull_request number - users want a link to actual github PR
https://mail.python.org/pipermail/tracker-discuss/2018-June/004547.html
.. [#] The commit reported in msg342882 closes the issue (see the history below)
https://bugs.python.org/issue36951#msg342882
.. [#] The cpython-emailer-webhook project
https://github.com/berkerpeksag/cpython-emailer-webhook
.. [#] A recent incident caused by GitHub
https://github.com/python/bedevere/pull/163
.. [#] Jython issue tracker
https://bugs.jython.org/
.. [#] Roundup issue tracker
https://issues.roundup-tracker.org/
.. [#] GitHub clone of Roundup
https://github.com/roundup-tracker/roundup
.. [#] Travis-CI for Roundup
https://travis-ci.org/roundup-tracker/roundup) and codecov
.. [#] Codecov for Roundup
https://codecov.io/gh/roundup-tracker/roundup/commits
.. [#] User listing -- Python tracker
https://bugs.python.org/user?@sort=username
.. [#] Generating Special Links in a Comment -- Python Devguide
https://devguide.python.org/triaging/#generating-special-links-in-a-comment
.. [#] The New-bugs-announce mailing list
https://mail.python.org/mailman/listinfo/new-bugs-announce
.. [#] The Python-bugs-list mailing list
https://mail.python.org/mailman/listinfo/python-bugs-list
.. [#] An example of [Python-Dev] Summary of Python tracker Issues
https://mail.python.org/pipermail/python-dev/2019-May/157483.html
.. [#] Issues stats -- Python tracker
https://bugs.python.org/issue?@template=stats
.. [#] s/n ratio -- Python -- Zulip
https://python.zulipchat.com/#narrow/stream/130206-pep581/topic/s.2Fn.20rat…
.. [#] For example this and other related PRs:
https://github.com/python/cpython/pull/9099
Copyright
=========
This document has been placed in the public domain.
..
Local Variables:
mode: indented-text
indent-tabs-mode: nil
sentence-end-double-space: t
fill-column: 70
coding: utf-8
End:
[View Less]
7
12

June 1, 2019
Hi,
Thanks to the constructive discussions, I enhanced my PEP 587. I don't
plan any further change, the PEP is now ready to review (and maybe
even for pronouncement, hi Thomas! :-)).
The Rationale now better explains all challenges and the complexity of
the Python Initialization Configuration.
The "Isolate Python" section is a short guide explaining how configure
Python to embed it into an application.
The "Path Configuration" section elaborates the most interesting part
of the …
[View More]configuration: configure where Python looks for modules
(sys.path). I added PyWideStringList_Insert() to allow to prepend a
path in module_search_paths.
The "Python Issues" section give a long list of issues solved directly
or indirectly by this PEP.
I'm open for bikeshedding on PyConfig fields names and added functions
names ;-) I hesitate for "use_module_search_paths": maybe
"module_search_paths_set" is a better name, as in "is
module_search_paths set?". The purpose of this field is to allow to
have an empty sys.path (ask PyConfig_Read() to not override it). IMHO
an empty sys.path makes sense for some specific use cases, like
executing Pyhon code without any external module.
My PEP 587 proposes better names: Py_FrozenFlag becomes
PyConfig.pathconfig_warnings and Py_DebugFlag becomes
PyConfig.parser_debug. I also avoided double negation. For example,
Py_DontWriteBytecodeFlag becomes write_bytecode.
Changes between version 3 and version 2:
* PyConfig: Add configure_c_stdio and parse_argv; rename _frozen to
pathconfig_warnings.
* Rename functions using bytes strings and wide strings. For example,
Py_PreInitializeFromWideArgs() becomes Py_PreInitializeFromArgs(), and
PyConfig_SetArgv() becomes PyConfig_SetBytesArgv().
* Add PyWideStringList_Insert() function.
* New "Path configuration", "Isolate Python", "Python Issues" and
"Version History" sections.
* PyConfig_SetString() and PyConfig_SetBytesString() now requires the
configuration as the first argument.
* Rename Py_UnixMain() to Py_BytesMain()
HTML version:
https://www.python.org/dev/peps/pep-0587/
Full PEP text below.
I know that the PEP is long, but well, it's a complex topic, and I
chose to add many examples to make the API easier to understand.
Victor
---
PEP: 587
Title: Python Initialization Configuration
Author: Victor Stinner <vstinner(a)redhat.com>, Nick Coghlan <ncoghlan(a)gmail.com>
BDFL-Delegate: Thomas Wouters <thomas(a)python.org>
Discussions-To: python-dev(a)python.org
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 27-Mar-2019
Python-Version: 3.8
Abstract
========
Add a new C API to configure the Python Initialization providing finer
control on the whole configuration and better error reporting.
It becomes possible to read the configuration and modify it before it is
applied. It also becomes possible to completely override how Python
computes the module search paths (``sys.path``).
Building a customized Python which behaves as regular Python becomes
easier using the new ``Py_RunMain()`` function. Moreover, command line
arguments passed to ``PyConfig.argv`` are now parsed as the regular
Python parses command line options, and ``PyConfig.xoptions`` are
handled as ``-X opt`` command line options.
This extracts a subset of the API design from the PEP 432 development and
refactoring work that is now considered sufficiently stable to make public
(allowing 3rd party embedding applications access to the same configuration
APIs that the native CPython CLI is now using).
Rationale
=========
Python is highly configurable but its configuration evolved organically.
The initialization configuration is scattered all around the code using
different ways to set them: global configuration variables (ex:
``Py_IsolatedFlag``), environment variables (ex: ``PYTHONPATH``),
command line arguments (ex: ``-b``), configuration files (ex:
``pyvenv.cfg``), function calls (ex: ``Py_SetProgramName()``). A
straightforward and reliable way to configure Python is needed.
Some configuration parameters are not accessible from the C API, or not
easily. For example, there is no API to override the default values of
``sys.executable``.
Some options like ``PYTHONPATH`` can only be set using an environment
variable which has a side effect on Python child processes.
Some options also depends on other options: see `Priority and Rules`_.
Python 3.7 API does not provide a consistent view of the overall
configuration.
The C API of Python 3.7 Initialization takes ``wchar_t*`` strings as
input whereas the Python filesystem encoding is set during the
initialization which can lead to mojibake.
Python 3.7 APIs like ``Py_Initialize()`` aborts the process on memory
allocation failure which is not convenient when Python is embedded.
Moreover, ``Py_Main()`` could exit directly the process rather than
returning an exit code. Proposed new API reports the error or exit code
to the caller which can decide how to handle it.
Implementing the PEP 540 (UTF-8 Mode) and the new ``-X dev`` correctly
was almost impossible in Python 3.6. The code base has been deeply
reworked in Python 3.7 and then in Python 3.8 to read the configuration
into a structure with no side effect. It becomes possible to clear the
configuration (release memory) and read again the configuration if the
encoding changed . It is required to implement properly the UTF-8 which
changes the encoding using ``-X utf8`` command line option. Internally,
bytes ``argv`` strings are decoded from the filesystem encoding. The
``-X dev`` changes the memory allocator (behaves as
``PYTHONMALLOC=debug``), whereas it was not possible to change the
memory allocation *while* parsing the command line arguments. The new
design of the internal implementation not only allowed to implement
properly ``-X utf8`` and ``-X dev``, it also allows to change the Python
behavior way more easily, especially for corner cases like that, and
ensure that the configuration remains consistent: see `Priority and
Rules`_.
This PEP is a partial implementation of PEP 432 which is the overall
design. New fields can be added later to ``PyConfig`` structure to
finish the implementation of the PEP 432 (e.g. by adding a new partial
initialization API which allows to configure Python using Python objects to
finish the full initialization). However, those features are omitted from this
PEP as even the native CPython CLI doesn't work that way - the public API
proposal in this PEP is limited to features which have already been implemented
and adopted as private APIs for us in the native CPython CLI.
Python Initialization C API
===========================
This PEP proposes to add the following new structures, functions and
macros.
New structures (4):
* ``PyConfig``
* ``PyInitError``
* ``PyPreConfig``
* ``PyWideStringList``
New functions (17):
* ``Py_PreInitialize(config)``
* ``Py_PreInitializeFromBytesArgs(config, argc, argv)``
* ``Py_PreInitializeFromArgs(config, argc, argv)``
* ``PyWideStringList_Append(list, item)``
* ``PyWideStringList_Insert(list, index, item)``
* ``PyConfig_SetString(config,config_str, str)``
* ``PyConfig_SetBytesString(config, config_str, str)``
* ``PyConfig_SetBytesArgv(config, argc, argv)``
* ``PyConfig_SetArgv(config, argc, argv)``
* ``PyConfig_Read(config)``
* ``PyConfig_Clear(config)``
* ``Py_InitializeFromConfig(config)``
* ``Py_InitializeFromBytesArgs(config, argc, argv)``
* ``Py_InitializeFromArgs(config, argc, argv)``
* ``Py_BytesMain(argc, argv)``
* ``Py_RunMain()``
* ``Py_ExitInitError(err)``
New macros (9):
* ``PyPreConfig_INIT``
* ``PyConfig_INIT``
* ``Py_INIT_OK()``
* ``Py_INIT_ERR(MSG)``
* ``Py_INIT_NO_MEMORY()``
* ``Py_INIT_EXIT(EXITCODE)``
* ``Py_INIT_IS_ERROR(err)``
* ``Py_INIT_IS_EXIT(err)``
* ``Py_INIT_FAILED(err)``
This PEP also adds ``_PyRuntimeState.preconfig`` (``PyPreConfig`` type)
and ``PyInterpreterState.config`` (``PyConfig`` type) fields to these
internal structures. ``PyInterpreterState.config`` becomes the new
reference configuration, replacing global configuration variables and
other private variables.
PyWideStringList
----------------
``PyWideStringList`` is a list of ``wchar_t*`` strings.
Example to initialize a string from C static array::
static wchar_t* argv[2] = {
L"-c",
L"pass",
};
PyWideStringList config_argv = PyWideStringList_INIT;
config_argv.length = Py_ARRAY_LENGTH(argv);
config_argv.items = argv;
``PyWideStringList`` structure fields:
* ``length`` (``Py_ssize_t``)
* ``items`` (``wchar_t**``)
Methods:
* ``PyInitError PyWideStringList_Append(PyWideStringList *list, const
wchar_t *item)``:
Append *item* to *list*.
* ``PyInitError PyWideStringList_Insert(PyWideStringList *list,
Py_ssize_t index, const wchar_t *item)``:
Insert *item* into *list* at *index*. If *index* is greater than
*list* length, just append *item* to *list*.
If *length* is non-zero, *items* must be non-NULL and all strings must
be non-NULL.
PyInitError
-----------
``PyInitError`` is a structure to store an error message or an exit code
for the Python Initialization. For an error, it stores the C function
name which created the error.
Example::
PyInitError alloc(void **ptr, size_t size)
{
*ptr = PyMem_RawMalloc(size);
if (*ptr == NULL) {
return Py_INIT_NO_MEMORY();
}
return Py_INIT_OK();
}
int main(int argc, char **argv)
{
void *ptr;
PyInitError err = alloc(&ptr, 16);
if (Py_INIT_FAILED(err)) {
Py_ExitInitError(err);
}
PyMem_Free(ptr);
return 0;
}
``PyInitError`` fields:
* ``exitcode`` (``int``):
argument passed to ``exit()``, only set by ``Py_INIT_EXIT()``.
* ``err_msg`` (``const char*``): error message
* private ``_func`` field: used by ``Py_INIT_ERR()`` to store the C
function name which created the error.
* private ``_type`` field: for internal usage only.
Macro to create an error:
* ``Py_INIT_OK()``: Success.
* ``Py_INIT_ERR(err_msg)``: Initialization error with a message.
* ``Py_INIT_NO_MEMORY()``: Memory allocation failure (out of memory).
* ``Py_INIT_EXIT(exitcode)``: Exit Python with the specified exit code.
Other macros and functions:
* ``Py_INIT_IS_ERROR(err)``: Is the result an error?
* ``Py_INIT_IS_EXIT(err)``: Is the result an exit?
* ``Py_INIT_FAILED(err)``: Is the result an error or an exit? Similar
to ``Py_INIT_IS_ERROR(err) || Py_INIT_IS_EXIT(err)``.
* ``Py_ExitInitError(err)``: Call ``exit(exitcode)`` on Unix or
``ExitProcess(exitcode)`` if the result is an exit, call
``Py_FatalError(err_msg)`` if the result is an error. Must not be
called if the result is a success.
Pre-Initialization with PyPreConfig
-----------------------------------
``PyPreConfig`` structure is used to pre-initialize Python:
* Set the memory allocator
* Configure the LC_CTYPE locale
* Set the UTF-8 mode
Example using the pre-initialization to enable the UTF-8 Mode::
PyPreConfig preconfig = PyPreConfig_INIT;
preconfig.utf8_mode = 1;
PyInitError err = Py_PreInitialize(&preconfig);
if (Py_INIT_FAILED(err)) {
Py_ExitInitError(err);
}
/* at this point, Python will speak UTF-8 */
Py_Initialize();
/* ... use Python API here ... */
Py_Finalize();
Functions to pre-initialize Python:
* ``PyInitError Py_PreInitialize(const PyPreConfig *config)``
* ``PyInitError Py_PreInitializeFromBytesArgs(const PyPreConfig
*config, int argc, char **argv)``
* ``PyInitError Py_PreInitializeFromArgs(const PyPreConfig *config,
int argc, wchar_t **argv)``
These functions can be called with *config* set to ``NULL``.
If Python is initialized with command line arguments, the command line
arguments must also be passed to pre-initialize Python, since they have
an effect on the pre-configuration like encodings. For example, the
``-X utf8`` command line option enables the UTF-8 Mode.
These functions can be called with *config* set to ``NULL``. The caller
is responsible to handle error or exit using ``Py_INIT_FAILED()`` and
``Py_ExitInitError()``.
``PyPreConfig`` fields:
* ``allocator`` (``char*``, default: ``NULL``):
Name of the memory allocator (ex: ``"malloc"``).
* ``coerce_c_locale`` (``int``, default: 0):
If equals to 2, coerce the C locale; if equals to 1, read the LC_CTYPE
locale to decide if it should be coerced.
* ``coerce_c_locale_warn`` (``int``, default: 0):
If non-zero, emit a warning if the C locale is coerced.
* ``dev_mode`` (``int``, default: 0):
See ``PyConfig.dev_mode``.
* ``isolated`` (``int``, default: 0):
See ``PyConfig.isolated``.
* ``legacy_windows_fs_encoding`` (``int``, Windows only, default: 0):
If non-zero, disable UTF-8 Mode, set the Python filesystem encoding to
``mbcs``, set the filesystem error handler to ``replace``.
* ``use_environment`` (``int``, default: 1):
See ``PyConfig.use_environment``.
* ``utf8_mode`` (``int``, default: 0):
If non-zero, enable the UTF-8 mode.
``PyPreConfig`` private field, for internal use only:
* ``_config_version`` (``int``, default: config version):
Configuration version, used for ABI compatibility.
The C locale coercion (PEP 538) and the UTF-8 Mode (PEP 540) are
disabled by default in ``PyPreConfig``. Set ``coerce_c_locale``,
``coerce_c_locale_warn`` and ``utf8_mode`` to ``-1`` to let Python
enable them depending on the user configuration. In this case, it's
safer to explicitly pre-initialize Python to ensure that encodings are
configured before the Python initialization starts. Example to get the
same encoding than regular Python::
PyPreConfig preconfig = PyPreConfig_INIT;
preconfig.coerce_c_locale = -1;
preconfig.coerce_c_locale_warn = -1;
preconfig.utf8_mode = -1;
PyInitError err = Py_PreInitialize(&preconfig);
if (Py_INIT_FAILED(err)) {
Py_ExitInitError(err);
}
Initialization with PyConfig
----------------------------
The ``PyConfig`` structure contains all parameters to configure Python.
Example setting the program name::
PyInitError err;
PyConfig config = PyConfig_INIT;
err = PyConfig_SetString(&config.program_name, L"my_program");
if (_Py_INIT_FAILED(err)) {
Py_ExitInitError(err);
}
err = Py_InitializeFromConfig(&config);
PyConfig_Clear(&config);
if (Py_INIT_FAILED(err)) {
Py_ExitInitError(err);
}
``PyConfig`` methods:
* ``PyInitError PyConfig_SetString(PyConfig *config, wchar_t
**config_str, const wchar_t *str)``:
Copy the wide character string *str* into ``*config_str``.
* ``PyInitError PyConfig_SetBytesString(PyConfig *config, wchar_t
**config_str, const char *str)``:
Decode *str* using ``Py_DecodeLocale()`` and set the result into
``*config_str``. Pre-initialize Python if needed to ensure that
encodings are properly configured.
* ``PyInitError PyConfig_SetArgv(PyConfig *config, int argc, wchar_t **argv)``:
Set command line arguments from wide character strings.
* ``PyInitError PyConfig_SetBytesArgv(PyConfig *config, int argc, char
**argv)``:
Set command line arguments: decode bytes using ``Py_DecodeLocale()``.
Pre-initialize Python if needed to ensure that encodings are properly
configured.
* ``PyInitError PyConfig_Read(PyConfig *config)``:
Read all Python configuration. Fields which are already set are left
unchanged.
* ``void PyConfig_Clear(PyConfig *config)``:
Release configuration memory.
Functions to initialize Python:
* ``PyInitError Py_InitializeFromConfig(const PyConfig *config)``:
Initialize Python from *config* configuration. *config* can be
``NULL``.
The caller of these methods and functions is responsible to handle
failure or exit using ``Py_INIT_FAILED()`` and ``Py_ExitInitError()``.
``PyConfig`` fields:
* ``argv`` (``PyWideStringList``, default: empty):
Command line arguments, ``sys.argv``.
It is parsed and updated by default, set ``parse_argv`` to 0 to avoid
that.
* ``base_exec_prefix`` (``wchar_t*``, default: ``NULL``):
``sys.base_exec_prefix``.
* ``base_prefix`` (``wchar_t*``, default: ``NULL``):
``sys.base_prefix``.
* ``buffered_stdio`` (``int``, default: 1):
If equals to 0, enable unbuffered mode, make stdout and stderr streams
to be unbuffered.
* ``bytes_warning`` (``int``, default: 0):
If equals to 1, issue a warning when comparing ``bytes`` or
``bytearray`` with ``str``, or comparing ``bytes`` with ``int``. If
equal or greater to 2, raise a ``BytesWarning`` exception.
* ``check_hash_pycs_mode`` (``wchar_t*``, default: ``"default"``):
``--check-hash-based-pycs`` command line option value (see PEP 552).
* ``configure_c_stdio`` (``int``, default: 1):
If non-zero, configure C standard streams (``stdio``, ``stdout``,
``stdout``). For example, set their mode to ``O_BINARY`` on Windows.
* ``dev_mode`` (``int``, default: 0):
Development mode
* ``dll_path`` (``wchar_t*``, Windows only, default: ``NULL``):
Windows DLL path.
* ``dump_refs`` (``int``, default: 0):
If non-zero, dump all objects which are still alive at exit
* ``exec_prefix`` (``wchar_t*``, default: ``NULL``):
``sys.exec_prefix``.
* ``executable`` (``wchar_t*``, default: ``NULL``):
``sys.executable``.
* ``faulthandler`` (``int``, default: 0):
If non-zero, call ``faulthandler.enable()``.
* ``filesystem_encoding`` (``wchar_t*``, default: ``NULL``):
Filesystem encoding, ``sys.getfilesystemencoding()``.
* ``filesystem_errors`` (``wchar_t*``, default: ``NULL``):
Filesystem encoding errors, ``sys.getfilesystemencodeerrors()``.
* ``use_hash_seed`` (``int``, default: 0),
``hash_seed`` (``unsigned long``, default: 0):
Randomized hash function seed.
* ``home`` (``wchar_t*``, default: ``NULL``):
Python home directory.
* ``import_time`` (``int``, default: 0):
If non-zero, profile import time.
* ``inspect`` (``int``, default: 0):
Enter interactive mode after executing a script or a command.
* ``install_signal_handlers`` (``int``, default: 1):
Install signal handlers?
* ``interactive`` (``int``, default: 0):
Interactive mode.
* ``legacy_windows_stdio`` (``int``, Windows only, default: 0):
If non-zero, use ``io.FileIO`` instead of ``WindowsConsoleIO`` for
``sys.stdin``, ``sys.stdout`` and ``sys.stderr``.
* ``malloc_stats`` (``int``, default: 0):
If non-zero, dump memory allocation statistics at exit.
* ``module_search_path_env`` (``wchar_t*``, default: ``NULL``):
``PYTHONPATH`` environment variale value.
* ``use_module_search_paths`` (``int``, default: 0),
``module_search_paths`` (``PyWideStringList``, default: empty):
``sys.path``.
* ``optimization_level`` (``int``, default: 0):
Compilation optimization level.
* ``parse_argv`` (``int``, default: 1):
If non-zero, parse ``argv`` command line arguments and update
``argv``.
* ``parser_debug`` (``int``, default: 0):
If non-zero, turn on parser debugging output (for expert only,
depending on compilation options).
* ``pathconfig_warnings`` (``int``, default: 1):
If equal to 0, suppress warnings when computing the path
configuration.
* ``prefix`` (``wchar_t*``, default: ``NULL``):
``sys.prefix``.
* ``program_name`` (``wchar_t*``, default: ``NULL``):
Program name.
* ``program`` (``wchar_t*``, default: ``NULL``):
``argv[0]`` or an empty string.
* ``pycache_prefix`` (``wchar_t*``, default: ``NULL``):
``.pyc`` cache prefix.
* ``quiet`` (``int``, default: 0):
Quiet mode. For example, don't display the copyright and version
messages even in interactive mode.
* ``run_command`` (``wchar_t*``, default: ``NULL``):
``-c COMMAND`` argument.
* ``run_filename`` (``wchar_t*``), default: ``NULL``:
``python3 SCRIPT`` argument.
* ``run_module`` (``wchar_t*``, default: ``NULL``):
``python3 -m MODULE`` argument.
* ``show_alloc_count`` (``int``, default: 0):
Show allocation counts at exit?
* ``show_ref_count`` (``int``, default: 0):
Show total reference count at exit?
* ``site_import`` (``int``, default: 1):
Import the ``site`` module at startup?
* ``skip_source_first_line`` (``int``, default: 0):
Skip the first line of the source?
* ``stdio_encoding`` (``wchar_t*``, default: ``NULL``),
``stdio_errors`` (``wchar_t*``, default: ``NULL``):
Encoding and encoding errors of ``sys.stdin``, ``sys.stdout``
and ``sys.stderr``.
* ``tracemalloc`` (``int``, default: 0):
If non-zero, call ``tracemalloc.start(value)``.
* ``user_site_directory`` (``int``, default: 1):
If non-zero, add user site directory to ``sys.path``.
* ``verbose`` (``int``, default: 0):
If non-zero, enable verbose mode.
* ``warnoptions`` (``PyWideStringList``, default: empty):
Options of the ``warnings`` module to build warnings filters.
* ``write_bytecode`` (``int``, default: 1):
If non-zero, write ``.pyc`` files.
* ``xoptions`` (``PyWideStringList``, default: empty):
``sys._xoptions``.
``PyConfig`` private fields, for internal use only:
* ``_config_version`` (``int``, default: config version):
Configuration version, used for ABI compatibility.
* ``_install_importlib`` (``int``, default: 1):
Install importlib?
* ``_init_main`` (``int``, default: 1):
If equal to 0, stop Python initialization before the "main" phase
(see PEP 432).
By default, the ``argv`` arguments are parsed as regular Python command
line arguments and ``argv`` is updated to strip parsed Python arguments:
see `Command Line Arguments`_. Set ``parse_argv`` to 0 to avoid parsing
and updating ``argv``. If ``argv`` is empty, an empty string is added to
ensure that ``sys.argv`` always exists and is never empty.
The ``xoptions`` options are parsed to set other options: see `-X
Options`_.
More complete example modifying the configuration before calling
``PyConfig_Read()``, and then modify the read configuration::
PyInitError init_python(const char *program_name)
{
PyInitError err;
PyConfig config = PyConfig_INIT;
/* Set the program name before reading the configuraton
(decode byte string from the locale encoding) */
err = PyConfig_SetBytesString(&config.program_name,
program_name);
if (_Py_INIT_FAILED(err)) {
goto fail;
}
/* Read all configuration at once */
err = PyConfig_Read(&config);
if (_Py_INIT_FAILED(err)) {
goto fail;
}
/* Append our custom search path to sys.path */
err = PyWideStringList_Append(&config.module_search_paths,
L"/path/to/more/modules");
if (_Py_INIT_FAILED(err)) {
goto fail;
}
/* Override executable computed by PyConfig_Read() */
err = PyConfig_SetString(&config, &config.executable, L"my_executable");
if (_Py_INIT_FAILED(err)) {
goto fail;
}
err = Py_InitializeFromConfig(&config);
/* Py_InitializeFromConfig() copied config which must now be
cleared to release memory */
PyConfig_Clear(&config);
return err;
fail:
PyConfig_Clear(&config);
Py_ExitInitError(err);
}
.. note::
``PyConfig`` does not have any field for extra inittab functions:
``PyImport_AppendInittab()`` and ``PyImport_ExtendInittab()``
functions are still relevant (and can be called before Python
initialization).
Initialization with constant PyConfig
-------------------------------------
When no ``PyConfig`` method is used but only
``Py_InitializeFromConfig()``, the caller is responsible for managing
``PyConfig`` memory. In that case, constant strings and constant string
lists can be used to avoid dynamically allocated memory. It can be used
for most simple configurations.
Example of Python initialization enabling the isolated mode::
PyConfig config = PyConfig_INIT;
config.isolated = 1;
PyInitError err = Py_InitializeFromConfig(&config);
if (Py_INIT_FAILED(err)) {
Py_ExitInitError(err);
}
/* ... use Python API here ... */
Py_Finalize();
``PyConfig_Clear()`` is not needed in this example since ``config`` does
not contain any dynamically allocated string:
``Py_InitializeFromConfig`` is responsible to fill other fields and
manage the memory.
For convenience, two other functions are provided for constant
``PyConfig``:
* ``PyInitError Py_InitializeFromArgs(const PyConfig *config, int
argc, wchar_t **argv)``
* ``PyInitError Py_InitializeFromBytesArgs(const PyConfig *config, int
argc, char **argv)``
They be called with *config* set to ``NULL``. The caller of these
functions is responsible to handle failure or exit using
``Py_INIT_FAILED()`` and ``Py_ExitInitError()``.
Path Configuration
------------------
``PyConfig`` contains multiple fields for the path configuration:
* Path configuration input fields:
* ``home``
* ``module_search_path_env``
* ``pathconfig_warnings``
* Path configuration output fields:
* ``dll_path`` (Windows only)
* ``exec_prefix``
* ``executable``
* ``prefix``
* ``use_module_search_paths``, ``module_search_paths``
Set ``pathconfig_warnings`` to 0 to suppress warnings when computing the
path configuration.
It is possible to completely ignore the function computing the default
path configuration by setting explicitly all path configuration output
fields listed above. A string is considered as set even if it's an empty
string. ``module_search_paths`` is considered as set if
``use_module_search_paths`` is set to 1. In this case, path
configuration input fields are ignored as well.
If ``base_prefix`` or ``base_exec_prefix`` fields are not set, they
inherit their value from ``prefix`` and ``exec_prefix`` respectively.
If ``site_import`` is non-zero, ``sys.path`` can be modified by the
``site`` module. For example, if ``user_site_directory`` is non-zero,
the user site directory is added to ``sys.path`` (if it exists).
Isolate Python
--------------
The default configuration is designed to behave as a regular Python.
To embed Python into an application, it's possible to tune the
configuration to better isolated the embedded Python from the system:
* Set ``isolated`` to 1 to ignore environment variables and not prepend
the current directory to ``sys.path``.
* Set the `Path Configuration`_ ("output fields") to ignore the function
computing the default path configuration.
Py_BytesMain()
--------------
Python 3.7 provides a high-level ``Py_Main()`` function which requires
to pass command line arguments as ``wchar_t*`` strings. It is
non-trivial to use the correct encoding to decode bytes. Python has its
own set of issues with C locale coercion and UTF-8 Mode.
This PEP adds a new ``Py_BytesMain()`` function which takes command line
arguments as bytes::
int Py_BytesMain(int argc, char **argv)
Py_RunMain()
------------
The new ``Py_RunMain()`` function executes the command
(``PyConfig.run_command``), the script (``PyConfig.run_filename``) or
the module (``PyConfig.run_module``) specified on the command line or in
the configuration, and then finalizes Python. It returns an exit status
that can be passed to the ``exit()`` function.
Example of customized Python in isolated mode::
#include <Python.h>
int main(int argc, char *argv[])
{
PyConfig config = PyConfig_INIT;
config.isolated = 1;
PyInitError err = Py_InitializeFromBytesArgs(&config, argc, argv);
if (Py_INIT_FAILED(err)) {
Py_ExitInitError(err);
}
/* put more configuration code here if needed */
return Py_RunMain();
}
The example is a basic implementation of the "System Python Executable"
discussed in PEP 432.
Memory allocations and Py_DecodeLocale()
----------------------------------------
Python memory allocation functions like ``PyMem_RawMalloc()`` must not
be used before Python pre-initialization, whereas calling directly
``malloc()`` and ``free()`` is always safe.
For ``PyPreConfig`` and constant ``PyConfig``, the caller is responsible
to manage dynamically allocated memory; constant strings and constant
string lists can be used to avoid memory allocations.
Dynamic ``PyConfig`` requires to call ``PyConfig_Clear()`` to release
memory.
``Py_DecodeLocale()`` must not be called before the pre-initialization.
Backwards Compatibility
=======================
This PEP only adds a new API: it leaves the existing API unchanged and
has no impact on the backwards compatibility.
The implementation ensures that the existing API is compatible with the
new API. For example, ``PyConfig`` uses the value of global
configuration variables as default values.
Annex: Python Configuration
===========================
Priority and Rules
------------------
Priority of configuration parameters, highest to lowest:
* ``PyConfig``
* ``PyPreConfig``
* Configuration files
* Command line options
* Environment variables
* Global configuration variables
Priority of warning options, highest to lowest:
* ``PyConfig.warnoptions``
* ``PyConfig.dev_mode`` (add ``"default"``)
* ``PYTHONWARNINGS`` environment variables
* ``-W WARNOPTION`` command line argument
* ``PyConfig.bytes_warning`` (add ``"error::BytesWarning"`` if greater
than 1, or add ``"default::BytesWarning``)
Rules on ``PyConfig`` parameters:
* If ``isolated`` is non-zero, ``use_environment`` and
``user_site_directory`` are set to 0.
* If ``legacy_windows_fs_encoding`` is non-zero, ``utf8_mode`` is set to
0.
* If ``dev_mode`` is non-zero, ``allocator`` is set to ``"debug"``,
``faulthandler`` is set to 1, and ``"default"`` filter is added to
``warnoptions``. But the ``PYTHONMALLOC`` environment variable has the
priority over ``dev_mode`` to set the memory allocator.
* If ``base_prefix`` is not set, it inherits ``prefix`` value.
* If ``base_exec_prefix`` is not set, it inherits ``exec_prefix`` value.
* If the ``python._pth`` configuration file is present, ``isolated`` is
set to 1 and ``site_import`` is set to 0; but ``site_import`` is set
to 1 if ``python._pth`` contains ``import site``.
Rules on ``PyConfig`` and ``PyPreConfig`` parameters:
* If ``PyPreConfig.legacy_windows_fs_encoding`` is non-zero,
set ``PyConfig.utf8_mode`` to 0, set ``PyConfig.filesystem_encoding``
to ``mbcs``, and set ``PyConfig.filesystem_errors`` to ``replace``.
Configuration Files
-------------------
Python configuration files:
* ``pyvenv.cfg``
* ``python._pth`` (Windows only)
* ``pybuilddir.txt`` (Unix only)
Global Configuration Variables
------------------------------
Global configuration variables mapped to ``PyPreConfig`` fields:
======================================== ================================
Variable Field
======================================== ================================
``Py_IgnoreEnvironmentFlag`` ``use_environment`` (NOT)
``Py_IsolatedFlag`` ``isolated``
``Py_LegacyWindowsFSEncodingFlag`` ``legacy_windows_fs_encoding``
``Py_UTF8Mode`` ``utf8_mode``
======================================== ================================
(NOT) means that the ``PyPreConfig`` value is the oposite of the global
configuration variable value.
Global configuration variables mapped to ``PyConfig`` fields:
======================================== ================================
Variable Field
======================================== ================================
``Py_BytesWarningFlag`` ``bytes_warning``
``Py_DebugFlag`` ``parser_debug``
``Py_DontWriteBytecodeFlag`` ``write_bytecode`` (NOT)
``Py_FileSystemDefaultEncodeErrors`` ``filesystem_errors``
``Py_FileSystemDefaultEncoding`` ``filesystem_encoding``
``Py_FrozenFlag`` ``pathconfig_warnings`` (NOT)
``Py_HasFileSystemDefaultEncoding`` ``filesystem_encoding``
``Py_HashRandomizationFlag`` ``use_hash_seed``, ``hash_seed``
``Py_IgnoreEnvironmentFlag`` ``use_environment`` (NOT)
``Py_InspectFlag`` ``inspect``
``Py_InteractiveFlag`` ``interactive``
``Py_IsolatedFlag`` ``isolated``
``Py_LegacyWindowsStdioFlag`` ``legacy_windows_stdio``
``Py_NoSiteFlag`` ``site_import`` (NOT)
``Py_NoUserSiteDirectory`` ``user_site_directory`` (NOT)
``Py_OptimizeFlag`` ``optimization_level``
``Py_QuietFlag`` ``quiet``
``Py_UnbufferedStdioFlag`` ``buffered_stdio`` (NOT)
``Py_VerboseFlag`` ``verbose``
``_Py_HasFileSystemDefaultEncodeErrors`` ``filesystem_errors``
======================================== ================================
(NOT) means that the ``PyConfig`` value is the oposite of the global
configuration variable value.
``Py_LegacyWindowsFSEncodingFlag`` and ``Py_LegacyWindowsStdioFlag`` are
only available on Windows.
Command Line Arguments
----------------------
Usage::
python3 [options]
python3 [options] -c COMMAND
python3 [options] -m MODULE
python3 [options] SCRIPT
Command line options mapped to pseudo-action on ``PyPreConfig`` fields:
================================ ================================
Option ``PyConfig`` field
================================ ================================
``-E`` ``use_environment = 0``
``-I`` ``isolated = 1``
``-X dev`` ``dev_mode = 1``
``-X utf8`` ``utf8_mode = 1``
``-X utf8=VALUE`` ``utf8_mode = VALUE``
================================ ================================
Command line options mapped to pseudo-action on ``PyConfig`` fields:
================================ ================================
Option ``PyConfig`` field
================================ ================================
``-b`` ``bytes_warning++``
``-B`` ``write_bytecode = 0``
``-c COMMAND`` ``run_command = COMMAND``
``--check-hash-based-pycs=MODE`` ``_check_hash_pycs_mode = MODE``
``-d`` ``parser_debug++``
``-E`` ``use_environment = 0``
``-i`` ``inspect++`` and ``interactive++``
``-I`` ``isolated = 1``
``-m MODULE`` ``run_module = MODULE``
``-O`` ``optimization_level++``
``-q`` ``quiet++``
``-R`` ``use_hash_seed = 0``
``-s`` ``user_site_directory = 0``
``-S`` ``site_import``
``-t`` ignored (kept for backwards compatibility)
``-u`` ``buffered_stdio = 0``
``-v`` ``verbose++``
``-W WARNING`` add ``WARNING`` to ``warnoptions``
``-x`` ``skip_source_first_line = 1``
``-X OPTION`` add ``OPTION`` to ``xoptions``
================================ ================================
``-h``, ``-?`` and ``-V`` options are handled without ``PyConfig``.
-X Options
----------
-X options mapped to pseudo-action on ``PyConfig`` fields:
================================ ================================
Option ``PyConfig`` field
================================ ================================
``-X dev`` ``dev_mode = 1``
``-X faulthandler`` ``faulthandler = 1``
``-X importtime`` ``import_time = 1``
``-X pycache_prefix=PREFIX`` ``pycache_prefix = PREFIX``
``-X showalloccount`` ``show_alloc_count = 1``
``-X showrefcount`` ``show_ref_count = 1``
``-X tracemalloc=N`` ``tracemalloc = N``
================================ ================================
Environment Variables
---------------------
Environment variables mapped to ``PyPreConfig`` fields:
================================= =============================================
Variable ``PyPreConfig`` field
================================= =============================================
``PYTHONCOERCECLOCALE`` ``coerce_c_locale``, ``coerce_c_locale_warn``
``PYTHONDEVMODE`` ``dev_mode``
``PYTHONLEGACYWINDOWSFSENCODING`` ``legacy_windows_fs_encoding``
``PYTHONMALLOC`` ``allocator``
``PYTHONUTF8`` ``utf8_mode``
================================= =============================================
Environment variables mapped to ``PyConfig`` fields:
================================= ====================================
Variable ``PyConfig`` field
================================= ====================================
``PYTHONDEBUG`` ``parser_debug``
``PYTHONDEVMODE`` ``dev_mode``
``PYTHONDONTWRITEBYTECODE`` ``write_bytecode``
``PYTHONDUMPREFS`` ``dump_refs``
``PYTHONEXECUTABLE`` ``program_name``
``PYTHONFAULTHANDLER`` ``faulthandler``
``PYTHONHASHSEED`` ``use_hash_seed``, ``hash_seed``
``PYTHONHOME`` ``home``
``PYTHONINSPECT`` ``inspect``
``PYTHONIOENCODING`` ``stdio_encoding``, ``stdio_errors``
``PYTHONLEGACYWINDOWSSTDIO`` ``legacy_windows_stdio``
``PYTHONMALLOCSTATS`` ``malloc_stats``
``PYTHONNOUSERSITE`` ``user_site_directory``
``PYTHONOPTIMIZE`` ``optimization_level``
``PYTHONPATH`` ``module_search_path_env``
``PYTHONPROFILEIMPORTTIME`` ``import_time``
``PYTHONPYCACHEPREFIX,`` ``pycache_prefix``
``PYTHONTRACEMALLOC`` ``tracemalloc``
``PYTHONUNBUFFERED`` ``buffered_stdio``
``PYTHONVERBOSE`` ``verbose``
``PYTHONWARNINGS`` ``warnoptions``
================================= ====================================
``PYTHONLEGACYWINDOWSFSENCODING`` and ``PYTHONLEGACYWINDOWSSTDIO`` are
specific to Windows.
Annex: Python 3.7 API
=====================
Python 3.7 has 4 functions in its C API to initialize and finalize
Python:
* ``Py_Initialize()``, ``Py_InitializeEx()``: initialize Python
* ``Py_Finalize()``, ``Py_FinalizeEx()``: finalize Python
Python 3.7 can be configured using `Global Configuration Variables`_,
`Environment Variables`_, and the following functions:
* ``PyImport_AppendInittab()``
* ``PyImport_ExtendInittab()``
* ``PyMem_SetAllocator()``
* ``PyMem_SetupDebugHooks()``
* ``PyObject_SetArenaAllocator()``
* ``Py_SetPath()``
* ``Py_SetProgramName()``
* ``Py_SetPythonHome()``
* ``Py_SetStandardStreamEncoding()``
* ``PySys_AddWarnOption()``
* ``PySys_AddXOption()``
* ``PySys_ResetWarnOptions()``
There is also a high-level ``Py_Main()`` function.
Python Issues
=============
Issues that will be fixed by this PEP, directly or indirectly:
* `bpo-1195571 <https://bugs.python.org/issue1195571>`_: "simple
callback system for Py_FatalError"
* `bpo-11320 <https://bugs.python.org/issue11320>`_:
"Usage of API method Py_SetPath causes errors in Py_Initialize()
(Posix ony)"
* `bpo-13533 <https://bugs.python.org/issue13533>`_: "Would like
Py_Initialize to play friendly with host app"
* `bpo-14956 <https://bugs.python.org/issue14956>`_: "custom PYTHONPATH
may break apps embedding Python"
* `bpo-19983 <https://bugs.python.org/issue19983>`_: "When interrupted
during startup, Python should not call abort() but exit()"
* `bpo-22213 <https://bugs.python.org/issue22213>`_: "Make pyvenv style
virtual environments easier to configure when embedding Python". This
PEP more or
* `bpo-22257 <https://bugs.python.org/issue22257>`_: "PEP 432: Redesign
the interpreter startup sequence"
* `bpo-29778 <https://bugs.python.org/issue29778>`_: "_Py_CheckPython3
uses uninitialized dllpath when embedder sets module path with
Py_SetPath"
* `bpo-30560 <https://bugs.python.org/issue30560>`_: "Add
Py_SetFatalErrorAbortFunc: Allow embedding program to handle fatal
errors".
* `bpo-31745 <https://bugs.python.org/issue31745>`_: "Overloading
"Py_GetPath" does not work"
* `bpo-32573 <https://bugs.python.org/issue32573>`_: "All sys attributes
(.argv, ...) should exist in embedded environments".
* `bpo-34725 <https://bugs.python.org/issue34725>`_:
"Py_GetProgramFullPath() odd behaviour in Windows"
* `bpo-36204 <https://bugs.python.org/issue36204>`_: "Deprecate calling
Py_Main() after Py_Initialize()? Add Py_InitializeFromArgv()?"
* `bpo-33135 <https://bugs.python.org/issue33135>`_: "Define field
prefixes for the various config structs". The PEP now defines well
how warnings options are handled.
Issues of the PEP implementation:
* `bpo-16961 <https://bugs.python.org/issue16961>`_: "No regression
tests for -E and individual environment vars"
* `bpo-20361 <https://bugs.python.org/issue20361>`_: "-W command line
options and PYTHONWARNINGS environmental variable should not override
-b / -bb command line options"
* `bpo-26122 <https://bugs.python.org/issue26122>`_: "Isolated mode
doesn't ignore PYTHONHASHSEED"
* `bpo-29818 <https://bugs.python.org/issue29818>`_:
"Py_SetStandardStreamEncoding leads to a memory error in debug mode"
* `bpo-31845 <https://bugs.python.org/issue31845>`_:
"PYTHONDONTWRITEBYTECODE and PYTHONOPTIMIZE have no effect"
* `bpo-32030 <https://bugs.python.org/issue32030>`_: "PEP 432: Rewrite
Py_Main()"
* `bpo-32124 <https://bugs.python.org/issue32124>`_: "Document functions
safe to be called before Py_Initialize()"
* `bpo-33042 <https://bugs.python.org/issue33042>`_: "New 3.7 startup
sequence crashes PyInstaller"
* `bpo-33932 <https://bugs.python.org/issue33932>`_: "Calling
Py_Initialize() twice now triggers a fatal error (Python 3.7)"
* `bpo-34008 <https://bugs.python.org/issue34008>`_: "Do we support
calling Py_Main() after Py_Initialize()?"
* `bpo-34170 <https://bugs.python.org/issue34170>`_: "Py_Initialize():
computing path configuration must not have side effect (PEP 432)"
* `bpo-34589 <https://bugs.python.org/issue34589>`_: "Py_Initialize()
and Py_Main() should not enable C locale coercion"
* `bpo-34639 <https://bugs.python.org/issue34639>`_:
"PYTHONCOERCECLOCALE is ignored when using -E or -I option"
* `bpo-36142 <https://bugs.python.org/issue36142>`_: "Add a new
_PyPreConfig step to Python initialization to setup memory allocator
and encodings"
* `bpo-36202 <https://bugs.python.org/issue36202>`_: "Calling
Py_DecodeLocale() before _PyPreConfig_Write() can produce mojibake"
* `bpo-36301 <https://bugs.python.org/issue36301>`_: "Add
_Py_PreInitialize() function"
* `bpo-36443 <https://bugs.python.org/issue36443>`_: "Disable
coerce_c_locale and utf8_mode by default in _PyPreConfig?"
* `bpo-36444 <https://bugs.python.org/issue36444>`_: "Python
initialization: remove _PyMainInterpreterConfig"
* `bpo-36471 <https://bugs.python.org/issue36471>`_: "PEP 432, PEP 587:
Add _Py_RunMain()"
* `bpo-36763 <https://bugs.python.org/issue36763>`_: "PEP 587: Rework
initialization API to prepare second version of the PEP"
* `bpo-36775 <https://bugs.python.org/issue36775>`_: "Rework filesystem
codec implementation"
* `bpo-36900 <https://bugs.python.org/issue36900>`_: "Use _PyCoreConfig
rather than global configuration variables"
Issues related to this PEP:
* `bpo-12598 <https://bugs.python.org/issue12598>`_: "Move sys variable
initialization from import.c to sysmodule.c"
* `bpo-15577 <https://bugs.python.org/issue15577>`_: "Real argc and argv
in embedded interpreter"
* `bpo-16202 <https://bugs.python.org/issue16202>`_: "sys.path[0]
security issues"
* `bpo-18309 <https://bugs.python.org/issue18309>`_: "Make python
slightly more relocatable"
* `bpo-25631 <https://bugs.python.org/issue25631>`_: "Segmentation fault
with invalid Unicode command-line arguments in embedded Python"
* `bpo-26007 <https://bugs.python.org/issue26007>`_: "Support embedding
the standard library in an executable"
* `bpo-31210 <https://bugs.python.org/issue31210>`_: "Can not import
modules if sys.prefix contains DELIM".
* `bpo-31349 <https://bugs.python.org/issue31349>`_: "Embedded
initialization ignores Py_SetProgramName()"
* `bpo-33919 <https://bugs.python.org/issue33919>`_: "Expose
_PyCoreConfig structure to Python"
* `bpo-35173 <https://bugs.python.org/issue35173>`_: "Re-use already
existing functionality to allow Python 2.7.x (both embedded and
standalone) to locate the module path according to the shared library"
Version History
===============
* Version 3:
* ``PyConfig``: Add ``configure_c_stdio`` and ``parse_argv``,
rename ``_frozen`` to ``pathconfig_warnings``.
* Rename functions using bytes strings and wide character strings. For
example, ``Py_PreInitializeFromWideArgs`` becomes
``Py_PreInitializeFromArgs``, and ``PyConfig_SetArgv`` becomes
``PyConfig_SetBytesArgv``.
* Add ``PyWideStringList_Insert()`` function.
* New "Path configuration", "Isolate Python", "Python Issues"
and "Version History" sections.
* ``PyConfig_SetString()`` and ``PyConfig_SetBytesString()`` now
requires the configuration as the first argument.
* Rename ``Py_UnixMain()`` to ``Py_BytesMain()``
* Version 2: Add ``PyConfig`` methods (ex: ``PyConfig_Read()``), add
``PyWideStringList_Append()``, rename ``PyWideCharList`` to
``PyWideStringList``.
* Version 1: Initial version.
Copyright
=========
This document has been placed in the public domain.
[View Less]
5
11
ACTIVITY SUMMARY (2019-05-24 - 2019-05-31)
Python tracker at https://bugs.python.org/
To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.
Issues counts and deltas:
open 7028 (-39)
closed 41839 (+118)
total 48867 (+79)
Open issues with patches: 2820
Issues opened (50)
==================
#26423: Integer overflow in wrap_lenfunc() on 64-bit build of Windows
https://bugs.python.org/issue26423 reopened by vstinner
#34725: …
[View More]Py_GetProgramFullPath() odd behaviour in Windows
https://bugs.python.org/issue34725 reopened by vstinner
#35753: Importing call from unittest.mock directly causes ValueError
https://bugs.python.org/issue35753 reopened by pablogsal
#36889: Merge StreamWriter and StreamReader into just asyncio.Stream
https://bugs.python.org/issue36889 reopened by koobs
#36991: zipfile: AttributeError on extract
https://bugs.python.org/issue36991 reopened by berker.peksag
#37038: Make idlelib/run.py runnable.
https://bugs.python.org/issue37038 opened by terry.reedy
#37039: IDLE: Improve zoomheight doc and behavior.
https://bugs.python.org/issue37039 opened by terry.reedy
#37040: checking for membership in itertools.count enters infinite loo
https://bugs.python.org/issue37040 opened by Dan Rose
#37041: IDLE: path browser unusable on some displays
https://bugs.python.org/issue37041 opened by aroberge
#37042: wait_for(coro, timeout=0) memleak
https://bugs.python.org/issue37042 opened by ixje
#37043: Buildbots fail when new files are added
https://bugs.python.org/issue37043 opened by jaraco
#37044: Build/test artifacts not ignored for framework build
https://bugs.python.org/issue37044 opened by jaraco
#37048: ssl module: QUIC support for HTTP/3
https://bugs.python.org/issue37048 opened by christian.heimes
#37052: Add examples for mocking async for and async context manager i
https://bugs.python.org/issue37052 opened by xtreak
#37055: Numerous warnings with blake2 module
https://bugs.python.org/issue37055 opened by remi.lapeyre
#37060: import ctypes fails with a statically linked interpreter due t
https://bugs.python.org/issue37060 opened by indygreg
#37062: `AutoNumber` class in enum documentation: support *args in con
https://bugs.python.org/issue37062 opened by rrt
#37063: Incorrect application of func.__defaults__ by inspect's signat
https://bugs.python.org/issue37063 opened by bup
#37064: Feature request: option to keep/add flags to pathfix.
https://bugs.python.org/issue37064 opened by pkopkan
#37068: Emit SyntaxWarning for f-strings without expressions ?
https://bugs.python.org/issue37068 opened by mbussonn
#37069: regrtest: log unraisable exceptions and uncaught thread except
https://bugs.python.org/issue37069 opened by vstinner
#37071: HTMLParser mistakenly inventing new tags while parsing
https://bugs.python.org/issue37071 opened by htran
#37073: clarify functions docs in IO modules and Bytes Objects
https://bugs.python.org/issue37073 opened by Windson Yang
#37074: os.stat() does not work for NUL and CON
https://bugs.python.org/issue37074 opened by serhiy.storchaka
#37075: Error message improvement for AsyncMock
https://bugs.python.org/issue37075 opened by xtreak
#37077: Threading: add builtin TID attribute to Thread objects for AIX
https://bugs.python.org/issue37077 opened by Michael.Felt
#37080: Cannot compile Python3.7.3 on Alt-F (ARM)
https://bugs.python.org/issue37080 opened by Jarl Gullberg
#37081: Test with OpenSSL 1.1.1c
https://bugs.python.org/issue37081 opened by christian.heimes
#37082: Assignment expression operator (walrus) not in built-in help()
https://bugs.python.org/issue37082 opened by mbussonn
#37083: Document TYPE_COMMENT in documentation reference for compound
https://bugs.python.org/issue37083 opened by pablogsal
#37084: _ctypes not failing, can't find reason
https://bugs.python.org/issue37084 opened by Brian Spratke
#37085: Expose additional socket constants for CAN_BCM flags
https://bugs.python.org/issue37085 opened by karlding
#37086: time.sleep error message misleading
https://bugs.python.org/issue37086 opened by Justin Fay
#37087: Adding native id support for openbsd
https://bugs.python.org/issue37087 opened by David Carlier
#37088: Add a way to schedule a function to be called from the main th
https://bugs.python.org/issue37088 opened by yselivanov
#37089: `import Lib.os` works on windows (but shouldn't)
https://bugs.python.org/issue37089 opened by Anthony Sottile
#37090: test_gdb's test_pycfunction should test all calling convention
https://bugs.python.org/issue37090 opened by petr.viktorin
#37091: subprocess - uncaught PermissionError in send_signal can cause
https://bugs.python.org/issue37091 opened by wesinator
#37093: http.client aborts header parsing upon encountering non-ASCII
https://bugs.python.org/issue37093 opened by tburke
#37095: [Feature Request]: Add zstd support in tarfile
https://bugs.python.org/issue37095 opened by evan0greenup
#37096: Add large-file tests for modules using sendfile(2)
https://bugs.python.org/issue37096 opened by giampaolo.rodola
#37097: python_is_optimized() false negatives
https://bugs.python.org/issue37097 opened by jdemeyer
#37098: test_memfd_create() test failure
https://bugs.python.org/issue37098 opened by jdemeyer
#37100: test_coroutine.test_unawaited_warning_when_module_broken fails
https://bugs.python.org/issue37100 opened by xtreak
#37102: Automatically dedent docstring constants by default
https://bugs.python.org/issue37102 opened by gregory.p.smith
#37103: Undo deprecation of optparse
https://bugs.python.org/issue37103 opened by andersk
#37105: Add deprecated-remove information on stream doc
https://bugs.python.org/issue37105 opened by eamanu
#37107: ensurepip --upgrade doesn't change the version of pip used by
https://bugs.python.org/issue37107 opened by cjw296
#37110: Clarify hashability of custom class instances
https://bugs.python.org/issue37110 opened by cito
#37116: Use PEP 570 syntax for positional-only parameters
https://bugs.python.org/issue37116 opened by serhiy.storchaka
Most recent 15 issues with no replies (15)
==========================================
#37116: Use PEP 570 syntax for positional-only parameters
https://bugs.python.org/issue37116
#37110: Clarify hashability of custom class instances
https://bugs.python.org/issue37110
#37107: ensurepip --upgrade doesn't change the version of pip used by
https://bugs.python.org/issue37107
#37105: Add deprecated-remove information on stream doc
https://bugs.python.org/issue37105
#37102: Automatically dedent docstring constants by default
https://bugs.python.org/issue37102
#37097: python_is_optimized() false negatives
https://bugs.python.org/issue37097
#37096: Add large-file tests for modules using sendfile(2)
https://bugs.python.org/issue37096
#37095: [Feature Request]: Add zstd support in tarfile
https://bugs.python.org/issue37095
#37093: http.client aborts header parsing upon encountering non-ASCII
https://bugs.python.org/issue37093
#37090: test_gdb's test_pycfunction should test all calling convention
https://bugs.python.org/issue37090
#37087: Adding native id support for openbsd
https://bugs.python.org/issue37087
#37085: Expose additional socket constants for CAN_BCM flags
https://bugs.python.org/issue37085
#37084: _ctypes not failing, can't find reason
https://bugs.python.org/issue37084
#37080: Cannot compile Python3.7.3 on Alt-F (ARM)
https://bugs.python.org/issue37080
#37077: Threading: add builtin TID attribute to Thread objects for AIX
https://bugs.python.org/issue37077
Most recent 15 issues waiting for review (15)
=============================================
#37116: Use PEP 570 syntax for positional-only parameters
https://bugs.python.org/issue37116
#37105: Add deprecated-remove information on stream doc
https://bugs.python.org/issue37105
#37098: test_memfd_create() test failure
https://bugs.python.org/issue37098
#37096: Add large-file tests for modules using sendfile(2)
https://bugs.python.org/issue37096
#37090: test_gdb's test_pycfunction should test all calling convention
https://bugs.python.org/issue37090
#37088: Add a way to schedule a function to be called from the main th
https://bugs.python.org/issue37088
#37085: Expose additional socket constants for CAN_BCM flags
https://bugs.python.org/issue37085
#37081: Test with OpenSSL 1.1.1c
https://bugs.python.org/issue37081
#37077: Threading: add builtin TID attribute to Thread objects for AIX
https://bugs.python.org/issue37077
#37075: Error message improvement for AsyncMock
https://bugs.python.org/issue37075
#37064: Feature request: option to keep/add flags to pathfix.
https://bugs.python.org/issue37064
#37039: IDLE: Improve zoomheight doc and behavior.
https://bugs.python.org/issue37039
#37038: Make idlelib/run.py runnable.
https://bugs.python.org/issue37038
#37034: Argument Clinic omits name of keyword-only parameter on _PyArg
https://bugs.python.org/issue37034
#37030: Lib/cmd.py: Hide undocumented commands in help and completenam
https://bugs.python.org/issue37030
Top 10 most discussed issues (10)
=================================
#37039: IDLE: Improve zoomheight doc and behavior.
https://bugs.python.org/issue37039 20 msgs
#33725: Python crashes on macOS after fork with no exec
https://bugs.python.org/issue33725 16 msgs
#36839: Support the buffer protocol in code objects
https://bugs.python.org/issue36839 12 msgs
#36974: Implement PEP 590
https://bugs.python.org/issue36974 12 msgs
#35545: asyncio.base_events.create_connection doesn't handle scoped IP
https://bugs.python.org/issue35545 9 msgs
#36709: Asyncio SSL keep-alive connections raise errors after loop clo
https://bugs.python.org/issue36709 9 msgs
#36889: Merge StreamWriter and StreamReader into just asyncio.Stream
https://bugs.python.org/issue36889 8 msgs
#37011: pdb: restore original tracing function instead of sys.settrace
https://bugs.python.org/issue37011 8 msgs
#26423: Integer overflow in wrap_lenfunc() on 64-bit build of Windows
https://bugs.python.org/issue26423 6 msgs
#28866: Type cache is not correctly invalidated on a class defining mr
https://bugs.python.org/issue28866 6 msgs
Issues closed (115)
===================
#5028: tokenize.generate_tokens doesn't always return logical line
https://bugs.python.org/issue5028 closed by cheryl.sabella
#11320: Usage of API method Py_SetPath causes errors in Py_Initialize(
https://bugs.python.org/issue11320 closed by vstinner
#13533: Would like Py_Initialize to play friendly with host app
https://bugs.python.org/issue13533 closed by vstinner
#14656: Add a macro for unreachable code
https://bugs.python.org/issue14656 closed by pablogsal
#14956: custom PYTHONPATH may break apps embedding Python
https://bugs.python.org/issue14956 closed by vstinner
#19569: Use __attribute__((deprecated)) to warn usage of deprecated fu
https://bugs.python.org/issue19569 closed by vstinner
#19983: When interrupted during startup, Python should not call abort(
https://bugs.python.org/issue19983 closed by vstinner
#22102: Zipfile generates Zipfile error in zip with 0 total number of
https://bugs.python.org/issue22102 closed by cheryl.sabella
#22454: Adding the opposite function of shlex.split()
https://bugs.python.org/issue22454 closed by berker.peksag
#22640: Add silent mode for py_compile
https://bugs.python.org/issue22640 closed by berker.peksag
#24564: shutil.copytree fails when copying NFS to NFS
https://bugs.python.org/issue24564 closed by giampaolo.rodola
#24732: 3.5.0b3 Windows accept() on unready non-blocking socket raises
https://bugs.python.org/issue24732 closed by vstinner
#25735: math.factorial doc should mention integer return type
https://bugs.python.org/issue25735 closed by cheryl.sabella
#27639: UserList.__getitem__ doesn't account for slices
https://bugs.python.org/issue27639 closed by SilentGhost
#27987: obmalloc's 8-byte alignment causes undefined behavior
https://bugs.python.org/issue27987 closed by inada.naoki
#29143: Logger should ignore propagate property for disabled handlers.
https://bugs.python.org/issue29143 closed by mariocj89
#29262: Provide a way to check for *real* typing.Union instances
https://bugs.python.org/issue29262 closed by levkivskyi
#29696: Use namedtuple in string.Formatter.parse iterator response
https://bugs.python.org/issue29696 closed by rhettinger
#29778: _Py_CheckPython3 uses uninitialized dllpath when embedder sets
https://bugs.python.org/issue29778 closed by vstinner
#29883: asyncio: Windows Proactor Event Loop UDP Support
https://bugs.python.org/issue29883 closed by asvetlov
#30006: Deadlocks in `concurrent.futures.ProcessPoolExecutor`
https://bugs.python.org/issue30006 closed by pitrou
#30560: Py_SetFatalErrorAbortFunc: Allow embedding program to handle f
https://bugs.python.org/issue30560 closed by vstinner
#30969: Docs should say that `x is z or x == z` is used for `x in y` i
https://bugs.python.org/issue30969 closed by cheryl.sabella
#31745: Overloading "Py_GetPath" does not work
https://bugs.python.org/issue31745 closed by vstinner
#32299: unittest.mock.patch.dict.__enter__ should return the dict
https://bugs.python.org/issue32299 closed by cheryl.sabella
#32388: Remove cross-version binary compatibility
https://bugs.python.org/issue32388 closed by pitrou
#32941: mmap should expose madvise()
https://bugs.python.org/issue32941 closed by pitrou
#33006: docstring of filter function is incorrect
https://bugs.python.org/issue33006 closed by ned.deily
#33135: Define field prefixes for the various config structs
https://bugs.python.org/issue33135 closed by vstinner
#33407: Implement Py_DEPRECATED() macro for Visual Studio
https://bugs.python.org/issue33407 closed by vstinner
#33918: Hooking into pause/resume of iterators/coroutines
https://bugs.python.org/issue33918 closed by asvetlov
#33919: Expose _PyCoreConfig structure to Python
https://bugs.python.org/issue33919 closed by vstinner
#34148: Fatal read error on socket transport
https://bugs.python.org/issue34148 closed by asvetlov
#34613: asyncio.StreamReader initialization documentation incorrectly
https://bugs.python.org/issue34613 closed by asvetlov
#34626: PEP 384's PyType_Spec and PyType_Slot are not documented
https://bugs.python.org/issue34626 closed by petr.viktorin
#34632: Port importlib_metadata to Python 3.8
https://bugs.python.org/issue34632 closed by barry
#34655: Support sendfile in asyncio streams API
https://bugs.python.org/issue34655 closed by asvetlov
#34791: xml package does not obey sys.flags.ignore_environment
https://bugs.python.org/issue34791 closed by vstinner
#34818: test.test_ssl.ThreadedTests.test_tls1_3 fails in 2.7 with Attr
https://bugs.python.org/issue34818 closed by cheryl.sabella
#34975: start_tls() difficult when using asyncio.start_server()
https://bugs.python.org/issue34975 closed by asvetlov
#34993: asyncio.streams.FlowControlMixin should be part of the API
https://bugs.python.org/issue34993 closed by asvetlov
#35246: asyncio.create_subprocess_exec doesn't accept pathlib.Path lik
https://bugs.python.org/issue35246 closed by asvetlov
#35279: asyncio uses too many threads by default
https://bugs.python.org/issue35279 closed by inada.naoki
#35397: Undeprecate and document urllib.parse.unwrap
https://bugs.python.org/issue35397 closed by cheryl.sabella
#35925: test_httplib test_nntplib test_ssl fail on ARMv7 Debian buster
https://bugs.python.org/issue35925 closed by ned.deily
#36204: Deprecate calling Py_Main() after Py_Initialize()? Add Py_Init
https://bugs.python.org/issue36204 closed by vstinner
#36271: '_io.TextIOWrapper' object has no attribute 'mode'
https://bugs.python.org/issue36271 closed by berker.peksag
#36540: PEP 570: Python Positional-Only Parameters
https://bugs.python.org/issue36540 closed by pablogsal
#36610: os.sendfile can return EINVAL on Solaris
https://bugs.python.org/issue36610 closed by giampaolo.rodola
#36686: Docs: asyncio.loop.subprocess_exec documentation is confusing,
https://bugs.python.org/issue36686 closed by sbstp
#36721: Add pkg-config python-3.8-embed and --embed to python3.8-confi
https://bugs.python.org/issue36721 closed by vstinner
#36739: "4.6. Defining Functions" should mention nonlocal
https://bugs.python.org/issue36739 closed by Mariatta
#36772: Let lru_cache be used as a decorator with no arguments
https://bugs.python.org/issue36772 closed by rhettinger
#36782: Add tests for the datetime C API
https://bugs.python.org/issue36782 closed by vstinner
#36794: asyncio.Lock documentation in Py3.8 lacks parts presented in d
https://bugs.python.org/issue36794 closed by asvetlov
#36809: Crash for test test_importlib
https://bugs.python.org/issue36809 closed by vstinner
#36829: Add sys.unraisablehook() to custom how "unraisable exceptions"
https://bugs.python.org/issue36829 closed by vstinner
#36840: Add stream.abort() async method
https://bugs.python.org/issue36840 closed by asvetlov
#36842: Implement PEP 578
https://bugs.python.org/issue36842 closed by steve.dower
#36843: AIX build fails with failure to get random numbers
https://bugs.python.org/issue36843 closed by vstinner
#36856: faulthandler._stack_overflow doesn't work on x86-linux with KP
https://bugs.python.org/issue36856 closed by vstinner
#36891: Additional startup plugin for vendors
https://bugs.python.org/issue36891 closed by inada.naoki
#36922: Implement Py_TPFLAGS_METHOD_DESCRIPTOR
https://bugs.python.org/issue36922 closed by petr.viktorin
#36933: sys.set_coroutine_wrapper documented as to be removed in 3.8 (
https://bugs.python.org/issue36933 closed by asvetlov
#36969: pdb.do_args: display keyword-only and positional only argument
https://bugs.python.org/issue36969 closed by pablogsal
#36983: typing.__all__ has drifted from actual contents
https://bugs.python.org/issue36983 closed by gvanrossum
#36996: unittest.mock.patch decorator doesn't work with async function
https://bugs.python.org/issue36996 closed by asvetlov
#36999: Expose the coroutine object in asyncio task objects
https://bugs.python.org/issue36999 closed by asvetlov
#37001: symtable.symtable doesn't accept bytes which leads to a mismat
https://bugs.python.org/issue37001 closed by dino.viehland
#37003: ast unparse does not support f-string new debug format.
https://bugs.python.org/issue37003 closed by mbussonn
#37007: Implement socket.if_{nametoindex,indextoname} for Windows
https://bugs.python.org/issue37007 closed by steve.dower
#37015: Fix asyncio mock warnings
https://bugs.python.org/issue37015 closed by asvetlov
#37017: Use LOAD_METHOD optimization in CallMethod C API functions
https://bugs.python.org/issue37017 closed by inada.naoki
#37021: Can _random.getrandbits() be converted to METH_FASTCALL?
https://bugs.python.org/issue37021 closed by pablogsal
#37023: test_gdb failed on AMD64 Debian PGO 3.x
https://bugs.python.org/issue37023 closed by steve.dower
#37027: Return a safe proxy over a socket from get_extra_info('socket'
https://bugs.python.org/issue37027 closed by asvetlov
#37028: Implement asyncio repl
https://bugs.python.org/issue37028 closed by yselivanov
#37032: Add CodeType.replace() method
https://bugs.python.org/issue37032 closed by vstinner
#37035: Don't log OSError exceptions in asyncio transports
https://bugs.python.org/issue37035 closed by asvetlov
#37045: Implement PEP 591: add Final qualifier and @final decorator to
https://bugs.python.org/issue37045 closed by levkivskyi
#37046: Implement PEP 586: add Literal type constructor to typing
https://bugs.python.org/issue37046 closed by levkivskyi
#37047: Refactor AsyncMock setup logic in create_autospec
https://bugs.python.org/issue37047 closed by yselivanov
#37049: Implement PEP 589: add TypedDict to typing
https://bugs.python.org/issue37049 closed by levkivskyi
#37050: Remove expr_text from ast node FormattedValue
https://bugs.python.org/issue37050 closed by eric.smith
#37051: Glossary item "hashable" incorrect
https://bugs.python.org/issue37051 closed by rhettinger
#37053: Tools/parser/unparse.py does not handle u"bar" correctly
https://bugs.python.org/issue37053 closed by pablogsal
#37054: Ignored exceptions in test_memoryio
https://bugs.python.org/issue37054 closed by vstinner
#37056: test_tools is failing on several buildbots
https://bugs.python.org/issue37056 closed by levkivskyi
#37057: suspicious.py sphinx extension access non-existing attribute.
https://bugs.python.org/issue37057 closed by mbussonn
#37058: Implement PEP 544: add Protocol to typing
https://bugs.python.org/issue37058 closed by levkivskyi
#37059: spam
https://bugs.python.org/issue37059 closed by Mariatta
#37061: The strangest glitch I have ever seen - incorrect indenterror,
https://bugs.python.org/issue37061 closed by eric.smith
#37065: File and lineno is not reported for syntax error in f-string
https://bugs.python.org/issue37065 closed by SilentGhost
#37066: os.execl opening a new bash shell doesn't work if initfile/rcf
https://bugs.python.org/issue37066 closed by xtreak
#37067: os.execl doesn't allow for empty string in mac
https://bugs.python.org/issue37067 closed by ned.deily
#37070: Clean up f-string debug handling, including potential memory l
https://bugs.python.org/issue37070 closed by eric.smith
#37072: PyNode_Compile() crashes in Python 3.8.
https://bugs.python.org/issue37072 closed by gvanrossum
#37076: _thread.start_new_thread(): call sys.unraisablehook() to handl
https://bugs.python.org/issue37076 closed by vstinner
#37078: spam
https://bugs.python.org/issue37078 closed by xtreak
#37079: PEM cadata causes ssl.SSLError: nested asn1 error
https://bugs.python.org/issue37079 closed by christian.heimes
#37092: LoggerAdapter doesn't have fatal() like Logger
https://bugs.python.org/issue37092 closed by vinay.sajip
#37094: Provide an example for TestCase.skipTest in unittest doc
https://bugs.python.org/issue37094 closed by zach.ware
#37099: test_inspect generates DeprecationWarning
https://bugs.python.org/issue37099 closed by xtreak
#37101: Filterer.filter can be rewritten using built-ins just as effic
https://bugs.python.org/issue37101 closed by dwvisser
#37104: logging.Logger.disabled is not documented
https://bugs.python.org/issue37104 closed by vinay.sajip
#37106: python re.escape doesn't escape some special characters.
https://bugs.python.org/issue37106 closed by eric.smith
#37108: Positional-only arguments break super()
https://bugs.python.org/issue37108 closed by pablogsal
#37109: Inacurrate documentation regarding return type of math.factori
https://bugs.python.org/issue37109 closed by Antony.Lee
#37111: Logging - Inconsistent behaviour when handling unicode
https://bugs.python.org/issue37111 closed by vinay.sajip
#37112: Error in compilig the AST for functions with optional position
https://bugs.python.org/issue37112 closed by pablogsal
#37113: 'ß'.upper() should return 'ẞ'
https://bugs.python.org/issue37113 closed by SilentGhost
#37114: lstrip remove extra characters in the presence of a matching n
https://bugs.python.org/issue37114 closed by christian.heimes
#37115: Support annotations in positional-only arguments
https://bugs.python.org/issue37115 closed by pablogsal
#1230540: Add threading.excepthook() to handle uncaught exceptions raise
https://bugs.python.org/issue1230540 closed by vstinner
#1195571: simple callback system for Py_FatalError
https://bugs.python.org/issue1195571 closed by vstinner
[View Less]
1
0

May 31, 2019
Hi folks,
The reference implementation for PEP 558 (my attempt to fix the interaction
between tracing functions and closure variables) is currently segfaulting
somewhere deep in the garbage collector, and I've found that there's an
issue with the PyEval_GetLocals() API returning a borrowed reference that
means I need to tweak the proposed C API a bit such that PyEval_GetLocals()
returns the proxy at function scope, and we add a new PyEval_GetPyLocals()
that matches the locals() builtin.
I don'…
[View More]t *want* to postpone this to Python 3.9, but there turned out to be
more remaining work than I thought there was to get this ready for
inclusion in beta 1.
I'll try to get the C API design details sorted today, but the segfault is
mystifying me, and prevents the option of putting the core implementation
in place for b1, and tidying up the documentation and comments for b2.
Cheers,
Nick.
[View Less]
4
4
Hello,
I have one implementation question about vectorcall which is not
specified in PEP 590: what should happen if a type implements vectorcall
(i.e. _Py_TPFLAGS_HAVE_VECTORCALL is set) but doesn't set tp_call (i.e.
tp_call == NULL)? I see the following possibilities:
1. Ignore this problem/assume that it won't happen. This would be bad,
since callable(obj) would be False even though obj() would succeed.
2. Raise SystemError.
3. Automatically set tp_call to PyVectorcall_Call.
I would …
[View More]vote for 3 since it's the most user-friendly option. There is
also no way how it could be wrong: it ensures that tp_call and
vectorcall are consistent.
Any opinions?
Jeroen.
[View Less]
5
7
Hi folks,
After a couple of years hiatus, I finally got back to working on the
reference implementation for PEP 558, my proposal to resolve some
weird interactions between closures and trace functions in CPython by
formally defining the expected semantics of the locals() builtin at
function scope, and then ensuring that CPython adheres to those
clarified semantics.
The full text of the PEP is included below, and the rendered version
is available at https://www.python.org/dev/peps/pep-0558/
…
[View More]The gist of the PEP is that:
1. The behaviour when no trace functions are installed and no frame
introspection APIs are invoked is fine, so nothing should change in
that regard (except that we elevate that behaviour from "the way
CPython happens to work" to "the way the language and library
reference says that Python implementations are supposed to work", and
make sure CPython continues to behave that way even when a trace hook
*is* installed)
2. If you get hold of a frame object in CPython (or another
implementation that emulates the CPython frame API), whether via a
trace hook or via a frame introspection API, then writing to the
returned mapping will update the actual function locals and/or closure
reference immediately, rather than relying on the
FastToLocals/LocalsToFast APIs
3. The LocalsToFast C API changes to always produce RuntimeError
(since there's no way for us to make it actually work correctly and
consistently in the presence of closures, and the trace hook
implementation won't need it any more given the write-through proxy
behaviour on frame objects' "f_locals" attribute)
The reference implementation still isn't quite done yet, but it's far
enough along that I'm happy with the semantics and C API updates
proposed in the current version of the PEP.
Cheers,
Nick.
P.S. I'm away this weekend, so I expect the reference implementation
to be done late next week, and I'd be submitting the PEP to Nathaniel
for formal pronouncement at that point. However, I'm posting this
thread now so that there's more time for discussion prior to the 3.8b1
deadline.
==============
PEP: 558
Title: Defined semantics for locals()
Author: Nick Coghlan <ncoghlan(a)gmail.com>
BDFL-Delegate: Nathaniel J. Smith
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 2017-09-08
Python-Version: 3.8
Post-History: 2017-09-08, 2019-05-22
Abstract
========
The semantics of the ``locals()`` builtin have historically been underspecified
and hence implementation dependent.
This PEP proposes formally standardising on the behaviour of the CPython 3.6
reference implementation for most execution scopes, with some adjustments to the
behaviour at function scope to make it more predictable and independent of the
presence or absence of tracing functions.
Rationale
=========
While the precise semantics of the ``locals()`` builtin are nominally undefined,
in practice, many Python programs depend on it behaving exactly as it behaves in
CPython (at least when no tracing functions are installed).
Other implementations such as PyPy are currently replicating that behaviour,
up to and including replication of local variable mutation bugs that
can arise when a trace hook is installed [1]_.
While this PEP considers CPython's current behaviour when no trace hooks are
installed to be acceptable (and largely desirable), it considers the current
behaviour when trace hooks are installed to be problematic, as it causes bugs
like [1]_ *without* even reliably enabling the desired functionality of allowing
debuggers like ``pdb`` to mutate local variables [3]_.
Proposal
========
The expected semantics of the ``locals()`` builtin change based on the current
execution scope. For this purpose, the defined scopes of execution are:
* module scope: top-level module code, as well as any other code executed using
``exec()`` or ``eval()`` with a single namespace
* class scope: code in the body of a ``class`` statement, as well as any other
code executed using ``exec()`` or ``eval()`` with separate local and global
namespaces
* function scope: code in the body of a ``def`` or ``async def`` statement
We also allow interpreters to define two "modes" of execution, with only the
first mode being considered part of the language specification itself:
* regular operation: the way the interpreter behaves by default
* tracing mode: the way the interpreter behaves when a trace hook has been
registered in one or more threads via an implementation dependent mechanism
like ``sys.settrace`` ([4]_) in CPython's ``sys`` module or
``PyEval_SetTrace`` ([5]_) in CPython's C API
For regular operation, this PEP proposes elevating the current behaviour of
the CPython reference implementation to become part of the language
specification.
For tracing mode, this PEP proposes changes to CPython's behaviour at function
scope that bring the ``locals()`` builtin semantics closer to those used in
regular operation, while also making the related frame API semantics clearer
and easier for interactive debuggers to rely on.
The proposed tracing mode changes also affect the semantics of frame object
references obtained through other means, such as via a traceback, or via the
``sys._getframe()`` API.
New ``locals()`` documentation
------------------------------
The heart of this proposal is to revise the documentation for the ``locals()``
builtin to read as follows:
Return a dictionary representing the current local symbol table, with
variable names as the keys, and their currently bound references as the
values. This will always be the same dictionary for a given runtime
execution frame.
At module scope, as well as when using ``exec()`` or ``eval()`` with a
single namespace, this function returns the same namespace as ``globals()``.
At class scope, it returns the namespace that will be passed to the
metaclass constructor.
When using ``exec()`` or ``eval()`` with separate local and global
namespaces, it returns the local namespace passed in to the function call.
At function scope (including for generators and coroutines), it returns a
dynamic snapshot of the function's local variables and any nonlocal cell
references. In this case, changes made via the snapshot are *not* written
back to the corresponding local variables or nonlocal cell references, and
any such changes to the snapshot will be overwritten if the snapshot is
subsequently refreshed (e.g. by another call to ``locals()``).
CPython implementation detail: the dynamic snapshot for the current frame
will be implicitly refreshed before each call to the trace function when a
trace function is active.
For reference, the current documentation of this builtin reads as follows:
Update and return a dictionary representing the current local symbol table.
Free variables are returned by locals() when it is called in function
blocks, but not in class blocks.
Note: The contents of this dictionary should not be modified; changes may
not affect the values of local and free variables used by the interpreter.
(In other words: the status quo is that the semantics and behaviour of
``locals()`` are currently formally implementation defined, whereas the proposed
state after this PEP is that the only implementation defined behaviour will be
that encountered at function scope when a tracing function is defined, with the
behaviour in all other cases being defined by the language and library
references)
Module scope
------------
At module scope, as well as when using ``exec()`` or ``eval()`` with a
single namespace, ``locals()`` must return the same object as ``globals()``,
which must be the actual execution namespace (available as
``inspect.currentframe().f_locals`` in implementations that provide access
to frame objects).
Variable assignments during subsequent code execution in the same scope must
dynamically change the contents of the returned mapping, and changes to the
returned mapping must change the values bound to local variable names in the
execution environment.
The semantics at module scope are required to be the same in both tracing
mode (if provided by the implementation) and in regular operation.
To capture this expectation as part of the language specification, the following
paragraph will be added to the documentation for ``locals()``:
At module scope, as well as when using ``exec()`` or ``eval()`` with a
single namespace, this function returns the same namespace as ``globals()``.
This part of the proposal does not require any changes to the reference
implementation - it is standardisation of the current behaviour.
Class scope
-----------
At class scope, as well as when using ``exec()`` or ``eval()`` with separate
global and local namespaces, ``locals()`` must return the specified local
namespace (which may be supplied by the metaclass ``__prepare__`` method
in the case of classes). As for module scope, this must be a direct reference
to the actual execution namespace (available as
``inspect.currentframe().f_locals`` in implementations that provide access
to frame objects).
Variable assignments during subsequent code execution in the same scope must
change the contents of the returned mapping, and changes to the returned mapping
must change the values bound to local variable names in the
execution environment.
The mapping returned by ``locals()`` will *not* be used as the actual class
namespace underlying the defined class (the class creation process will copy
the contents to a fresh dictionary that is only accessible by going through the
class machinery).
For nested classes defined inside a function, any nonlocal cells referenced from
the class scope are *not* included in the ``locals()`` mapping.
The semantics at class scope are required to be the same in both tracing
mode (if provided by the implementation) and in regular operation.
To capture this expectation as part of the language specification, the following
two paragraphs will be added to the documentation for ``locals()``:
When using ``exec()`` or ``eval()`` with separate local and global
namespaces, [this function] returns the given local namespace.
At class scope, it returns the namespace that will be passed to the metaclass
constructor.
This part of the proposal does not require any changes to the reference
implementation - it is standardisation of the current behaviour.
Function scope
--------------
At function scope, interpreter implementations are granted significant freedom
to optimise local variable access, and hence are NOT required to permit
arbitrary modification of local and nonlocal variable bindings through the
mapping returned from ``locals()``.
Historically, this leniency has been described in the language specification
with the words "The contents of this dictionary should not be modified; changes
may not affect the values of local and free variables used by the interpreter."
This PEP proposes to change that text to instead say:
At function scope (including for generators and coroutines), [this function]
returns a dynamic snapshot of the function's local variables and any
nonlocal cell references. In this case, changes made via the snapshot are
*not* written back to the corresponding local variables or nonlocal cell
references, and any such changes to the snapshot will be overwritten if the
snapshot is subsequently refreshed (e.g. by another call to ``locals()``).
CPython implementation detail: the dynamic snapshot for the currently
executing frame will be implicitly refreshed before each call to the trace
function when a trace function is active.
This part of the proposal *does* require changes to the CPython reference
implementation, as while it accurately describes the behaviour in regular
operation, the "write back" strategy currently used to support namespace changes
from trace functions doesn't comply with it (and also causes the quirky
behavioural problems mentioned in the Rationale).
CPython Implementation Changes
==============================
The current cause of CPython's tracing mode quirks (both the side effects from
simply installing a tracing function and the fact that writing values back to
function locals only works for the specific function being traced) is the way
that locals mutation support for trace hooks is currently implemented: the
``PyFrame_LocalsToFast`` function.
When a trace function is installed, CPython currently does the following for
function frames (those where the code object uses "fast locals" semantics):
1. Calls ``PyFrame_FastToLocals`` to update the dynamic snapshot
2. Calls the trace hook (with tracing of the hook itself disabled)
3. Calls ``PyFrame_LocalsToFast`` to capture any changes made to the dynamic
snapshot
This approach is problematic for a few different reasons:
* Even if the trace function doesn't mutate the snapshot, the final step resets
any cell references back to the state they were in before the trace function
was called (this is the root cause of the bug report in [1]_)
* If the trace function *does* mutate the snapshot, but then does something
that causes the snapshot to be refreshed, those changes are lost (this is
one aspect of the bug report in [3]_)
* If the trace function attempts to mutate the local variables of a frame other
than the one being traced (e.g. ``frame.f_back.f_locals``), those changes
will almost certainly be lost (this is another aspect of the bug report in
[3]_)
* If a ``locals()`` reference is passed to another function, and *that*
function mutates the snapshot namespace, then those changes *may* be written
back to the execution frame *if* a trace hook is installed
The proposed resolution to this problem is to take advantage of the fact that
whereas functions typically access their *own* namespace using the language
defined ``locals()`` builtin, trace functions necessarily use the implementation
dependent ``frame.f_locals`` interface, as a frame reference is what gets
passed to hook implementations.
Instead of being a direct reference to the dynamic snapshot returned by
``locals()``, ``frame.f_locals`` will be updated to instead return a dedicated
proxy type (implemented as a private subclass of the existing
``types.MappingProxyType``) that has two internal attributes not exposed as
part of either the Python or public C API:
* *mapping*: the dynamic snapshot that is returned by the ``locals()`` builtin
* *frame*: the underlying frame that the snapshot is for
``__setitem__`` and ``__delitem__`` operations on the proxy will affect not only
the dynamic snapshot, but *also* the corresponding fast local or cell reference
on the underlying frame.
The ``locals()`` builtin will be made aware of this proxy type, and continue to
return a reference to the dynamic snapshot rather than to the write-through
proxy.
At the C API layer, ``PyEval_GetLocals()`` will implement the same semantics
as the Python level ``locals()`` builtin, and a new
``PyFrame_GetPyLocals(frame)`` accessor API will be provided to allow the
function level proxy bypass logic to be encapsulated entirely inside the frame
implementation.
The C level equivalent of accessing ``pyframe.f_locals`` in Python will be a
new ``PyFrame_GetLocalsAttr(frame)`` API. Like the Python level descriptor, the
new API will implicitly refresh the dynamic snapshot at function scope before
returning a reference to the write-through proxy.
The ``PyFrame_LocalsToFast()`` function will be changed to always emit
``RuntimeError``, explaining that it is no longer a supported operation, and
affected code should be updated to rely on the write-through tracing mode
proxy instead.
Design Discussion
=================
Ensuring ``locals()`` returns a shared snapshot at function scope
-----------------------------------------------------------------
The ``locals()`` builtin is a required part of the language, and in the
reference implementation it has historically returned a mutable mapping with
the following characteristics:
* each call to ``locals()`` returns the *same* mapping
* for namespaces where ``locals()`` returns a reference to something other than
the actual local execution namespace, each call to ``locals()`` updates the
mapping with the current state of the local variables and any referenced
nonlocal cells
* changes to the returned mapping *usually* aren't written back to the
local variable bindings or the nonlocal cell references, but write backs
can be triggered by doing one of the following:
* installing a Python level trace hook (write backs then happen whenever
the trace hook is called)
* running a function level wildcard import (requires bytecode
injection in Py3)
* running an ``exec`` statement in the function's scope (Py2 only, since
``exec`` became an ordinary builtin in Python 3)
The proposal in this PEP aims to retain the first two properties (to maintain
backwards compatibility with as much code as possible) while ensuring that
simply installing a trace hook can't enable rebinding of function locals via
the ``locals()`` builtin (whereas enabling rebinding via
``frame.f_locals`` inside the tracehook implementation is fully intended).
Keeping ``locals()`` as a dynamic snapshot at function scope
------------------------------------------------------------
It would theoretically be possible to change the semantics of the ``locals()``
builtin to return the write-through proxy at function scope, rather than
continuing to return a dynamic snapshot.
This PEP doesn't (and won't) propose this as it's a backwards incompatible
change in practice, even though code that relies on the current behaviour is
technically operating in an undefined area of the language specification.
Consider the following code snippet::
def example():
x = 1
locals()["x"] = 2
print(x)
Even with a trace hook installed, that function will consistently print ``1``
on the current reference interpreter implementation::
>>> example()
1
>>> import sys
>>> def basic_hook(*args):
... return basic_hook
...
>>> sys.settrace(basic_hook)
>>> example()
1
Similarly, ``locals()`` can be passed to the ``exec()`` and ``eval()`` builtins
at function scope without risking unexpected rebinding of local variables.
Provoking the reference interpreter into incorrectly mutating the local variable
state requires a more complex setup where a nested function closes over a
variable being rebound in the outer function, and due to the use of either
threads, generators, or coroutines, it's possible for a trace function to start
running for the nested function before the rebinding operation in the outer
function, but finish running after the rebinding operation has taken place (in
which case the rebinding will be reverted, which is the bug reported in [1]_).
In addition to preserving the de facto semantics which have been in place since
PEP 227 introduced nested scopes in Python 2.1, the other benefit of restricting
the write-through proxy support to the implementation-defined frame object API
is that it means that only interpreter implementations which emulate the full
frame API need to offer the write-through capability at all, and that
JIT-compiled implementations only need to enable it when a frame introspection
API is invoked, or a trace hook is installed, not whenever ``locals()`` is
accessed at function scope.
What happens with the default args for ``eval()`` and ``exec()``?
-----------------------------------------------------------------
These are formally defined as inheriting ``globals()`` and ``locals()`` from
the calling scope by default.
There isn't any need for the PEP to change these defaults, so it doesn't.
Changing the frame API semantics in regular operation
-----------------------------------------------------
Earlier versions of this PEP proposed having the semantics of the frame
``f_locals`` attribute depend on whether or not a tracing hook was currently
installed - only providing the write-through proxy behaviour when a tracing hook
was active, and otherwise behaving the same as the ``locals()`` builtin.
That was adopted as the original design proposal for a couple of key reasons,
one pragmatic and one more philosophical:
* Object allocations and method wrappers aren't free, and tracing functions
aren't the only operations that access frame locals from outside the function.
Restricting the changes to tracing mode meant that the additional memory and
execution time overhead of these changes would as close to zero in regular
operation as we can possibly make them.
* "Don't change what isn't broken": the current tracing mode problems are caused
by a requirement that's specific to tracing mode (support for external
rebinding of function local variable references), so it made sense to also
restrict any related fixes to tracing mode
However, actually attempting to implement and document that dynamic approach
highlighted the fact that it makes for a really subtle runtime state dependent
behaviour distinction in how ``frame.f_locals`` works, and creates several
new edge cases around how ``f_locals`` behaves as trace functions are added
and removed.
Accordingly, the design was switched to the current one, where
``frame.f_locals`` is always a write-through proxy, and ``locals()`` is always
a dynamic snapshot, which is both simpler to implement and easier to explain.
Regardless of how the CPython reference implementation chooses to handle this,
optimising compilers and interpreters also remain free to impose additional
restrictions on debuggers, by making local variable mutation through frame
objects an opt-in behaviour that may disable some optimisations (just as the
emulation of CPython's frame API is already an opt-in flag in some Python
implementations).
Historical semantics at function scope
--------------------------------------
The current semantics of mutating ``locals()`` and ``frame.f_locals`` in CPython
are rather quirky due to historical implementation details:
* actual execution uses the fast locals array for local variable bindings and
cell references for nonlocal variables
* there's a ``PyFrame_FastToLocals`` operation that populates the frame's
``f_locals`` attribute based on the current state of the fast locals array
and any referenced cells. This exists for three reasons:
* allowing trace functions to read the state of local variables
* allowing traceback processors to read the state of local variables
* allowing ``locals()`` to read the state of local variables
* a direct reference to ``frame.f_locals`` is returned from ``locals()``, so if
you hand out multiple concurrent references, then all those references will be
to the exact same dictionary
* the two common calls to the reverse operation, ``PyFrame_LocalsToFast``, were
removed in the migration to Python 3: ``exec`` is no longer a statement (and
hence can no longer affect function local namespaces), and the compiler now
disallows the use of ``from module import *`` operations at function scope
* however, two obscure calling paths remain: ``PyFrame_LocalsToFast`` is called
as part of returning from a trace function (which allows debuggers to make
changes to the local variable state), and you can also still inject the
``IMPORT_STAR`` opcode when creating a function directly from a code object
rather than via the compiler
This proposal deliberately *doesn't* formalise these semantics as is, since they
only make sense in terms of the historical evolution of the language and the
reference implementation, rather than being deliberately designed.
Implementation
==============
The reference implementation update is in development as a draft pull
request on GitHub ([6]_).
Acknowledgements
================
Thanks to Nathaniel J. Smith for proposing the write-through proxy idea in
[1]_ and pointing out some critical design flaws in earlier iterations of the
PEP that attempted to avoid introducing such a proxy.
References
==========
.. [1] Broken local variable assignment given threads + trace hook + closure
(https://bugs.python.org/issue30744)
.. [2] Clarify the required behaviour of ``locals()``
(https://bugs.python.org/issue17960)
.. [3] Updating function local variables from pdb is unreliable
(https://bugs.python.org/issue9633)
.. [4] CPython's Python API for installing trace hooks
(https://docs.python.org/dev/library/sys.html#sys.settrace)
.. [5] CPython's C API for installing trace hooks
(https://docs.python.org/3/c-api/init.html#c.PyEval_SetTrace)
.. [6] PEP 558 reference implementation
(https://github.com/python/cpython/pull/3640/files)
Copyright
=========
This document has been placed in the public domain.
--
Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia
[View Less]
13
30
Hi, all.
Yury implemented per opcode cache for LOAD_GLOBAL,
LOAD_ATTR, and LOAD_METHOD. [1]
I update the patch for current master branch, but only for
LOAD_GLOBAL for now. [2] It sped up LOAD_GLOBAL
about 40%. [3] It is attractive optimization.
Now 3.8b1 will be coming soon, but the pull request is not
reviewed well yet.
For example, should we add some switch to disable the cache?
May I merge it before beta1 and polish (or revert) it by rc1?
Or should I postpone it for 3.9?
Regards,
[1]…
[View More]: https://bugs.python.org/issue26219
[2]: https://github.com/python/cpython/pull/12884
[3]: https://github.com/methane/sandbox/tree/master/2019/opcache_load_global#opc…
--
Inada Naoki <songofacandy(a)gmail.com>
[View Less]
2
1

May 29, 2019
Ok, now I'm mildly curious to knpw:
What is the justification for causing list == 3 to evaluate to False,
besides the obvious "a list cannot equal a number"?
On Tue, May 28, 2019 at 9:34 PM Montana Burr <montana.burr(a)gmail.com> wrote:
> Ok, now I'm mildly curious to knpw:
>
> What is the justification for causing list == 3 to evaluate to False,
> besides the obvious "a list cannot equal a number"?
>
> On Tue, May 28, 2019 at 7:52 PM Rob Cliffe <rob.cliffe(a)…
[View More]btinternet.com>
> wrote:
>
>>
>>
>> On 27/05/2019 04:52:17, Montana Burr wrote:
>> > NumPy arrays have this awesome feature, where array == 3 does an
>> > element-wise comparison and returns a list. For example:
>> >
>> > np.array([1,2,3,4,5])==3
>> >
>> > returns
>> >
>> > [False,False,True,False,False]
>> >
>> > It would be cool if Python had similar functionality for lists.
>> >
>> Well, it does have:
>> >>> [x==3 for x in [1,2,3,4,5]]
>> [False, False, True, False, False]
>>
>> This is IMHO much more intuitive than your construct overloading "==".
>> It is also more flexible (any operation can be performed on x, not just
>> an equality comparison).
>> So sorry, but I can see no justification for changing the Python
>> language to do something, which can already be done, to be done in a
>> more obscure way.
>> Also
>> [1,2,3,4,5]==3
>> is already legal syntax (it evaluates to False, since the operands are
>> not equal), so you are proposing a code-breaking change.
>>
>> Apologies if someone has already pointed all this out, as is very likely
>> (I have only just (re-)joined python-ideas and may have missed the
>> relevant posts).
>> Best wishes
>> Rob Cliffe
>>
>
>
> --
>
> WebRep
> Overall rating
>
>
--
WebRep
Overall rating
[View Less]
3
2