Python-Dev
Threads by month
- ----- 2025 -----
- February
- January
- ----- 2024 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2023 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2022 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2021 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2020 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2019 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2018 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2017 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2016 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2015 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2014 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2013 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2012 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2011 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2010 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2009 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2008 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2007 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2006 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2005 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2004 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2003 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2002 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2001 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2000 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 1999 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
June 2019
- 68 participants
- 48 discussions
Hi,
I have updated the PEP with feedback from discussions. The highlights are:
* Deprecate parser module
* Keep fileinput module
* Elaborate why crypt and spwd are dangerous and bad
* Improve sections for cgitb, colorsys, nntplib, and smtpd modules
* The colorsys, crypt, imghdr, sndhdr, and spwd sections now list suitable substitutions.
* Mention that socketserver is going to stay for http.server and xmlrpc.server
* The future maintenance section now states that the deprecated modules may be …
[View More]adopted by Python community members.
https://github.com/python/peps/compare/7799178a...2d536899?diff=unified#dif…
I'll be traveling the next couple of days and will only have limited opportunities to respond on feedback.
Christian
-------------------------------------------------------------------
PEP: 594
Title: Removing dead batteries from the standard library
Author: Christian Heimes <christian(a)python.org>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 20-May-2019
Post-History: 21-May-2019
Abstract
========
This PEP proposed a list of standard library modules to be removed from the
standard library. The modules are mostly historic data formats and APIs that
have been superseded a long time ago, e.g. Mac OS 9 and Commodore.
Rationale
=========
Back in the early days of Python, the interpreter came with a large set of
useful modules. This was often refrained to as "batteries included"
philosophy and was one of the corner stones to Python's success story.
Users didn't have to figure out how to download and install separate
packages in order to write a simple web server or parse email.
Times have changed. The introduction of the cheese shop (PyPI), setuptools,
and later pip, it became simple and straight forward to download and install
packages. Nowadays Python has a rich and vibrant ecosystem of third party
packages. It's pretty much standard to either install packages from PyPI or
use one of the many Python or Linux distributions.
On the other hand, Python's standard library is piling up cruft, unnecessary
duplication of functionality, and dispensable features. This is undesirable
for several reasons.
* Any additional module increases the maintenance cost for the Python core
development team. The team has limited resources, reduced maintenance cost
frees development time for other improvements.
* Modules in the standard library are generally favored and seen as the
de-facto solution for a problem. A majority of users only pick 3rd party
modules to replace a stdlib module, when they have a compelling reason, e.g.
lxml instead of `xml`. The removal of an unmaintained stdlib module
increases the chances of a community contributed module to become widely
used.
* A lean and mean standard library benefits platforms with limited resources
like devices with just a few hundred kilobyte of storage (e.g. BBC
Micro:bit). Python on mobile platforms like BeeWare or WebAssembly
(e.g. pyodide) also benefit from reduced download size.
The modules in the PEP have been selected for deprecation because their
removal is either least controversial or most beneficial. For example
least controversial are 30 years old multimedia formats like ``sunau``
audio format, which was used on SPARC and NeXT workstations in the late
1980ties. The ``crypt`` module has fundamental flaws that are better solved
outside the standard library.
This PEP also designates some modules as not scheduled for removal. Some
modules have been deprecated for several releases or seem unnecessary at
first glance. However it is beneficial to keep the modules in the standard
library, mostly for environments where installing a package from PyPI is not
an option. This can be cooperate environments or class rooms where external
code is not permitted without legal approval.
* The usage of FTP is declining, but some files are still provided over
the FTP protocol or hosters offer FTP to upload content. Therefore
``ftplib`` is going to stay.
* The ``optparse`` and ``getopt`` module are widely used. They are mature
modules with very low maintenance overhead.
* According to David Beazley [5]_ the ``wave`` module is easy to teach to
kids and can make crazy sounds. Making a computer generate crazy sounds is
powerful and highly motivating exercise for a 9yo aspiring developer. It's
a fun battery to keep.
Deprecation schedule
====================
3.8
---
This PEP targets Python 3.8. Version 3.8.0 final is scheduled to be released
a few months before Python 2.7 will reach its end of lifetime. We expect that
Python 3.8 will be targeted by users that migrate to Python 3 in 2019 and
2020. To reduce churn and to allow a smooth transition from Python 2,
Python 3.8 will neither raise `DeprecationWarning` nor remove any
modules that have been scheduled for removal. Instead deprecated modules will
just be *documented* as deprecated. Optionally modules may emit a
`PendingDeprecationWarning`.
All deprecated modules will also undergo a feature freeze. No additional
features should be added. Bug should still be fixed.
3.9
---
Starting with Python 3.9, deprecated modules will start issuing
`DeprecationWarning`. The `parser`_ module is removed and potentially
replaced with a new module.
All other deprecated modules are fully supported and will receive security
updates until Python 3.9 reaches its end of lifetime. Python 3.9.0 will
be released about 18 months after 3.8.0 (April 2021?) and most likely
be supported for 5 years after the release. The estimated EOL of Python 3.9
is in 2026.
3.10
----
In 3.10 all deprecated modules will be removed from the CPython repository
together with tests, documentation, and autoconf rules.
PEP acceptance process
======================
3.8.0b1 is scheduled to be release shortly after the PEP is officially
submitted. Since it's improbable that the PEP will pass all stages of the
PEP process in time, I propose a two step acceptance process that is
analogous Python's two release deprecation process.
The first *provisionally accepted* phase targets Python 3.8.0b1. In the first
phase no code is changes or removed. Modules are only documented as
deprecated. The only exception is the `parser`_ module. It has been
documented as deprecated since Python 2.5 and is scheduled for removal for
3.9 to make place for a more advanced parser.
The final decision, which modules will be removed and how the removed code
is preserved, can be delayed for another year.
Deprecated modules
==================
The modules are grouped as data encoding, multimedia, network, OS interface,
and misc modules. The majority of modules are for old data formats or
old APIs. Some others are rarely useful and have better replacements on
PyPI, e.g. Pillow for image processing or NumPy-based projects to deal with
audio processing.
.. csv-table:: Table 1: Proposed modules deprecations
:header: "Module", "Deprecated in", "To be removed", "Replacement"
:widths: 1, 1, 1, 2
aifc,3.8,3.10,\-
asynchat,3.8,3.10,asyncio
asyncore,3.8,3.10,asyncio
audioop,3.8,3.10,\-
binhex,3.8,3.10,\-
cgi,3.8,3.10,\-
cgitb,3.8,3.10,\-
chunk,3.8,3.10,\-
colorsys,3.8,3.10,"colormath, colour, colorspacious, Pillow"
crypt,3.8,3.10,"bcrypt, argon2cffi, hashlib, passlib"
fileinput,\-,**keep**,argparse
formatter,3.4,3.10,\-
fpectl,**3.7**,**3.7**,\-
getopt,**3.2**,**keep**,"argparse, optparse"
imghdr,3.8,3.10,"filetype, puremagic, python-magic"
imp,**3.4**,3.10,importlib
lib2to3,\-,**keep**,
macpath,**3.7**,**3.8**,\-
msilib,3.8,3.10,\-
nntplib,3.8,3.10,\-
nis,3.8,3.10,\-
optparse,\-,**keep**,argparse
ossaudiodev,3.8,3.10,\-
parser,**2.5**,**3.9**,"ast, lib2to3.pgen2"
pipes,3.8,3.10,subprocess
smtpd,"**3.4.7**, **3.5.4**",3.10,aiosmtpd
sndhdr,3.8,3.10,"filetype, puremagic, python-magic"
spwd,3.8,3.10,"python-pam, simplepam"
sunau,3.8,3.10,\-
uu,3.8,3.10,\-
wave,\-,**keep**,
xdrlib,3.8,3.10,\-
Data encoding modules
---------------------
binhex
~~~~~~
The `binhex <https://docs.python.org/3/library/binhex.html>`_ module encodes
and decodes Apple Macintosh binhex4 data. It was originally developed for
TSR-80. In the 1980s and early 1990s it was used on classic Mac OS 9 to
encode binary email attachments.
Module type
pure Python
Deprecated in
3.8
To be removed in
3.10
Substitute
**none**
uu
~~
The `uu <https://docs.python.org/3/library/uu.html>`_ module provides
uuencode format, an old binary encoding format for email from 1980. The uu
format has been replaced by MIME. The uu codec is provided by the binascii
module.
Module type
pure Python
Deprecated in
3.8
To be removed in
3.10
Substitute
**none**
xdrlib
~~~~~~
The `xdrlib <https://docs.python.org/3/library/xdrlib.html>`_ module supports
the Sun External Data Representation Standard. XDR is an old binary
serialization format from 1987. These days it's rarely used outside
specialized domains like NFS.
Module type
pure Python
Deprecated in
3.8
To be removed in
3.10
Substitute
**none**
Multimedia modules
------------------
aifc
~~~~
The `aifc <https://docs.python.org/3/library/aifc.html>`_ module provides
support for reading and writing AIFF and AIFF-C files. The Audio Interchange
File Format is an old audio format from 1988 based on Amiga IFF. It was most
commonly used on the Apple Macintosh. These days only few specialized
application use AIFF.
Module type
pure Python (depends on `audioop`_ C extension)
Deprecated in
3.8
To be removed in
3.10
Substitute
**none**
audioop
~~~~~~~
The `audioop <https://docs.python.org/3/library/audioop.html>`_ module
contains helper functions to manipulate raw audio data and adaptive
differential pulse-code modulated audio data. The module is implemented in
C without any additional dependencies. The `aifc`_, `sunau`_, and `wave`_
module depend on `audioop`_ for some operations. The byteswap operation in
the `wave`_ module can be substituted with little work.
Module type
C extension
Deprecated in
3.8
To be removed in
3.10
Substitute
**none**
colorsys
~~~~~~~~
The `colorsys <https://docs.python.org/3/library/colorsys.html>`_ module
defines color conversion functions between RGB, YIQ, HSL, and HSV coordinate
systems.
The PyPI packages *colormath*, *colour*, and *colorspacious* provide more and
advanced features. The Pillow library is better suited to transform images
between color systems.
Module type
pure Python
Deprecated in
3.8
To be removed in
3.10
Substitute
`colormath <https://pypi.org/project/colormath/>`_,
`colour <https://pypi.org/project/colour/>`_
`colorspacious <https://pypi.org/project/colorspacious/>`_,
`Pillow <https://pypi.org/project/Pillow/>`_
chunk
~~~~~
The `chunk <https://docs.python.org/3/library/chunk.html>`_ module provides
support for reading and writing Electronic Arts' Interchange File Format.
IFF is an old audio file format originally introduced for Commodore and
Amiga. The format is no longer relevant.
Module type
pure Python
Deprecated in
3.8
To be removed in
3.10
Substitute
**none**
imghdr
~~~~~~
The `imghdr <https://docs.python.org/3/library/imghdr.html>`_ module is a
simple tool to guess the image file format from the first 32 bytes
of a file or buffer. It supports only a limited amount of formats and
neither returns resolution nor color depth.
Module type
pure Python
Deprecated in
3.8
To be removed in
3.10
Substitute
`puremagic <https://pypi.org/project/puremagic/>`_,
`filetype <https://pypi.org/project/filetype/>`_,
`python-magic <https://pypi.org/project/python-magic/>`_
ossaudiodev
~~~~~~~~~~~
The `ossaudiodev <https://docs.python.org/3/library/ossaudiodev.html>`_
module provides support for Open Sound System, an interface to sound
playback and capture devices. OSS was initially free software, but later
support for newer sound devices and improvements were proprietary. Linux
community abandoned OSS in favor of ALSA [1]_. Some operation systems like
OpenBSD and NetBSD provide an incomplete [2]_ emulation of OSS.
Module type
C extension
Deprecated in
3.8
To be removed in
3.10
Substitute
**none**
sndhdr
~~~~~~
The `sndhdr <https://docs.python.org/3/library/sndhdr.html>`_ module is
similar to the `imghdr`_ module but for audio formats. It guesses file
format, channels, frame rate, and sample widths from the first 512 bytes of
a file or buffer. The module only supports AU, AIFF, HCOM, VOC, WAV, and
other ancient formats.
Module type
pure Python (depends on `audioop`_ C extension for some operations)
Deprecated in
3.8
To be removed in
3.10
Substitute
`puremagic <https://pypi.org/project/puremagic/>`_,
`filetype <https://pypi.org/project/filetype/>`_,
`python-magic <https://pypi.org/project/python-magic/>`_
sunau
~~~~~
The `sunau <https://docs.python.org/3/library/sunhdr.html>`_ module provides
support for Sun AU sound format. It's yet another old, obsolete file format.
Module type
pure Python (depends on `audioop`_ C extension for some operations)
Deprecated in
3.8
To be removed in
3.10
Substitute
**none**
Networking modules
------------------
asynchat
~~~~~~~~
The `asynchat <https://docs.python.org/3/library/asynchat.html>`_ module
is build on top of `asyncore`_ and has been deprecated since Python 3.6.
Module type
pure Python
Deprecated in
3.6
Removed in
3.10
Substitute
asyncio
asyncore
~~~~~~~~
The `asyncore <https://docs.python.org/3/library/asyncore.html>`_ module was
the first module for asynchronous socket service clients and servers. It
has been replaced by asyncio and is deprecated since Python 3.6.
The ``asyncore`` module is also used in stdlib tests. The tests for
``ftplib``, ``logging``, ``smptd``, ``smtplib``, and ``ssl`` are partly
based on ``asyncore``. These tests must be updated to use asyncio or
threading.
Module type
pure Python
Deprecated in
3.6
Removed in
3.10
Substitute
asyncio
cgi
~~~
The `cgi <https://docs.python.org/3/library/cgi.html>`_ module is a support
module for Common Gateway Interface (CGI) scripts. CGI is deemed as
inefficient because every incoming request is handled in a new process. PEP
206 considers the module as *designed poorly and are now near-impossible
to fix*.
Several people proposed to either keep the cgi module for features like
`cgi.parse_qs()` or move `cgi.escape()` to a different module. The
functions `cgi.parse_qs` and `cgi.parse_qsl` have been
deprecated for a while and are actually aliases for
`urllib.parse.parse_qs` and `urllib.parse.parse_qsl`. The
function `cgi.quote` has been deprecated in favor of `html.quote`
with secure default values.
Module type
pure Python
Deprecated in
3.8
To be removed in
3.10
Substitute
**none**
cgitb
~~~~~
The `cgitb <https://docs.python.org/3/library/cgitb.html>`_ module is a
helper for the cgi module for configurable tracebacks.
The ``cgitb`` module is not used by any major Python web framework (Django,
Pyramid, Plone, Flask, CherryPy, or Bottle). Only Paste uses it in an
optional debugging middleware.
Module type
pure Python
Deprecated in
3.8
To be removed in
3.10
Substitute
**none**
smtpd
~~~~~
The `smtpd <https://docs.python.org/3/library/smtpd.html>`_ module provides
a simple implementation of a SMTP mail server. The module documentation
marks the module as deprecated and recommends ``aiosmtpd`` instead. The
deprecation message was added in releases 3.4.7, 3.5.4, and 3.6.1.
Module type
pure Python
Deprecated in
**3.7**
To be removed in
3.10
Substitute
aiosmtpd
nntplib
~~~~~~~
The `nntplib <https://docs.python.org/3/library/nntplib.html>`_ module
implements the client side of the Network News Transfer Protocol (nntp). News
groups used to be a dominant platform for online discussions. Over the last
two decades, news has been slowly but steadily replaced with mailing lists
and web-based discussion platforms. Twisted is also
`planning <https://twistedmatrix.com/trac/ticket/9405>`_ to deprecate NNTP
support and `pynnt <https://github.com/greenbender/pynntp>`_ hasn't seen any
activity since 2014. This is a good indicator that the public interest in
NNTP support is declining.
The ``nntplib`` tests have been the cause of additional work in the recent
past. Python only contains client side of NNTP. The tests connect to
external news server. The servers are sometimes unavailble, too slow, or do
not work correctly over IPv6. The situation causes flaky test runs on
buildbots.
Module type
pure Python
Deprecated in
3.8
To be removed in
3.10
Substitute
**none**
Operating system interface
--------------------------
crypt
~~~~~
The `crypt <https://docs.python.org/3/library/crypt.html>`_ module implements
password hashing based on ``crypt(3)`` function from ``libcrypt`` or
``libxcrypt`` on Unix-like platform. The algorithms are mostly old, of poor
quality and insecure. Users are discouraged to use them.
* The module is not available on Windows. Cross-platform application need
an alternative implementation any way.
* Only DES encryption is guarenteed to be available. DES has an extremely
limited key space of 2**56.
* MD5, salted SHA256, salted SHA512, and Blowfish are optional extension.
SSHA256 and SSHA512 are glibc extensions. Blowfish (bcrypt) is the only
algorithm that is still secure. However it's in glibc and therefore not
commonly available on Linux.
* Depending on the platform, the ``crypt`` module is not thread safe. Only
implementations with ``crypt_r(3)`` are thread safe.
* The module was never useful to interact with system user and password
databases. On BSD, macOS, and Linux, all user authentication and
password modification operations must go through PAM (pluggable
authentication module), see `spwd`_ deprecation.
Module type
C extension + Python module
Deprecated in
3.8
To be removed in
3.10
Substitute
`bcrypt <https://pypi.org/project/bcrypt/>`_,
`passlib <https://pypi.org/project/passlib/>`_,
`argon2cffi <https://pypi.org/project/argon2-cffi/>`_,
hashlib module (PBKDF2, scrypt)
macpath
~~~~~~~
The `macpath <https://docs.python.org/3/library/macpath.html>`_ module
provides Mac OS 9 implementation of os.path routines. Mac OS 9 is no longer
supported
Module type
pure Python
Deprecated in
3.7
Removed in
3.8
Substitute
**none**
nis
~~~
The `nis <https://docs.python.org/3/library/nis.html>`_ module provides
NIS/YP support. Network Information Service / Yellow Pages is an old and
deprecated directory service protocol developed by Sun Microsystems. It's
designed successor NIS+ from 1992 never took off. For a long time, libc's
Name Service Switch, LDAP, and Kerberos/GSSAPI are considered a more powerful
and more secure replacement of NIS.
Module type
C extension
Deprecated in
3.8
To be removed in
3.10
Substitute
**none**
spwd
~~~~
The `spwd <https://docs.python.org/3/library/spwd.html>`_ module provides
direct access to Unix shadow password database using non-standard APIs.
In general it's a bad idea to use the spwd. The spwd circumvents system
security policies, it does not use the PAM stack, and is only compatible
with local user accounts, because it ignores NSS. The use of the ``spwd``
module for access control must be consider a *security bug*, as it bypasses
PAM's access control.
Further more the ``spwd`` module uses the
`shadow(3) <http://man7.org/linux/man-pages/man3/shadow.3.html>`_ APIs.
Functions like ``getspnam(3)`` access the ``/etc/shadow`` file directly. This
is dangerous and even forbidden for confined services on systems with a
security engine like SELinux or AppArmor.
Module type
C extension
Deprecated in
3.8
To be removed in
3.10
Substitute
`python-pam <https://pypi.org/project/python-pam/>`_,
`simpleplam <https://pypi.org/project/simplepam/>`_
Misc modules
------------
formatter
~~~~~~~~~
The `formatter <https://docs.python.org/3/library/formatter.html>`_ module
is an old text formatting module which has been deprecated since Python 3.4.
Module type
pure Python
Deprecated in
3.4
To be removed in
3.10
Substitute
*n/a*
imp
~~~
The `imp <https://docs.python.org/3/library/imp.html>`_ module is the
predecessor of the
`importlib <https://docs.python.org/3/library/importlib.html>`_ module. Most
functions have been deprecated since Python 3.3 and the module since
Python 3.4.
Module type
C extension
Deprecated in
3.4
To be removed in
3.10
Substitute
importlib
msilib
~~~~~~
The `msilib <https://docs.python.org/3/library/msilib.html>`_ package is a
Windows-only package. It supports the creation of Microsoft Installers (MSI).
The package also exposes additional APIs to create cabinet files (CAB). The
module is used to facilitate distutils to create MSI installers with
``bdist_msi`` command. In the past it was used to create CPython's official
Windows installer, too.
Microsoft is slowly moving away from MSI in favor of Windows 10 Apps (AppX)
as new deployment model [3]_.
Module type
C extension + Python code
Deprecated in
3.8
To be removed in
3.10
Substitute
**none**
parser
~~~~~~
The `parser <https://docs.python.org/3/library/parser.html>`_ module provides
an interface to Python’s internal parser and byte-code compiler. The stdlib
has superior ways to interact with the parse tree. From Python 2.5 onward,
it's much more convenient to cut in at the Abstract Syntax Tree (AST)
generation and compilation stage.
The ``parser`` module causes additional work. It's C code that must be
kept in sync with any change to Python's grammar and internal parser.
Pablo wants to remove the parser module and promote lib2to3's pgen2 instead
[6]_.
Most importantly the presence of the ``parser`` module makes it harder to
switch to something more powerful than a LL(1) parser [7]_. Since the
``parser`` module is documented as deprecated since Python 2.5 and a new
parsing technology is planned for 3.9, the ``parser`` module is scheduled for
removal in 3.9.
Module type
C extension
Deprecated in
3.8, documented as deprecated since **2.5**
To be removed in
**3.9**
Substitute
ast, lib2to3.pgen2
pipes
~~~~~
The `pipes <https://docs.python.org/3/library/pipes.html>`_ module provides
helpers to pipe the input of one command into the output of another command.
The module is built on top of ``os.popen``. Users are encouraged to use
the subprocess module instead.
Module type
pure Python
Deprecated in
3.8
To be removed in
3.10
Substitute
subprocess module
Removed modules
===============
fpectl
------
The `fpectl <https://docs.python.org/3.6/library/fpectl.html>`_ module was
never built by default, its usage was discouraged and considered dangerous.
It also required a configure flag that caused an ABI incompatibility. The
module was removed in 3.7 by Nathaniel J. Smith in
`bpo-29137 <https://bugs.python.org/issue29137>`_.
Module type
C extension + CAPI
Deprecated in
3.7
Removed in
3.7
Substitute
**none**
Modules to keep
===============
Some modules were originally proposed for deprecation.
fileinput
---------
The `fileinput <https://docs.python.org/3/library/fileinput.html>`_ module
implements a helpers to iterate over a list of files from ``sys.argv``. The
module predates the optparser and argparser module. The same functionality
can be implemented with the argparser module.
Several core developers expressed their interest to keep the module in the
standard library, as it is handy for quick scripts.
Module type
pure Python
lib2to3
-------
The `lib2to3 <https://docs.python.org/3/library/2to3.html>`_ package provides
the ``2to3`` command to transpile Python 2 code to Python 3 code.
The package is useful for other tasks besides porting code from Python 2 to
3. For example `black`_ uses it for code reformatting.
Module type
pure Python
getopt
------
The `getopt <https://docs.python.org/3/library/getopt.html>`_ module mimics
C's getopt() option parser.
Although users are encouraged to use argparse instead, the getopt module is
still widely used. The module is small, simple, and handy for C developers
to write simple Python scripts.
Module type
pure Python
Substitute
argparse
optparse
--------
The `optparse <https://docs.python.org/3/library/optparse.html>`_ module is
the predecessor of the argparse module.
Although it has been deprecated for many years, it's still too widely used
to remove it.
Module type
pure Python
Deprecated in
3.2
Substitute
argparse
wave
----
The `wave <https://docs.python.org/3/library/wave.html>`_ module provides
support for the WAV sound format.
The module is not deprecated, because The WAV format is still relevant these
days. The ``wave`` module is also used in education, e.g. to show kids how
to make noise with a computer.
The module uses one simple function from the `audioop`_ module to perform
byte swapping between little and big endian formats. Before 24 bit WAV
support was added, byte swap used to be implemented with the ``array``
module. To remove ``wave``'s dependency on the ``audioop``, the byte swap
function could be either be moved to another module (e.g. ``operator``) or
the ``array`` module could gain support for 24 bit (3 byte) arrays.
Module type
pure Python (depends on *byteswap* from `audioop`_ C extension)
Deprecated in
3.8
To be removed in
3.10
Substitute
*n/a*
Future maintenance of removed modules
=====================================
The main goal of the PEP is to reduce the burden and workload on the Python
core developer team. Therefore removed modules will not be maintained by
the core team as separate PyPI packages. However the removed code, tests and
documentation may be moved into a new git repository, so community members
have a place from which they can pick up and fork code.
A first draft of a `legacylib <https://github.com/tiran/legacylib>`_
repository is available on my private Github account. The modules could be
made available on PyPI. The Python core team will not publish or maintain
the packages. It is my hope that members of the Python community will
adopt, maintain, and perhaps improve the deprecated modules.
It's my hope that some of the deprecated modules will be picked up and
adopted by users that actually care about them. For example ``colorsys`` and
``imghdr`` are useful modules, but have limited feature set. A fork of
``imghdr`` can add new features and support for more image formats, without
being constrained by Python's release cycle.
Most of the modules are in pure Python and can be easily packaged. Some
depend on a simple C module, e.g. `audioop`_ and `crypt`_. Since `audioop`_
does not depend on any external libraries, it can be shipped in as binary
wheels with some effort. Other C modules can be replaced with ctypes or cffi.
For example I created `legacycrypt <https://github.com/tiran/legacycrypt>`_
with ``_crypt`` extension reimplemented with a few lines of ctypes code.
Discussions
===========
* Elana Hashman and Nick Coghlan suggested to keep the *getopt* module.
* Berker Peksag proposed to deprecate and removed *msilib*.
* Brett Cannon recommended to delay active deprecation warnings and removal
of modules like *imp* until Python 3.10. Version 3.8 will be released
shortly before Python 2 reaches end of lifetime. A delay reduced churn for
users that migrate from Python 2 to 3.8.
* Brett also came up with the idea to keep lib2to3. The package is useful
for other purposes, e.g. `black <https://pypi.org/project/black/>`_ uses
it to reformat Python code.
* At one point, distutils was mentioned in the same sentence as this PEP.
To avoid lengthy discussion and delay of the PEP, I decided against dealing
with distutils. Deprecation of the distutils package will be handled by
another PEP.
* Multiple people (Gregory P. Smith, David Beazley, Nick Coghlan, ...)
convinced me to keep the `wave`_ module. [4]_
* Gregory P. Smith proposed to deprecate `nntplib`_. [4]_
* Andrew Svetlov mentioned the ``socketserver`` module is questionable.
However it's used to implement ``http.server`` and ``xmlrpc.server``. The
stdlib doesn't have a replacement for the servers, yet.
Update history
==============
Update 1
--------
* Deprecate `parser`_ module
* Keep `fileinput`_ module
* Elaborate why `crypt`_ and `spwd`_ are dangerous and bad
* Improve sections for `cgitb`_, `colorsys`_, `nntplib`_, and `smtpd`_ modules
* The `colorsys`_, `crypt`_, `imghdr`_, `sndhdr`_, and `spwd`_ sections now
list suitable substitutions.
* Mention that ``socketserver`` is going to stay for ``http.server`` and
``xmlrpc.server``
* The future maintenance section now states that the deprecated modules
may be adopted by Python community members.
References
==========
.. [1] https://en.wikipedia.org/wiki/Open_Sound_System#Free,_proprietary,_free
.. [2] https://man.openbsd.org/ossaudio
.. [3] https://blogs.msmvps.com/installsite/blog/2015/05/03/the-future-of-windows-…
.. [4] https://twitter.com/ChristianHeimes/status/1130257799475335169
.. [5] https://twitter.com/dabeaz/status/1130278844479545351
.. [6] https://mail.python.org/pipermail/python-dev/2019-May/157464.html
.. [7] https://discuss.python.org/t/switch-pythons-parsing-tech-to-something-more-…
Copyright
=========
This document has been placed in the public domain.
..
Local Variables:
mode: indented-text
indent-tabs-mode: nil
sentence-end-double-space: t
fill-column: 70
coding: utf-8
End:
[View Less]
17
35
The time has come for Python 3.8.0b1:
https://www.python.org/downloads/release/python-380b1/ <https://www.python.org/downloads/release/python-380b1/>
This release is the first of four planned beta release previews. Beta release previews are intended to give the wider community the opportunity to test new features and bug fixes and to prepare their projects to support the new feature release. The next pre-release of Python 3.8 will be 3.8.0b2, currently scheduled for 2019-07-01.
Call to …
[View More]action
We strongly encourage maintainers of third-party Python projects to test with 3.8 during the beta phase and report issues found to the Python bug tracker <https://bugs.python.org/> as soon as possible. While the release is planned to be feature complete entering the beta phase, it is possible that features may be modified or, in rare cases, deleted up until the start of the release candidate phase (2019-09-30). Our goal is have no ABI changes after beta 3 and no code changes after 3.8.0rc1, the release candidate. To achieve that, it will be extremely important to get as much exposure for 3.8 as possible during the beta phase.
Please keep in mind that this is a preview release and its use is not recommended for production environments.
A new challenger has appeared!
With the release of Python 3.8.0b1, development started on Python 3.9. The “master” branch in the cpython repository now tracks development of 3.9 while Python 3.8 received its own branch, called simply “3.8”.
Acknowledgments
As you might expect, creating new branches triggers a lot of changes in configuration for all sorts of tooling that we’re using. Additionally, the inevitable deadline for new features caused a flurry of activity that tested the buildbots to the max. The revert hammer got used more than once.
I would not be able to make this release available alone. Many thanks to the fearless duo of Pablo Galindo Salgado and Victor Stinner for spending tens of hours during the past week working on getting the buildbots green for release. Seriously, that took a lot of effort. We are all so lucky to have you both.
Thanks to Andrew Svetlov for his swift fixes to asyncio and to Yury Selivanov for code reviews, even when jetlagged. Thanks to Julien Palard for untangling the documentation configs. Thank you to Zachary Ware for help with buildbot and CI configuration. Thanks to Mariatta for helping with the bots. Thank you to Steve Dower for delivering the Windows installers.
Most importantly though, huge thanks to Ned Deily who not only helped me understand the scope of this special release but also did some of the grunt work involved.
Last but not least, thanks to you for making this release more meaty than I expected. There’s plenty of super exciting changes in there. Just take a look at “What’s New <https://docs.python.org/3.8/whatsnew/3.8.html>”!
One more thing
Hey, fellow Core Developer, Beta 2 is in four weeks. If your important new feature got reverted last minute, or you decided not to merge due to inadequate time, I have a one time offer for you (restrictions apply). If you:
find a second core developer champion for your change; and
in tandem you finish your change complete with tests and documentation before Beta 2
then I will let it in. I’m asking for a champion because it’s too late now for changes with hasty design or code review. And as I said, restrictions apply. For instance, at this point changes to existing APIs are unlikely to be accepted. Don’t start new work with 3.8 in mind. 3.9 is going to come sooner than you think!
- Ł
[View Less]
1
0

June 4, 2019
Hi everyone,
Just a heads-up regarding some messages you will see in your pull requests.
There is an intermittent failure on some buildbots
regarding asyncio:
https://buildbot.python.org/all/#/builders/21
As the builds do not fail all the time, the systems understand that if your
(merged) commits fail to build, they may be the cause
of the failure and then it does a report into the pull request.
I am working on investigating a way to improve the report mechanism to make
it less noisy in …
[View More]this case, but bear in mind that
the correct way to solve this is fixing the asyncio bug in the test suite
and this won't likely go away completely until is solved.
We are doing all that we can to solve all the recent leaks and failures on
the test suite, but there is a noticeable increase in the
number of merged pull requests because of the imminent feature freeze and
because this happens across several timezones
is very difficult to get them all.
Thanks to everyone that is helping solving these bugs :)
Regards from sunny London,
Pablo Galindo Salgado
[View Less]
2
2
First, I want to say: I'm very happy with PEP 558's changes to
f_locals. It solves the weird threading bugs, and exposes the
fundamental operations you need for debugging in a simple and clean
way, while leaving a lot of implementation flexibility for future
Python VMs. It's a huge improvement over what we had before.
I'm not as sure about the locals() parts of the proposal. It might be
fine, but there are some complex trade-offs here that I'm still trying
to wrap my head around. The rest of …
[View More]this document is me thinking out
loud to try to clarify these issues.
##### What are we trying to solve?
There are two major questions, which are somewhat distinct:
- What should the behavior of locals() be in CPython?
- How much of that should be part of the language definition, vs
CPython implementation details?
The status quo is that for locals() inside function scope, the
behavior is quite complex and subtle, and it's entirely implementation
defined. In the current PEP draft, there are some small changes to the
semantics, and also it promotes them becoming part of the official
language semantics.
I think the first question, about semantics, is the more important
one. If we're promoting them to the language definition, the main
effect is just to make it more important we get the semantics right.
##### What are the PEP's proposed semantics for locals()?
They're kinda subtle. [Nick: please double-check this section, both
for errors and because I think it includes some edge cases that the
PEP currently doesn't mention.]
For module/class scopes, locals() has always returned a mapping object
which acts as a "source of truth" for the actual local environment –
mutating the environment directly changes the mapping object, and
vice-versa. That's not going to change.
In function scopes, things are more complicated. The *local
environment* is conceptually well-defined, and includes:
- local variables (current source of truth: "fast locals" array)
- closed-over variables (current source of truth: cell objects)
- any arbitrary key/values written to frame.f_locals that don't
correspond to local or closed-over variables, e.g. you can do
frame.f_locals[object()] = 10, and then later read it out again.
However, the mapping returned by locals() does not directly reflect
this local environment. Instead, each function frame has a dict
associated with it. locals() returns this dict. The dict always holds
any non-local/non-closed-over variables, and also, in certain
circumstances, we write a snapshot of local and closed-over variables
back into the dict.
Specifically, we write back:
- Whenever locals() is called
- Whenever exec() or eval() is called without passing an explicit
locals argument
- After every trace/profile event, if a Python-level tracing/profiling
function is registered.
(Note: in CPython, the use of Python-level tracing/profiling functions
is extremely rare. It's more common in alternative implementations
like PyPy. For example, the coverage package uses a C-level tracing
function on CPython, which does not trigger locals updates, but on
PyPy it uses a Python-level tracing function, which does trigger
updates.)
In addition, the PEP doesn't say, but I think that any writes to
f_locals immediately update both the environment and the locals dict.
These semantics have some surprising consequences. Most obviously, in
function scope (unlike other scopes), mutating locals() does not
affect the actual local environment:
def f():
a = 1
locals()["a"] = 2
assert a == 1
The writeback rules can also produce surprising results:
def f():
loc1 = locals()
# Since it's a snapshot created at the time of the call
# to locals(), it doesn't contain 'loc1':
assert "loc1" not in loc1
loc2 = locals()
# Now loc1 has changed:
assert "loc1" in loc1
However, the results here are totally different if a Python-level
tracing/profiling function is installed – in particular, the first
assertion fails.
The interaction between f_locals and and locals() is also subtle:
def f():
a = 1
loc = locals()
assert "loc" not in loc
# Regular variable updates don't affect 'loc'
a = 2
assert loc["a"] == 1
# But debugging updates do:
sys._getframe().f_locals["a"] = 3
assert a == 3
assert loc["a"] == 3
# But it's not a full writeback
assert "loc" not in loc
# Mutating 'loc' doesn't affect f_locals:
loc["a"] = 1
assert sys._getframe().f_locals["a"] == 1
# Except when it does:
loc["b"] = 3
assert sys._getframe().f_locals["b"] == 3
Again, the results here are totally different if a Python-level
tracing/profiling function is installed.
And you can also hit these subtleties via 'exec' and 'eval':
def f():
a = 1
loc = locals()
assert "loc" not in loc
# exec() triggers writeback, and then mutates the locals dict
exec("a = 2; b = 3")
# So now the current environment has been reflected into 'loc'
assert "loc" in loc
# Also loc["a"] has been changed to reflect the exec'ed assignments
assert loc["a"] == 2
# But if we look at the actual environment, directly or via
# f_locals, we can see that 'a' has not changed:
assert a == 1
assert sys._getframe().f_locals["a"] == 1
# loc["b"] changed as well:
assert loc["b"] == 3
# And this *does* show up in f_locals:
assert sys._getframe().f_locals["b"] == 3
Of course, many of these edge cases are pretty obscure, so it's not
clear how much they matter. But I think we can at least agree that
this isn't the one obvious way to do it :-).
##### What's the landscape of possible semantics?
I did some brainstorming, and came up with 4 sets of semantics that
seem plausible enough to at least consider:
- [PEP]: the semantics in the current PEP draft.
- [PEP-minus-tracing]: same as [PEP], except dropping the writeback on
Python-level trace/profile events.
- [snapshot]: in function scope, each call to locals() returns a new,
*static* snapshot of the local environment, removing all this
writeback stuff. Something like:
def locals():
frame = get_caller_frame()
if is_function_scope(frame):
# make a point-in-time copy of the "live" proxy object
return dict(frame.f_locals)
else:
# in module/class scope, return the actual local environment
return frame.f_locals
- [proxy]: Simply return the .f_locals object, so in all contexts
locals() returns a live mutable view of the actual environment:
def locals():
return get_caller_frame().f_locals
##### How to evaluate our options?
I can think of a lot of criteria that all-else-being-equal we would
like Python to meet. (Of course, in practice they conflict.)
Consistency across APIs: it's surprising if locals() and
frame.f_locals do different things. This argues for [proxy].
Consistency across contexts: it's surprising if locals() has acts
differently in module/class scope versus function scope. This argues
for [proxy].
Consistent behavior when the environment shifts, or small maintenance
changes are made: it's nice if code that works today keeps working
tomorrow. On this criterion, I think [snapshot] > [proxy] >
[PEP-minus-tracing] >>> [PEP]. [PEP] is particularly bad here because
a very rare environmental change that almost no-one tests and mostly
only happens when debugging (i.e., enabling tracing) causes a radical
change in semantics.
Simplicity of explaining to users: all else being equal, it's nice if
our docs are short and clear and the language fits in your head. On
this criterion, I think: [proxy] > [snapshot] > [PEP-minus-tracing] >
[PEP]. As evidence that the current behavior is confusing, see:
- Ned gets confused and writes a long blog post after he figures it
out: https://nedbatchelder.com/blog/201211/tricky_locals.html
- A linter that warns against mutating locals():
https://lgtm.com/rules/10030096/
Simplicity of implementation: "If the implementation is easy to
explain, it may be a good idea." Since we need the proxy code anyway
to implement f_locals, I think it's: [proxy] (free) > [snapshot] (one
'if' statement) > ([PEP] = [PEP-minus-tracing]).
Impact on other interpreter implementations: all else being equal,
we'd like to give new interpreters maximal freedom to do clever
things. (And local variables are a place where language VMs often
expend a lot of cleverness.) [proxy] and [snapshot] are both easily
implemented in terms of f_locals, so they basically don't constrain
alternative implementations at all. I'm not as sure about [PEP] and
[PEP-minus-tracing]. I originally thought they must be horrible. On
further thought, I'm not convinced they're *that* bad, since the need
to support people doing silly stuff like frame.f_locals[object()] = 10
means that implementations will already need to sometimes attach
something like a dict object to their function frames. But perhaps
alternative implementations would like to disallow this, or are OK
with making it really slow but care about locals() performance more.
Anyway, it's definitely ([proxy] = [snapshot]) > ([PEP] =
[PEP-minus-tracing]), but I'm not sure whether the '>' is large or
small.
Backwards compatibility: help(locals) says:
NOTE: Whether or not updates to this dictionary will affect
name lookups in the local scope and vice-versa is
*implementation dependent* and not covered by any backwards
compatibility guarantees.
So that claims that there are ~no backwards compatibility issues here.
I'm going to ignore that; no matter what the docs say, we still don't
want to break everyone's code. And unfortunately, I can't think of any
realistic way to do a gradual transition, with like deprecation
warnings and all that (can anyone else?), so whatever we do will be a
flag-day change.
Of our four options, [PEP] is intuitively the closest to what CPython
has traditionally done. But what exactly breaks under the different
approaches? I'll split this off into its own section.
##### Backwards compatibility
I'll split this into three parts: code that treats locals() as
read-only, exec()/eval(), and code that mutates locals().
I believe (but haven't checked) that the majority of uses of locals()
are in simple cases like:
def f():
....
print("{a} {b}".format(**locals()))
Luckily, this code remains totally fine under all four of our candidates.
exec() and eval() are an interesting case. In Python 2, exec'ing some
assignments actually *did* mutate the local environment, e.g. you
could do:
# Python 2
def f():
exec "a = 1"
assert a == 1
In Python 3, this was changed, so now exec() inside a function cannot
mutate the enclosing scope. We got some bug reports about this change,
and there are a number of questions on stackoverflow about it, e.g.:
- https://bugs.python.org/issue4831
- https://stackoverflow.com/questions/52217525/how-can-i-change-the-value-of-…
- https://stackoverflow.com/questions/50995581/eval-exec-with-assigning-varia…
In all released versions of Python, eval() was syntactically unable to
rebind variables, so eval()'s interaction with the local environment
was undefined. However, in 3.8, eval() *will* be able to rebind
variables using the ':=' operator, so this interaction will become
user-visible. Presumably we'll want eval() to match exec().
OK, with that background out of the way, let's look at our candidates.
If we adopt [proxy], then that will mean exec()-inside-functions will
go back to the Python 2 behavior, where executing assignments in the
enclosing scope actually changes the enclosing scope. This will likely
break some code out there that's relying on the Python 3 behavior,
though I don't know how common that is. (I'm guessing not too common?
Using the same variable inside and outside an 'exec' and trusting that
they *won't* be the same seems like an unusual thing to do. But I
don't know.)
With [PEP] and [PEP-minus-tracing], exec() is totally unchanged. With
[snapshot], there's technically a small difference: if you call
locals() and then exec(), the exec() no longer triggers an implicit
writeback to the dict that locals() returned. I think we can ignore
this, and say that for all three of these, exec() is unlikely to
produce backwards compatibility issues.
OK, finally, let's talk about code that calls locals() and then
mutates the return value. The main difference between our candidates
is how they handle mutation, so this seems like the most important
case to focus on.
Conveniently, Mark Shannon and friends have statically analyzed a
large corpus of Python code and made a list of cases where people do
this: https://lgtm.com/rules/10030096/alerts/
Thanks! I haven't gone through the whole list, but I read through the
first few in the hopes of getting a better sense of what kind of code
does this in the real world and how it would be impacted by our
different options.
https://lgtm.com/projects/g/pydata/xarray/snapshot/a2ac6af744584c8afed3d56d…
Current: raises if a Python-level trace/profile function is set
[PEP]: raises if a Python-level trace/profile function is set
[PEP-minus-tracing]: ok
[snapshot]: ok
[proxy]: always raises
Comment: uses locals() to capture a bunch of passed in kwargs so it
can pass them as **kwargs to another function, and treats them like a
snapshot. The authors were clearly aware of the dangers, because this
pattern appears multiple times in this file, and all the other places
make an explicit copy of locals() before using it, but this place
apparently got missed. Fix is trivial: just do that here too.
https://github.com/swagger-api/swagger-codegen/blob/master/modules/swagger-…
Current: ok
[PEP]: ok
[PEP-minus-tracing]: ok
[snapshot]: ok
[proxy]: ok
Comment: this is inside a tool used to generate Python wrappers for
REST APIs. The vast majority of entries in the lgtm database are from
code generated by this tool. This was tricky to analyze, because it's
complex templated code, and it does mutate locals(). But I'm pretty
confident that the mutations end up just... not mattering, and all
possible generated code works under all of our candidates. Which is
lucky, because if we did have to fix this it wouldn't be trivial:
fixing up the code generator itself wouldn't be too hard, but it'll
take years for everyone to regenerate their old wrappers.
https://lgtm.com/projects/g/saltstack/salt/snapshot/bb0950e5eafbb897c8e969e…
Current: ok
[PEP]: ok
[PEP-minus-tracing]: ok
[snapshot]: raises
[proxy]: ok
Comment: the use of locals() here is totally superfluous – it
repeatedly reads and writes to locals()[mod], and in all cases this
could be replaced by a simple variable, or any other dict. And it's
careful not to assign to any name that matches an actual local
variable, so it works fine with [proxy] too. But it does assume that
multiple calls to locals() return the same dict, so [snapshot] breaks
it. In this case the fix is trivial: just use a variable.
https://lgtm.com/projects/g/materialsproject/pymatgen/snapshot/fd6900ed1040…
Current: buggy if a trace/profile function is set
[PEP]: buggy if a trace/profile function is set
[PEP-minus-tracing]: ok
[snapshot]: ok
[proxy]: raises
Comment: Another example of collecting kwargs into a dict. Actually
this code is always buggy, because they seem to think that
dict.pop("a", "b") removes both "a" and "b" from the dict... but it
would raise an exception on [proxy], because they use one of those
local variables after popping it from locals(). Fix is trivial: take
an explicit snapshot of locals() before modifying it.
##### Conclusion so far
[PEP-minus-tracing] seems to strictly dominate [PEP]. It's equal or
better on all criteria, and actually *more* compatible with all the
legacy code I looked at, even though it's technically less consistent
with what CPython used to do. Unless someone points out some new
argument, I think we can reject the writeback-when-tracing part of the
PEP draft.
Choosing between the remaining three is more of a judgement call.
I'm leaning towards saying that on net, [snapshot] beats
[PEP-minus-tracing]: it's dramatically simpler, and the backwards
incompatibilities that we've found so far seem pretty minor, on par
with what we do in every point release. (In fact, in 3/4 of the cases
I looked at, [snapshot] is actually what users seemed to trying to use
in the first place.)
For [proxy] versus [snapshot], a lot depends on what we think of
changing the semantics of exec(). [proxy] is definitely more
consistent and elegant, and if we could go back in time I think it's
what we'd have done from the start. Its compatibility is maybe a bit
worse than [snapshot] on non-exec() cases, but this seems pretty minor
overall (it often doesn't matter, and if it does just write
dict(locals()) instead of locals(), like you would in non-function
scope). But the change in exec() semantics is an actual language
change, even though it may not affect much real code, so that's what
stands out for me.
I'd very much like to hear about any considerations I've missed, and
any opinions on the "judgement call" part.
--
Nathaniel J. Smith -- https://vorpus.org
[View Less]
10
26
Hello,
Berker and I have been working on a PEP that suggests we keep using
and improving bugs.python.org and Roundup instead of switching to
GitHub Issues as proposed by PEP 581.
The PEP covers:
* What are the advantages of Roundup over GitHub issues;
* What features are missing in Roundup and how can we add them;
* Issues with PEP 581;
* Issues with the migration plan proposed by PEP 588;
The rendered version of PEP 595 is available at
https://www.python.org/dev/peps/pep-0595/
For …
[View More]reference, you can consult PEP 581 and 588 at
https://www.python.org/dev/peps/pep-0581/ and
https://www.python.org/dev/peps/pep-0588/
The full text of the PEP is include below. We are planning to update
the PEP to include the feedback we receive and to update the status of
features as we implement them (we also have a Google Summer of Code
students working on it).
Best Regards,
Ezio Melotti
================
PEP: 595
Title: Improving bugs.python.org
Author: Ezio Melotti <ezio.melotti(a)gmail.com>, Berker Peksag
<berker.peksag(a)gmail.com>
Status: Draft
Type: Process
Content-Type: text/x-rst
Created: 12-May-2019
Abstract
========
This PEP proposes a list of improvements to make bugs.python.org
more usable for contributors and core developers. This PEP also
discusses why remaining on Roundup should be preferred over
switching to GitHub Issues, as proposed by :pep:`581`.
Motivation
==========
On May 14th, 2019 :pep:`581` has been accepted [#]_ without much
public discussion and without a clear consensus [#]_. The PEP
contains factual errors and doesn't address some of the
issues that the migration to GitHub Issues might present.
Given the scope of the migration, the amount of work required,
and how it will negatively affect the workflow during the
transition phase, this decision should be re-evaluated.
Roundup advantages over GitHub Issues
=====================================
This section discusses reasons why Roundup should be preferred
over GitHub Issues and Roundup features that are not available
on GitHub Issues.
* **Roundup is the status quo.** Roundup has been an integral
part of the CPython workflow for years. It is a stable product
that has been tested and customized to adapt to our needs as the
workflow evolved.
It is possible to gradually improve it and avoid the disruption
that a switch to a different system would inevitabily bring to
the workflow.
* **Open-source and Python powered.** Roundup is an open-source
project and is written in Python. By using it and supporting
it, we also support the Python ecosystem. Several features
developed for bpo have also been ported to upstream Roundup
over the years.
* **Fully customizable.** Roundup can be (and has been) fully
customized to fit our needs.
* **Finer-grained access control.** Roundup allows the creation
of different roles with different permissions (e.g. create,
view, edit, etc.) for each individual property, and users can
have multiple roles.
* **Flexible UI.** While Roundup UI might look dated, it is
convenient and flexible.
For example, on the issue page, each field (e.g. title, type,
versions, status, linked files and PRs, etc.) have appropriate
UI elements (input boxes, dropdowns, tables, etc.) that are
easy to set and also provide a convenient way to get info about
the issue at a glance. The number of fields, their values, and
the UI element they use is also fully customizable.
GitHub only provides labels.
The issue list page presents the issues in a compact and easy
to read table with separate columns for different fields. For
comparison, Roundup lists 50 issues in a screen, whereas GitHub
takes two screens to shows 25 issues.
* **Advanced search.** Roundup provides an accurate way to search
and filter by using any combination of issue fields.
It is also possible to customize the number of results and the
fields displayed in the table, and the sorting and grouping
(up to two levels).
bpo also provides predefined summaries (e.g. "Created by you",
"Assigned to you", etc.) and allows the creation of custom
search queries that can be conveniently accessed from the sidebar.
* **Nosy list autocomplete.** The nosy list has an autocomplete
feature that suggests maintainers and experts. The suggestions
are automatically updated when the experts index [#]_ changes.
* **Dependencies and Superseders.** Roundup allows to specify
dependencies that must be addressed before the current issues
can be closed and a superseder issue to easily mark duplicates
[#]_. The list of dependencies can also be used to create
meta-issues that references several other sub-issues [#]_.
Improving Roundup
=================
This section lists some of the issues mentioned by :pep:`581`
and other desired features and discusses how they can be implemented
by improving Roundup and/or our instance.
* **REST API support.** A REST API will make integration with other
services and the development of new tools and applications easiers.
Upstream Roundup now supports a REST API. Updating the tracker will
make the REST API available.
* **GitHub login support.** This will allow users to login
to bugs.python.org (bpo) without having to create a new account.
It will also solve issues with confirmation emails being marked
as spam, and provide two-factor authentication.
A patch to add this functionality is already available and is
being integrated at the time of writing [#]_.
* **Markdown support and message preview and editing.** This feature
will allow the use of Markdown in messages and the ability to
preview the message before the submission and edit it afterward.
This can be done, but it will take some work. Possible solutions
have been proposed on the roundup-devel mailing list [#]_.
* **"Remove me from nosy list" button.** Add a button on issue pages
to remove self from the nosy list.
This feature will be added during GSoC 2019.
* **Mobile friendly theme.** Current theme of bugs.python.org looks
dated and it doesn't work well with mobile browsers.
A mobile-friendly theme that is more modern but still familiar
will be added.
* **Add PR link to BPO emails.** Currently bpo emails don't include
links to the corresponding PRs.
A patch [#]_ is available to change the content of the bpo emails
from::
components: +Tkinter
versions: +Python 3.4
pull_requests: +42
to::
components: +Tkinter
versions: +Python 3.4
pull_request: https://github.com/python/cpython/pull/341
* **Python 3 support.** Using Python 3 will make maintenance easier.
Upstream Roundup now supports Python 3. Updating the tracker will
allow us to switch to Python 3. The instances will need to be
updated as well.
* **Use upstream Roundup.** We currently use a fork of Roundup with
a few modifications, most notably the GitHub integration. If this
is ported upstream, we can start using upstream Roundup without
having to maintain our fork.
PEP 581 issues
==============
This section addresses some errors and inaccuracies found in :pep:`581`.
The "Why GitHub?" section of PEP 581 lists features currently
available on GitHub Issues but not on Roundup. Some of this features
are currently supported:
* "Ability to reply to issue and pull request conversations via email."
* Being able to reply by email has been one of the core features of
Roundup since the beginning. It is also possible to create new
issues or close existing ones, set or modify fields, and add
attachments.
* "Email notifications containing metadata, integrated with Gmail,
allowing systematic filtering of emails."
* Emails sent by Roundup contains metadata that can be used for
filtering.
* "Additional privacy, such as offering the user a choice to hide an
email address, while still allowing communication with the user
through @-mentions."
* Email addresses are hidden by default to users that are not
registered. Registered users can see other users' addresses
because we configured the tracker to show them. It can easily
be changed if desired. Users can still be added to the nosy
list by using their username even if their address is hidden.
* "Ability to automatically close issues when a PR has been merged."
* The GitHub integration of Roundup automatically closes issues
when a commit that contains "fixes issue <id>" is merged.
(Alternative spellings such as "closes" or "bug" are also supported.)
See [#]_ for a recent example of this feature.
* "Support for permalinks, allowing easy quoting and copying &
pasting of source code."
* Roundup has permalinks for issues, messages, attachments, etc.
In addition, Roundup allows to easily rewrite broken URLs in
messages (e.g. if the code hosting changes).
* "Core developers, volunteers, and the PSF don't have to maintain the
issue infrastructure/site, giving us more time and resources to focus
on the development of Python."
* While this is partially true, additional resources are required to
write and maintain bots.
In some cases, bots are required to workaround GitHub's lack of
features rather than expanding. [#]_ was written
specifically to workaround GitHub's email integration.
Updating our bots to stay up-to-date with changes in the GitHub API
has also maintenance cost. [#]_ took two days to be fixed.
In addition, we will still need to maintain Roundup for bpo (even
if it becomes read-only) and for the other trackers
we currently host/maintain (Jython [#]_ and Roundup [#]_).
The "Issues with Roundup / bpo" section of :pep:`581` lists some issues
that have already been fixed:
* "The upstream Roundup code is in Mercurial. Without any CI available,
it puts heavy burden on the few existing maintainers in terms of
reviewing, testing, and applying patches."
* While Roundup uses Mercurial by default, there is a git clone
available on GitHub [#]_. Roundup also has CI available [#]_ [#]_.
* "There is no REST API available. There is an open issue in Roundup for
adding REST API. Last activity was in 2016."
* The REST API has been integrated and it's now available in Roundup.
* "Users email addresses are exposed. There is no option to mask it."
* Exposing addresses to registered and logged in users was a decision
taken when our instance was set up.
This has now been changed to make the email addresses hidden for
regular users too (Developers and Coordinators can still see them).
The "Email address"" column from the user listing page [#]_ has been
removed too.
* "It sends a number of unnecessary emails and notifications, and it is
difficult, if not impossible, to configure."
* This can be configured.
* "Creating an account has been a hassle. There have been reports of people
having trouble creating accounts or logging in."
* The main issue is confirmation emails being marked as spam. Work has
been done to resolve the issue.
Migration considerations
========================
This section describes issues with the migrations that might not
have been addressed by :pep:`581` and :pep:`588`.
:pep:`588` suggests to add a button to migrate issues to GitHub
only when someone wants to keep working on them. This approach
has several issues:
* bpo will need to be updated in order to add a button that,
once pressed, creates a new issue on GitHub, copies over all
the messages, attachments, and creates/adds label for the
existing fields. Permissions will also need to be tweaked
to make individual issues read-only once they are migrated,
and to prevent users to create new accounts.
* The issues will be split between two trackers; searching issues
will take significant more effort.
* The conversion from Roundup to GitHub is lossy, unless all
the bpo fields are converted into labels or preserved somewhere
else.
* bpo converts a number of references into links, including
issue, message, and PR IDs, changeset numbers, legacy SVN
revision numbers, paths to files in the repo, files in
tracebacks (detecting the correct branch), links to devguide
pages and sections [#]_. This happens when messages are
requested so it is possible to create the correct link (e.g.
all the file links used to point to hg.python.org and now
point to GitHub).
If the links are hardcoded during the migration, it will be
difficult (if not impossible) to change them later. If they
aren't, they will either be lost, or a tool to generate the
links and updating them will need to be written.
* GitHub doesn't provide a way to set and preserve issue IDs
if they are migrated automatically with the use of a button.
(Some projects managed to preserve the IDs by contaacting
the GitHub staff and migrating the issues *en masse*.)
* On top of the work and changes required to migrate to GitHub
issues, we will still need to keep running and maintaining
Roundup, for both our instance (read-only) and for the Jython
and Roundup trackers (read-write).
In addition to the issues listed in the "Open issues" section of
:pep:`588`, this issues will need to be addressed:
* GitHub is properietary and there is risk of vendor lock-in.
Their business model might change and they could shut down
altogether.
* Switching to GitHub Issues will likely increase the number of
invalid reports and increase the triaging effort. This concern
has been raised in the past in a Zulip topic [#]_.
There have been already cases where people posted comments on
PRs that required moderators to mark them as off-topic or
disruptive, delete them altogether, and even lock the
conversation [#]_.
* Roundup sends weekly reports to python-dev with a summary that
includes new issues, recent issues with no replies, recent
issues waiting for review, most discussed issues, closed issues,
and deltas for open/closed/total issue counts [#]_. The report
provides an easy way to keep track of the tracker activity and
to make sure that issues that require attention are noticed.
The data collect by the weekly report is also use to generate
statistics and graphs that can be used to gain new insights [#]_.
* There are currently two mailing lists where Roundup posts new
tracker issues and all messages respectively: new-bugs-announce
[#]_ and python-bugs-list [#]_. A new system will need to be
developed to preserve this functionality. These MLs offer
additional ways to keep track of the tracker activity.
References
==========
.. [#] [Python-Dev] PEP 581 (Using GitHub issues for CPython) is accepted
https://mail.python.org/pipermail/python-dev/2019-May/157399.html
.. [#] [python-committers] [Python-Dev] PEP 581 (Using GitHub issues
for CPython) is accepted
https://mail.python.org/pipermail/python-committers/2019-May/006755.html
.. [#] Experts Index -- Python Devguide
https://devguide.python.org/experts/
.. [#] An example of superseded issues:
"re.sub() replaces only several matches"
https://bugs.python.org/issue12078
.. [#] An example of meta issue using dependencies to track sub-issues:
"Meta-issue: support of the android platform""
https://bugs.python.org/issue26865
.. [#] Support logging in with GitHub
https://github.com/python/bugs.python.org/issues/7
.. [#] Re: [Roundup-devel] PEP 581 and Google Summer of Code
https://sourceforge.net/p/roundup/mailman/message/36667828/
.. [#] [Tracker-discuss] [issue624] bpo emails contain useless non-github
pull_request number - users want a link to actual github PR
https://mail.python.org/pipermail/tracker-discuss/2018-June/004547.html
.. [#] The commit reported in msg342882 closes the issue (see the history below)
https://bugs.python.org/issue36951#msg342882
.. [#] The cpython-emailer-webhook project
https://github.com/berkerpeksag/cpython-emailer-webhook
.. [#] A recent incident caused by GitHub
https://github.com/python/bedevere/pull/163
.. [#] Jython issue tracker
https://bugs.jython.org/
.. [#] Roundup issue tracker
https://issues.roundup-tracker.org/
.. [#] GitHub clone of Roundup
https://github.com/roundup-tracker/roundup
.. [#] Travis-CI for Roundup
https://travis-ci.org/roundup-tracker/roundup) and codecov
.. [#] Codecov for Roundup
https://codecov.io/gh/roundup-tracker/roundup/commits
.. [#] User listing -- Python tracker
https://bugs.python.org/user?@sort=username
.. [#] Generating Special Links in a Comment -- Python Devguide
https://devguide.python.org/triaging/#generating-special-links-in-a-comment
.. [#] The New-bugs-announce mailing list
https://mail.python.org/mailman/listinfo/new-bugs-announce
.. [#] The Python-bugs-list mailing list
https://mail.python.org/mailman/listinfo/python-bugs-list
.. [#] An example of [Python-Dev] Summary of Python tracker Issues
https://mail.python.org/pipermail/python-dev/2019-May/157483.html
.. [#] Issues stats -- Python tracker
https://bugs.python.org/issue?@template=stats
.. [#] s/n ratio -- Python -- Zulip
https://python.zulipchat.com/#narrow/stream/130206-pep581/topic/s.2Fn.20rat…
.. [#] For example this and other related PRs:
https://github.com/python/cpython/pull/9099
Copyright
=========
This document has been placed in the public domain.
..
Local Variables:
mode: indented-text
indent-tabs-mode: nil
sentence-end-double-space: t
fill-column: 70
coding: utf-8
End:
[View Less]
7
12

Re: [Python-Dev] Expected stability of PyCode_New() and types.CodeType() signatures
by Pablo Galindo Salgado June 1, 2019
by Pablo Galindo Salgado June 1, 2019
June 1, 2019
Opened https://bugs.python.org/issue37122 to track this in the bug tracker.
1
0

Re: [Python-Dev] Expected stability of PyCode_New() and types.CodeType() signatures
by Pablo Galindo Salgado June 1, 2019
by Pablo Galindo Salgado June 1, 2019
June 1, 2019
>
> I propose to make co_argcount meaning the number of positional
> parameters (i.e. positional-only + positional-or-keyword). This would
> remove the need of changing the code that uses co_argcount.
>
I like the proposal, it will certainly make handling normal cases
downstream much easier because
if you do not care about positional-only arguments you can keep
inspecting co_argcount
and that
will give you what you expect. Note that if we choose to do this, it has to
be done now-…
[View More]ish IMHO to
avoid making the change painful because it will change the semantics of
co_argcount.
> As for the code object constructor, I propose to make posonlyargcount an
> optional parameter (default 0) added after existing parameters.
> PyCode_New() can be kept unchanged, but we can add new PyCode_New2() or
> PyCode_NewEx() with different signature.
I am not convinced about having a default argument in the code constructor.
The code constructor
is kept with all arguments positional for efficiency and adding defaults
will make it slower or having
a more confusing an asymmetrical interface. Also, this will be misaligned
on how keyword-only
parameters are provided. This is by far not the first time this constructor
has changed.
On the Python side, the new code.replace should cover most of the
Python-side use cases regarding
creating code objects from the Python side.
[View Less]
1
0

June 1, 2019
Hi,
Thanks to the constructive discussions, I enhanced my PEP 587. I don't
plan any further change, the PEP is now ready to review (and maybe
even for pronouncement, hi Thomas! :-)).
The Rationale now better explains all challenges and the complexity of
the Python Initialization Configuration.
The "Isolate Python" section is a short guide explaining how configure
Python to embed it into an application.
The "Path Configuration" section elaborates the most interesting part
of the …
[View More]configuration: configure where Python looks for modules
(sys.path). I added PyWideStringList_Insert() to allow to prepend a
path in module_search_paths.
The "Python Issues" section give a long list of issues solved directly
or indirectly by this PEP.
I'm open for bikeshedding on PyConfig fields names and added functions
names ;-) I hesitate for "use_module_search_paths": maybe
"module_search_paths_set" is a better name, as in "is
module_search_paths set?". The purpose of this field is to allow to
have an empty sys.path (ask PyConfig_Read() to not override it). IMHO
an empty sys.path makes sense for some specific use cases, like
executing Pyhon code without any external module.
My PEP 587 proposes better names: Py_FrozenFlag becomes
PyConfig.pathconfig_warnings and Py_DebugFlag becomes
PyConfig.parser_debug. I also avoided double negation. For example,
Py_DontWriteBytecodeFlag becomes write_bytecode.
Changes between version 3 and version 2:
* PyConfig: Add configure_c_stdio and parse_argv; rename _frozen to
pathconfig_warnings.
* Rename functions using bytes strings and wide strings. For example,
Py_PreInitializeFromWideArgs() becomes Py_PreInitializeFromArgs(), and
PyConfig_SetArgv() becomes PyConfig_SetBytesArgv().
* Add PyWideStringList_Insert() function.
* New "Path configuration", "Isolate Python", "Python Issues" and
"Version History" sections.
* PyConfig_SetString() and PyConfig_SetBytesString() now requires the
configuration as the first argument.
* Rename Py_UnixMain() to Py_BytesMain()
HTML version:
https://www.python.org/dev/peps/pep-0587/
Full PEP text below.
I know that the PEP is long, but well, it's a complex topic, and I
chose to add many examples to make the API easier to understand.
Victor
---
PEP: 587
Title: Python Initialization Configuration
Author: Victor Stinner <vstinner(a)redhat.com>, Nick Coghlan <ncoghlan(a)gmail.com>
BDFL-Delegate: Thomas Wouters <thomas(a)python.org>
Discussions-To: python-dev(a)python.org
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 27-Mar-2019
Python-Version: 3.8
Abstract
========
Add a new C API to configure the Python Initialization providing finer
control on the whole configuration and better error reporting.
It becomes possible to read the configuration and modify it before it is
applied. It also becomes possible to completely override how Python
computes the module search paths (``sys.path``).
Building a customized Python which behaves as regular Python becomes
easier using the new ``Py_RunMain()`` function. Moreover, command line
arguments passed to ``PyConfig.argv`` are now parsed as the regular
Python parses command line options, and ``PyConfig.xoptions`` are
handled as ``-X opt`` command line options.
This extracts a subset of the API design from the PEP 432 development and
refactoring work that is now considered sufficiently stable to make public
(allowing 3rd party embedding applications access to the same configuration
APIs that the native CPython CLI is now using).
Rationale
=========
Python is highly configurable but its configuration evolved organically.
The initialization configuration is scattered all around the code using
different ways to set them: global configuration variables (ex:
``Py_IsolatedFlag``), environment variables (ex: ``PYTHONPATH``),
command line arguments (ex: ``-b``), configuration files (ex:
``pyvenv.cfg``), function calls (ex: ``Py_SetProgramName()``). A
straightforward and reliable way to configure Python is needed.
Some configuration parameters are not accessible from the C API, or not
easily. For example, there is no API to override the default values of
``sys.executable``.
Some options like ``PYTHONPATH`` can only be set using an environment
variable which has a side effect on Python child processes.
Some options also depends on other options: see `Priority and Rules`_.
Python 3.7 API does not provide a consistent view of the overall
configuration.
The C API of Python 3.7 Initialization takes ``wchar_t*`` strings as
input whereas the Python filesystem encoding is set during the
initialization which can lead to mojibake.
Python 3.7 APIs like ``Py_Initialize()`` aborts the process on memory
allocation failure which is not convenient when Python is embedded.
Moreover, ``Py_Main()`` could exit directly the process rather than
returning an exit code. Proposed new API reports the error or exit code
to the caller which can decide how to handle it.
Implementing the PEP 540 (UTF-8 Mode) and the new ``-X dev`` correctly
was almost impossible in Python 3.6. The code base has been deeply
reworked in Python 3.7 and then in Python 3.8 to read the configuration
into a structure with no side effect. It becomes possible to clear the
configuration (release memory) and read again the configuration if the
encoding changed . It is required to implement properly the UTF-8 which
changes the encoding using ``-X utf8`` command line option. Internally,
bytes ``argv`` strings are decoded from the filesystem encoding. The
``-X dev`` changes the memory allocator (behaves as
``PYTHONMALLOC=debug``), whereas it was not possible to change the
memory allocation *while* parsing the command line arguments. The new
design of the internal implementation not only allowed to implement
properly ``-X utf8`` and ``-X dev``, it also allows to change the Python
behavior way more easily, especially for corner cases like that, and
ensure that the configuration remains consistent: see `Priority and
Rules`_.
This PEP is a partial implementation of PEP 432 which is the overall
design. New fields can be added later to ``PyConfig`` structure to
finish the implementation of the PEP 432 (e.g. by adding a new partial
initialization API which allows to configure Python using Python objects to
finish the full initialization). However, those features are omitted from this
PEP as even the native CPython CLI doesn't work that way - the public API
proposal in this PEP is limited to features which have already been implemented
and adopted as private APIs for us in the native CPython CLI.
Python Initialization C API
===========================
This PEP proposes to add the following new structures, functions and
macros.
New structures (4):
* ``PyConfig``
* ``PyInitError``
* ``PyPreConfig``
* ``PyWideStringList``
New functions (17):
* ``Py_PreInitialize(config)``
* ``Py_PreInitializeFromBytesArgs(config, argc, argv)``
* ``Py_PreInitializeFromArgs(config, argc, argv)``
* ``PyWideStringList_Append(list, item)``
* ``PyWideStringList_Insert(list, index, item)``
* ``PyConfig_SetString(config,config_str, str)``
* ``PyConfig_SetBytesString(config, config_str, str)``
* ``PyConfig_SetBytesArgv(config, argc, argv)``
* ``PyConfig_SetArgv(config, argc, argv)``
* ``PyConfig_Read(config)``
* ``PyConfig_Clear(config)``
* ``Py_InitializeFromConfig(config)``
* ``Py_InitializeFromBytesArgs(config, argc, argv)``
* ``Py_InitializeFromArgs(config, argc, argv)``
* ``Py_BytesMain(argc, argv)``
* ``Py_RunMain()``
* ``Py_ExitInitError(err)``
New macros (9):
* ``PyPreConfig_INIT``
* ``PyConfig_INIT``
* ``Py_INIT_OK()``
* ``Py_INIT_ERR(MSG)``
* ``Py_INIT_NO_MEMORY()``
* ``Py_INIT_EXIT(EXITCODE)``
* ``Py_INIT_IS_ERROR(err)``
* ``Py_INIT_IS_EXIT(err)``
* ``Py_INIT_FAILED(err)``
This PEP also adds ``_PyRuntimeState.preconfig`` (``PyPreConfig`` type)
and ``PyInterpreterState.config`` (``PyConfig`` type) fields to these
internal structures. ``PyInterpreterState.config`` becomes the new
reference configuration, replacing global configuration variables and
other private variables.
PyWideStringList
----------------
``PyWideStringList`` is a list of ``wchar_t*`` strings.
Example to initialize a string from C static array::
static wchar_t* argv[2] = {
L"-c",
L"pass",
};
PyWideStringList config_argv = PyWideStringList_INIT;
config_argv.length = Py_ARRAY_LENGTH(argv);
config_argv.items = argv;
``PyWideStringList`` structure fields:
* ``length`` (``Py_ssize_t``)
* ``items`` (``wchar_t**``)
Methods:
* ``PyInitError PyWideStringList_Append(PyWideStringList *list, const
wchar_t *item)``:
Append *item* to *list*.
* ``PyInitError PyWideStringList_Insert(PyWideStringList *list,
Py_ssize_t index, const wchar_t *item)``:
Insert *item* into *list* at *index*. If *index* is greater than
*list* length, just append *item* to *list*.
If *length* is non-zero, *items* must be non-NULL and all strings must
be non-NULL.
PyInitError
-----------
``PyInitError`` is a structure to store an error message or an exit code
for the Python Initialization. For an error, it stores the C function
name which created the error.
Example::
PyInitError alloc(void **ptr, size_t size)
{
*ptr = PyMem_RawMalloc(size);
if (*ptr == NULL) {
return Py_INIT_NO_MEMORY();
}
return Py_INIT_OK();
}
int main(int argc, char **argv)
{
void *ptr;
PyInitError err = alloc(&ptr, 16);
if (Py_INIT_FAILED(err)) {
Py_ExitInitError(err);
}
PyMem_Free(ptr);
return 0;
}
``PyInitError`` fields:
* ``exitcode`` (``int``):
argument passed to ``exit()``, only set by ``Py_INIT_EXIT()``.
* ``err_msg`` (``const char*``): error message
* private ``_func`` field: used by ``Py_INIT_ERR()`` to store the C
function name which created the error.
* private ``_type`` field: for internal usage only.
Macro to create an error:
* ``Py_INIT_OK()``: Success.
* ``Py_INIT_ERR(err_msg)``: Initialization error with a message.
* ``Py_INIT_NO_MEMORY()``: Memory allocation failure (out of memory).
* ``Py_INIT_EXIT(exitcode)``: Exit Python with the specified exit code.
Other macros and functions:
* ``Py_INIT_IS_ERROR(err)``: Is the result an error?
* ``Py_INIT_IS_EXIT(err)``: Is the result an exit?
* ``Py_INIT_FAILED(err)``: Is the result an error or an exit? Similar
to ``Py_INIT_IS_ERROR(err) || Py_INIT_IS_EXIT(err)``.
* ``Py_ExitInitError(err)``: Call ``exit(exitcode)`` on Unix or
``ExitProcess(exitcode)`` if the result is an exit, call
``Py_FatalError(err_msg)`` if the result is an error. Must not be
called if the result is a success.
Pre-Initialization with PyPreConfig
-----------------------------------
``PyPreConfig`` structure is used to pre-initialize Python:
* Set the memory allocator
* Configure the LC_CTYPE locale
* Set the UTF-8 mode
Example using the pre-initialization to enable the UTF-8 Mode::
PyPreConfig preconfig = PyPreConfig_INIT;
preconfig.utf8_mode = 1;
PyInitError err = Py_PreInitialize(&preconfig);
if (Py_INIT_FAILED(err)) {
Py_ExitInitError(err);
}
/* at this point, Python will speak UTF-8 */
Py_Initialize();
/* ... use Python API here ... */
Py_Finalize();
Functions to pre-initialize Python:
* ``PyInitError Py_PreInitialize(const PyPreConfig *config)``
* ``PyInitError Py_PreInitializeFromBytesArgs(const PyPreConfig
*config, int argc, char **argv)``
* ``PyInitError Py_PreInitializeFromArgs(const PyPreConfig *config,
int argc, wchar_t **argv)``
These functions can be called with *config* set to ``NULL``.
If Python is initialized with command line arguments, the command line
arguments must also be passed to pre-initialize Python, since they have
an effect on the pre-configuration like encodings. For example, the
``-X utf8`` command line option enables the UTF-8 Mode.
These functions can be called with *config* set to ``NULL``. The caller
is responsible to handle error or exit using ``Py_INIT_FAILED()`` and
``Py_ExitInitError()``.
``PyPreConfig`` fields:
* ``allocator`` (``char*``, default: ``NULL``):
Name of the memory allocator (ex: ``"malloc"``).
* ``coerce_c_locale`` (``int``, default: 0):
If equals to 2, coerce the C locale; if equals to 1, read the LC_CTYPE
locale to decide if it should be coerced.
* ``coerce_c_locale_warn`` (``int``, default: 0):
If non-zero, emit a warning if the C locale is coerced.
* ``dev_mode`` (``int``, default: 0):
See ``PyConfig.dev_mode``.
* ``isolated`` (``int``, default: 0):
See ``PyConfig.isolated``.
* ``legacy_windows_fs_encoding`` (``int``, Windows only, default: 0):
If non-zero, disable UTF-8 Mode, set the Python filesystem encoding to
``mbcs``, set the filesystem error handler to ``replace``.
* ``use_environment`` (``int``, default: 1):
See ``PyConfig.use_environment``.
* ``utf8_mode`` (``int``, default: 0):
If non-zero, enable the UTF-8 mode.
``PyPreConfig`` private field, for internal use only:
* ``_config_version`` (``int``, default: config version):
Configuration version, used for ABI compatibility.
The C locale coercion (PEP 538) and the UTF-8 Mode (PEP 540) are
disabled by default in ``PyPreConfig``. Set ``coerce_c_locale``,
``coerce_c_locale_warn`` and ``utf8_mode`` to ``-1`` to let Python
enable them depending on the user configuration. In this case, it's
safer to explicitly pre-initialize Python to ensure that encodings are
configured before the Python initialization starts. Example to get the
same encoding than regular Python::
PyPreConfig preconfig = PyPreConfig_INIT;
preconfig.coerce_c_locale = -1;
preconfig.coerce_c_locale_warn = -1;
preconfig.utf8_mode = -1;
PyInitError err = Py_PreInitialize(&preconfig);
if (Py_INIT_FAILED(err)) {
Py_ExitInitError(err);
}
Initialization with PyConfig
----------------------------
The ``PyConfig`` structure contains all parameters to configure Python.
Example setting the program name::
PyInitError err;
PyConfig config = PyConfig_INIT;
err = PyConfig_SetString(&config.program_name, L"my_program");
if (_Py_INIT_FAILED(err)) {
Py_ExitInitError(err);
}
err = Py_InitializeFromConfig(&config);
PyConfig_Clear(&config);
if (Py_INIT_FAILED(err)) {
Py_ExitInitError(err);
}
``PyConfig`` methods:
* ``PyInitError PyConfig_SetString(PyConfig *config, wchar_t
**config_str, const wchar_t *str)``:
Copy the wide character string *str* into ``*config_str``.
* ``PyInitError PyConfig_SetBytesString(PyConfig *config, wchar_t
**config_str, const char *str)``:
Decode *str* using ``Py_DecodeLocale()`` and set the result into
``*config_str``. Pre-initialize Python if needed to ensure that
encodings are properly configured.
* ``PyInitError PyConfig_SetArgv(PyConfig *config, int argc, wchar_t **argv)``:
Set command line arguments from wide character strings.
* ``PyInitError PyConfig_SetBytesArgv(PyConfig *config, int argc, char
**argv)``:
Set command line arguments: decode bytes using ``Py_DecodeLocale()``.
Pre-initialize Python if needed to ensure that encodings are properly
configured.
* ``PyInitError PyConfig_Read(PyConfig *config)``:
Read all Python configuration. Fields which are already set are left
unchanged.
* ``void PyConfig_Clear(PyConfig *config)``:
Release configuration memory.
Functions to initialize Python:
* ``PyInitError Py_InitializeFromConfig(const PyConfig *config)``:
Initialize Python from *config* configuration. *config* can be
``NULL``.
The caller of these methods and functions is responsible to handle
failure or exit using ``Py_INIT_FAILED()`` and ``Py_ExitInitError()``.
``PyConfig`` fields:
* ``argv`` (``PyWideStringList``, default: empty):
Command line arguments, ``sys.argv``.
It is parsed and updated by default, set ``parse_argv`` to 0 to avoid
that.
* ``base_exec_prefix`` (``wchar_t*``, default: ``NULL``):
``sys.base_exec_prefix``.
* ``base_prefix`` (``wchar_t*``, default: ``NULL``):
``sys.base_prefix``.
* ``buffered_stdio`` (``int``, default: 1):
If equals to 0, enable unbuffered mode, make stdout and stderr streams
to be unbuffered.
* ``bytes_warning`` (``int``, default: 0):
If equals to 1, issue a warning when comparing ``bytes`` or
``bytearray`` with ``str``, or comparing ``bytes`` with ``int``. If
equal or greater to 2, raise a ``BytesWarning`` exception.
* ``check_hash_pycs_mode`` (``wchar_t*``, default: ``"default"``):
``--check-hash-based-pycs`` command line option value (see PEP 552).
* ``configure_c_stdio`` (``int``, default: 1):
If non-zero, configure C standard streams (``stdio``, ``stdout``,
``stdout``). For example, set their mode to ``O_BINARY`` on Windows.
* ``dev_mode`` (``int``, default: 0):
Development mode
* ``dll_path`` (``wchar_t*``, Windows only, default: ``NULL``):
Windows DLL path.
* ``dump_refs`` (``int``, default: 0):
If non-zero, dump all objects which are still alive at exit
* ``exec_prefix`` (``wchar_t*``, default: ``NULL``):
``sys.exec_prefix``.
* ``executable`` (``wchar_t*``, default: ``NULL``):
``sys.executable``.
* ``faulthandler`` (``int``, default: 0):
If non-zero, call ``faulthandler.enable()``.
* ``filesystem_encoding`` (``wchar_t*``, default: ``NULL``):
Filesystem encoding, ``sys.getfilesystemencoding()``.
* ``filesystem_errors`` (``wchar_t*``, default: ``NULL``):
Filesystem encoding errors, ``sys.getfilesystemencodeerrors()``.
* ``use_hash_seed`` (``int``, default: 0),
``hash_seed`` (``unsigned long``, default: 0):
Randomized hash function seed.
* ``home`` (``wchar_t*``, default: ``NULL``):
Python home directory.
* ``import_time`` (``int``, default: 0):
If non-zero, profile import time.
* ``inspect`` (``int``, default: 0):
Enter interactive mode after executing a script or a command.
* ``install_signal_handlers`` (``int``, default: 1):
Install signal handlers?
* ``interactive`` (``int``, default: 0):
Interactive mode.
* ``legacy_windows_stdio`` (``int``, Windows only, default: 0):
If non-zero, use ``io.FileIO`` instead of ``WindowsConsoleIO`` for
``sys.stdin``, ``sys.stdout`` and ``sys.stderr``.
* ``malloc_stats`` (``int``, default: 0):
If non-zero, dump memory allocation statistics at exit.
* ``module_search_path_env`` (``wchar_t*``, default: ``NULL``):
``PYTHONPATH`` environment variale value.
* ``use_module_search_paths`` (``int``, default: 0),
``module_search_paths`` (``PyWideStringList``, default: empty):
``sys.path``.
* ``optimization_level`` (``int``, default: 0):
Compilation optimization level.
* ``parse_argv`` (``int``, default: 1):
If non-zero, parse ``argv`` command line arguments and update
``argv``.
* ``parser_debug`` (``int``, default: 0):
If non-zero, turn on parser debugging output (for expert only,
depending on compilation options).
* ``pathconfig_warnings`` (``int``, default: 1):
If equal to 0, suppress warnings when computing the path
configuration.
* ``prefix`` (``wchar_t*``, default: ``NULL``):
``sys.prefix``.
* ``program_name`` (``wchar_t*``, default: ``NULL``):
Program name.
* ``program`` (``wchar_t*``, default: ``NULL``):
``argv[0]`` or an empty string.
* ``pycache_prefix`` (``wchar_t*``, default: ``NULL``):
``.pyc`` cache prefix.
* ``quiet`` (``int``, default: 0):
Quiet mode. For example, don't display the copyright and version
messages even in interactive mode.
* ``run_command`` (``wchar_t*``, default: ``NULL``):
``-c COMMAND`` argument.
* ``run_filename`` (``wchar_t*``), default: ``NULL``:
``python3 SCRIPT`` argument.
* ``run_module`` (``wchar_t*``, default: ``NULL``):
``python3 -m MODULE`` argument.
* ``show_alloc_count`` (``int``, default: 0):
Show allocation counts at exit?
* ``show_ref_count`` (``int``, default: 0):
Show total reference count at exit?
* ``site_import`` (``int``, default: 1):
Import the ``site`` module at startup?
* ``skip_source_first_line`` (``int``, default: 0):
Skip the first line of the source?
* ``stdio_encoding`` (``wchar_t*``, default: ``NULL``),
``stdio_errors`` (``wchar_t*``, default: ``NULL``):
Encoding and encoding errors of ``sys.stdin``, ``sys.stdout``
and ``sys.stderr``.
* ``tracemalloc`` (``int``, default: 0):
If non-zero, call ``tracemalloc.start(value)``.
* ``user_site_directory`` (``int``, default: 1):
If non-zero, add user site directory to ``sys.path``.
* ``verbose`` (``int``, default: 0):
If non-zero, enable verbose mode.
* ``warnoptions`` (``PyWideStringList``, default: empty):
Options of the ``warnings`` module to build warnings filters.
* ``write_bytecode`` (``int``, default: 1):
If non-zero, write ``.pyc`` files.
* ``xoptions`` (``PyWideStringList``, default: empty):
``sys._xoptions``.
``PyConfig`` private fields, for internal use only:
* ``_config_version`` (``int``, default: config version):
Configuration version, used for ABI compatibility.
* ``_install_importlib`` (``int``, default: 1):
Install importlib?
* ``_init_main`` (``int``, default: 1):
If equal to 0, stop Python initialization before the "main" phase
(see PEP 432).
By default, the ``argv`` arguments are parsed as regular Python command
line arguments and ``argv`` is updated to strip parsed Python arguments:
see `Command Line Arguments`_. Set ``parse_argv`` to 0 to avoid parsing
and updating ``argv``. If ``argv`` is empty, an empty string is added to
ensure that ``sys.argv`` always exists and is never empty.
The ``xoptions`` options are parsed to set other options: see `-X
Options`_.
More complete example modifying the configuration before calling
``PyConfig_Read()``, and then modify the read configuration::
PyInitError init_python(const char *program_name)
{
PyInitError err;
PyConfig config = PyConfig_INIT;
/* Set the program name before reading the configuraton
(decode byte string from the locale encoding) */
err = PyConfig_SetBytesString(&config.program_name,
program_name);
if (_Py_INIT_FAILED(err)) {
goto fail;
}
/* Read all configuration at once */
err = PyConfig_Read(&config);
if (_Py_INIT_FAILED(err)) {
goto fail;
}
/* Append our custom search path to sys.path */
err = PyWideStringList_Append(&config.module_search_paths,
L"/path/to/more/modules");
if (_Py_INIT_FAILED(err)) {
goto fail;
}
/* Override executable computed by PyConfig_Read() */
err = PyConfig_SetString(&config, &config.executable, L"my_executable");
if (_Py_INIT_FAILED(err)) {
goto fail;
}
err = Py_InitializeFromConfig(&config);
/* Py_InitializeFromConfig() copied config which must now be
cleared to release memory */
PyConfig_Clear(&config);
return err;
fail:
PyConfig_Clear(&config);
Py_ExitInitError(err);
}
.. note::
``PyConfig`` does not have any field for extra inittab functions:
``PyImport_AppendInittab()`` and ``PyImport_ExtendInittab()``
functions are still relevant (and can be called before Python
initialization).
Initialization with constant PyConfig
-------------------------------------
When no ``PyConfig`` method is used but only
``Py_InitializeFromConfig()``, the caller is responsible for managing
``PyConfig`` memory. In that case, constant strings and constant string
lists can be used to avoid dynamically allocated memory. It can be used
for most simple configurations.
Example of Python initialization enabling the isolated mode::
PyConfig config = PyConfig_INIT;
config.isolated = 1;
PyInitError err = Py_InitializeFromConfig(&config);
if (Py_INIT_FAILED(err)) {
Py_ExitInitError(err);
}
/* ... use Python API here ... */
Py_Finalize();
``PyConfig_Clear()`` is not needed in this example since ``config`` does
not contain any dynamically allocated string:
``Py_InitializeFromConfig`` is responsible to fill other fields and
manage the memory.
For convenience, two other functions are provided for constant
``PyConfig``:
* ``PyInitError Py_InitializeFromArgs(const PyConfig *config, int
argc, wchar_t **argv)``
* ``PyInitError Py_InitializeFromBytesArgs(const PyConfig *config, int
argc, char **argv)``
They be called with *config* set to ``NULL``. The caller of these
functions is responsible to handle failure or exit using
``Py_INIT_FAILED()`` and ``Py_ExitInitError()``.
Path Configuration
------------------
``PyConfig`` contains multiple fields for the path configuration:
* Path configuration input fields:
* ``home``
* ``module_search_path_env``
* ``pathconfig_warnings``
* Path configuration output fields:
* ``dll_path`` (Windows only)
* ``exec_prefix``
* ``executable``
* ``prefix``
* ``use_module_search_paths``, ``module_search_paths``
Set ``pathconfig_warnings`` to 0 to suppress warnings when computing the
path configuration.
It is possible to completely ignore the function computing the default
path configuration by setting explicitly all path configuration output
fields listed above. A string is considered as set even if it's an empty
string. ``module_search_paths`` is considered as set if
``use_module_search_paths`` is set to 1. In this case, path
configuration input fields are ignored as well.
If ``base_prefix`` or ``base_exec_prefix`` fields are not set, they
inherit their value from ``prefix`` and ``exec_prefix`` respectively.
If ``site_import`` is non-zero, ``sys.path`` can be modified by the
``site`` module. For example, if ``user_site_directory`` is non-zero,
the user site directory is added to ``sys.path`` (if it exists).
Isolate Python
--------------
The default configuration is designed to behave as a regular Python.
To embed Python into an application, it's possible to tune the
configuration to better isolated the embedded Python from the system:
* Set ``isolated`` to 1 to ignore environment variables and not prepend
the current directory to ``sys.path``.
* Set the `Path Configuration`_ ("output fields") to ignore the function
computing the default path configuration.
Py_BytesMain()
--------------
Python 3.7 provides a high-level ``Py_Main()`` function which requires
to pass command line arguments as ``wchar_t*`` strings. It is
non-trivial to use the correct encoding to decode bytes. Python has its
own set of issues with C locale coercion and UTF-8 Mode.
This PEP adds a new ``Py_BytesMain()`` function which takes command line
arguments as bytes::
int Py_BytesMain(int argc, char **argv)
Py_RunMain()
------------
The new ``Py_RunMain()`` function executes the command
(``PyConfig.run_command``), the script (``PyConfig.run_filename``) or
the module (``PyConfig.run_module``) specified on the command line or in
the configuration, and then finalizes Python. It returns an exit status
that can be passed to the ``exit()`` function.
Example of customized Python in isolated mode::
#include <Python.h>
int main(int argc, char *argv[])
{
PyConfig config = PyConfig_INIT;
config.isolated = 1;
PyInitError err = Py_InitializeFromBytesArgs(&config, argc, argv);
if (Py_INIT_FAILED(err)) {
Py_ExitInitError(err);
}
/* put more configuration code here if needed */
return Py_RunMain();
}
The example is a basic implementation of the "System Python Executable"
discussed in PEP 432.
Memory allocations and Py_DecodeLocale()
----------------------------------------
Python memory allocation functions like ``PyMem_RawMalloc()`` must not
be used before Python pre-initialization, whereas calling directly
``malloc()`` and ``free()`` is always safe.
For ``PyPreConfig`` and constant ``PyConfig``, the caller is responsible
to manage dynamically allocated memory; constant strings and constant
string lists can be used to avoid memory allocations.
Dynamic ``PyConfig`` requires to call ``PyConfig_Clear()`` to release
memory.
``Py_DecodeLocale()`` must not be called before the pre-initialization.
Backwards Compatibility
=======================
This PEP only adds a new API: it leaves the existing API unchanged and
has no impact on the backwards compatibility.
The implementation ensures that the existing API is compatible with the
new API. For example, ``PyConfig`` uses the value of global
configuration variables as default values.
Annex: Python Configuration
===========================
Priority and Rules
------------------
Priority of configuration parameters, highest to lowest:
* ``PyConfig``
* ``PyPreConfig``
* Configuration files
* Command line options
* Environment variables
* Global configuration variables
Priority of warning options, highest to lowest:
* ``PyConfig.warnoptions``
* ``PyConfig.dev_mode`` (add ``"default"``)
* ``PYTHONWARNINGS`` environment variables
* ``-W WARNOPTION`` command line argument
* ``PyConfig.bytes_warning`` (add ``"error::BytesWarning"`` if greater
than 1, or add ``"default::BytesWarning``)
Rules on ``PyConfig`` parameters:
* If ``isolated`` is non-zero, ``use_environment`` and
``user_site_directory`` are set to 0.
* If ``legacy_windows_fs_encoding`` is non-zero, ``utf8_mode`` is set to
0.
* If ``dev_mode`` is non-zero, ``allocator`` is set to ``"debug"``,
``faulthandler`` is set to 1, and ``"default"`` filter is added to
``warnoptions``. But the ``PYTHONMALLOC`` environment variable has the
priority over ``dev_mode`` to set the memory allocator.
* If ``base_prefix`` is not set, it inherits ``prefix`` value.
* If ``base_exec_prefix`` is not set, it inherits ``exec_prefix`` value.
* If the ``python._pth`` configuration file is present, ``isolated`` is
set to 1 and ``site_import`` is set to 0; but ``site_import`` is set
to 1 if ``python._pth`` contains ``import site``.
Rules on ``PyConfig`` and ``PyPreConfig`` parameters:
* If ``PyPreConfig.legacy_windows_fs_encoding`` is non-zero,
set ``PyConfig.utf8_mode`` to 0, set ``PyConfig.filesystem_encoding``
to ``mbcs``, and set ``PyConfig.filesystem_errors`` to ``replace``.
Configuration Files
-------------------
Python configuration files:
* ``pyvenv.cfg``
* ``python._pth`` (Windows only)
* ``pybuilddir.txt`` (Unix only)
Global Configuration Variables
------------------------------
Global configuration variables mapped to ``PyPreConfig`` fields:
======================================== ================================
Variable Field
======================================== ================================
``Py_IgnoreEnvironmentFlag`` ``use_environment`` (NOT)
``Py_IsolatedFlag`` ``isolated``
``Py_LegacyWindowsFSEncodingFlag`` ``legacy_windows_fs_encoding``
``Py_UTF8Mode`` ``utf8_mode``
======================================== ================================
(NOT) means that the ``PyPreConfig`` value is the oposite of the global
configuration variable value.
Global configuration variables mapped to ``PyConfig`` fields:
======================================== ================================
Variable Field
======================================== ================================
``Py_BytesWarningFlag`` ``bytes_warning``
``Py_DebugFlag`` ``parser_debug``
``Py_DontWriteBytecodeFlag`` ``write_bytecode`` (NOT)
``Py_FileSystemDefaultEncodeErrors`` ``filesystem_errors``
``Py_FileSystemDefaultEncoding`` ``filesystem_encoding``
``Py_FrozenFlag`` ``pathconfig_warnings`` (NOT)
``Py_HasFileSystemDefaultEncoding`` ``filesystem_encoding``
``Py_HashRandomizationFlag`` ``use_hash_seed``, ``hash_seed``
``Py_IgnoreEnvironmentFlag`` ``use_environment`` (NOT)
``Py_InspectFlag`` ``inspect``
``Py_InteractiveFlag`` ``interactive``
``Py_IsolatedFlag`` ``isolated``
``Py_LegacyWindowsStdioFlag`` ``legacy_windows_stdio``
``Py_NoSiteFlag`` ``site_import`` (NOT)
``Py_NoUserSiteDirectory`` ``user_site_directory`` (NOT)
``Py_OptimizeFlag`` ``optimization_level``
``Py_QuietFlag`` ``quiet``
``Py_UnbufferedStdioFlag`` ``buffered_stdio`` (NOT)
``Py_VerboseFlag`` ``verbose``
``_Py_HasFileSystemDefaultEncodeErrors`` ``filesystem_errors``
======================================== ================================
(NOT) means that the ``PyConfig`` value is the oposite of the global
configuration variable value.
``Py_LegacyWindowsFSEncodingFlag`` and ``Py_LegacyWindowsStdioFlag`` are
only available on Windows.
Command Line Arguments
----------------------
Usage::
python3 [options]
python3 [options] -c COMMAND
python3 [options] -m MODULE
python3 [options] SCRIPT
Command line options mapped to pseudo-action on ``PyPreConfig`` fields:
================================ ================================
Option ``PyConfig`` field
================================ ================================
``-E`` ``use_environment = 0``
``-I`` ``isolated = 1``
``-X dev`` ``dev_mode = 1``
``-X utf8`` ``utf8_mode = 1``
``-X utf8=VALUE`` ``utf8_mode = VALUE``
================================ ================================
Command line options mapped to pseudo-action on ``PyConfig`` fields:
================================ ================================
Option ``PyConfig`` field
================================ ================================
``-b`` ``bytes_warning++``
``-B`` ``write_bytecode = 0``
``-c COMMAND`` ``run_command = COMMAND``
``--check-hash-based-pycs=MODE`` ``_check_hash_pycs_mode = MODE``
``-d`` ``parser_debug++``
``-E`` ``use_environment = 0``
``-i`` ``inspect++`` and ``interactive++``
``-I`` ``isolated = 1``
``-m MODULE`` ``run_module = MODULE``
``-O`` ``optimization_level++``
``-q`` ``quiet++``
``-R`` ``use_hash_seed = 0``
``-s`` ``user_site_directory = 0``
``-S`` ``site_import``
``-t`` ignored (kept for backwards compatibility)
``-u`` ``buffered_stdio = 0``
``-v`` ``verbose++``
``-W WARNING`` add ``WARNING`` to ``warnoptions``
``-x`` ``skip_source_first_line = 1``
``-X OPTION`` add ``OPTION`` to ``xoptions``
================================ ================================
``-h``, ``-?`` and ``-V`` options are handled without ``PyConfig``.
-X Options
----------
-X options mapped to pseudo-action on ``PyConfig`` fields:
================================ ================================
Option ``PyConfig`` field
================================ ================================
``-X dev`` ``dev_mode = 1``
``-X faulthandler`` ``faulthandler = 1``
``-X importtime`` ``import_time = 1``
``-X pycache_prefix=PREFIX`` ``pycache_prefix = PREFIX``
``-X showalloccount`` ``show_alloc_count = 1``
``-X showrefcount`` ``show_ref_count = 1``
``-X tracemalloc=N`` ``tracemalloc = N``
================================ ================================
Environment Variables
---------------------
Environment variables mapped to ``PyPreConfig`` fields:
================================= =============================================
Variable ``PyPreConfig`` field
================================= =============================================
``PYTHONCOERCECLOCALE`` ``coerce_c_locale``, ``coerce_c_locale_warn``
``PYTHONDEVMODE`` ``dev_mode``
``PYTHONLEGACYWINDOWSFSENCODING`` ``legacy_windows_fs_encoding``
``PYTHONMALLOC`` ``allocator``
``PYTHONUTF8`` ``utf8_mode``
================================= =============================================
Environment variables mapped to ``PyConfig`` fields:
================================= ====================================
Variable ``PyConfig`` field
================================= ====================================
``PYTHONDEBUG`` ``parser_debug``
``PYTHONDEVMODE`` ``dev_mode``
``PYTHONDONTWRITEBYTECODE`` ``write_bytecode``
``PYTHONDUMPREFS`` ``dump_refs``
``PYTHONEXECUTABLE`` ``program_name``
``PYTHONFAULTHANDLER`` ``faulthandler``
``PYTHONHASHSEED`` ``use_hash_seed``, ``hash_seed``
``PYTHONHOME`` ``home``
``PYTHONINSPECT`` ``inspect``
``PYTHONIOENCODING`` ``stdio_encoding``, ``stdio_errors``
``PYTHONLEGACYWINDOWSSTDIO`` ``legacy_windows_stdio``
``PYTHONMALLOCSTATS`` ``malloc_stats``
``PYTHONNOUSERSITE`` ``user_site_directory``
``PYTHONOPTIMIZE`` ``optimization_level``
``PYTHONPATH`` ``module_search_path_env``
``PYTHONPROFILEIMPORTTIME`` ``import_time``
``PYTHONPYCACHEPREFIX,`` ``pycache_prefix``
``PYTHONTRACEMALLOC`` ``tracemalloc``
``PYTHONUNBUFFERED`` ``buffered_stdio``
``PYTHONVERBOSE`` ``verbose``
``PYTHONWARNINGS`` ``warnoptions``
================================= ====================================
``PYTHONLEGACYWINDOWSFSENCODING`` and ``PYTHONLEGACYWINDOWSSTDIO`` are
specific to Windows.
Annex: Python 3.7 API
=====================
Python 3.7 has 4 functions in its C API to initialize and finalize
Python:
* ``Py_Initialize()``, ``Py_InitializeEx()``: initialize Python
* ``Py_Finalize()``, ``Py_FinalizeEx()``: finalize Python
Python 3.7 can be configured using `Global Configuration Variables`_,
`Environment Variables`_, and the following functions:
* ``PyImport_AppendInittab()``
* ``PyImport_ExtendInittab()``
* ``PyMem_SetAllocator()``
* ``PyMem_SetupDebugHooks()``
* ``PyObject_SetArenaAllocator()``
* ``Py_SetPath()``
* ``Py_SetProgramName()``
* ``Py_SetPythonHome()``
* ``Py_SetStandardStreamEncoding()``
* ``PySys_AddWarnOption()``
* ``PySys_AddXOption()``
* ``PySys_ResetWarnOptions()``
There is also a high-level ``Py_Main()`` function.
Python Issues
=============
Issues that will be fixed by this PEP, directly or indirectly:
* `bpo-1195571 <https://bugs.python.org/issue1195571>`_: "simple
callback system for Py_FatalError"
* `bpo-11320 <https://bugs.python.org/issue11320>`_:
"Usage of API method Py_SetPath causes errors in Py_Initialize()
(Posix ony)"
* `bpo-13533 <https://bugs.python.org/issue13533>`_: "Would like
Py_Initialize to play friendly with host app"
* `bpo-14956 <https://bugs.python.org/issue14956>`_: "custom PYTHONPATH
may break apps embedding Python"
* `bpo-19983 <https://bugs.python.org/issue19983>`_: "When interrupted
during startup, Python should not call abort() but exit()"
* `bpo-22213 <https://bugs.python.org/issue22213>`_: "Make pyvenv style
virtual environments easier to configure when embedding Python". This
PEP more or
* `bpo-22257 <https://bugs.python.org/issue22257>`_: "PEP 432: Redesign
the interpreter startup sequence"
* `bpo-29778 <https://bugs.python.org/issue29778>`_: "_Py_CheckPython3
uses uninitialized dllpath when embedder sets module path with
Py_SetPath"
* `bpo-30560 <https://bugs.python.org/issue30560>`_: "Add
Py_SetFatalErrorAbortFunc: Allow embedding program to handle fatal
errors".
* `bpo-31745 <https://bugs.python.org/issue31745>`_: "Overloading
"Py_GetPath" does not work"
* `bpo-32573 <https://bugs.python.org/issue32573>`_: "All sys attributes
(.argv, ...) should exist in embedded environments".
* `bpo-34725 <https://bugs.python.org/issue34725>`_:
"Py_GetProgramFullPath() odd behaviour in Windows"
* `bpo-36204 <https://bugs.python.org/issue36204>`_: "Deprecate calling
Py_Main() after Py_Initialize()? Add Py_InitializeFromArgv()?"
* `bpo-33135 <https://bugs.python.org/issue33135>`_: "Define field
prefixes for the various config structs". The PEP now defines well
how warnings options are handled.
Issues of the PEP implementation:
* `bpo-16961 <https://bugs.python.org/issue16961>`_: "No regression
tests for -E and individual environment vars"
* `bpo-20361 <https://bugs.python.org/issue20361>`_: "-W command line
options and PYTHONWARNINGS environmental variable should not override
-b / -bb command line options"
* `bpo-26122 <https://bugs.python.org/issue26122>`_: "Isolated mode
doesn't ignore PYTHONHASHSEED"
* `bpo-29818 <https://bugs.python.org/issue29818>`_:
"Py_SetStandardStreamEncoding leads to a memory error in debug mode"
* `bpo-31845 <https://bugs.python.org/issue31845>`_:
"PYTHONDONTWRITEBYTECODE and PYTHONOPTIMIZE have no effect"
* `bpo-32030 <https://bugs.python.org/issue32030>`_: "PEP 432: Rewrite
Py_Main()"
* `bpo-32124 <https://bugs.python.org/issue32124>`_: "Document functions
safe to be called before Py_Initialize()"
* `bpo-33042 <https://bugs.python.org/issue33042>`_: "New 3.7 startup
sequence crashes PyInstaller"
* `bpo-33932 <https://bugs.python.org/issue33932>`_: "Calling
Py_Initialize() twice now triggers a fatal error (Python 3.7)"
* `bpo-34008 <https://bugs.python.org/issue34008>`_: "Do we support
calling Py_Main() after Py_Initialize()?"
* `bpo-34170 <https://bugs.python.org/issue34170>`_: "Py_Initialize():
computing path configuration must not have side effect (PEP 432)"
* `bpo-34589 <https://bugs.python.org/issue34589>`_: "Py_Initialize()
and Py_Main() should not enable C locale coercion"
* `bpo-34639 <https://bugs.python.org/issue34639>`_:
"PYTHONCOERCECLOCALE is ignored when using -E or -I option"
* `bpo-36142 <https://bugs.python.org/issue36142>`_: "Add a new
_PyPreConfig step to Python initialization to setup memory allocator
and encodings"
* `bpo-36202 <https://bugs.python.org/issue36202>`_: "Calling
Py_DecodeLocale() before _PyPreConfig_Write() can produce mojibake"
* `bpo-36301 <https://bugs.python.org/issue36301>`_: "Add
_Py_PreInitialize() function"
* `bpo-36443 <https://bugs.python.org/issue36443>`_: "Disable
coerce_c_locale and utf8_mode by default in _PyPreConfig?"
* `bpo-36444 <https://bugs.python.org/issue36444>`_: "Python
initialization: remove _PyMainInterpreterConfig"
* `bpo-36471 <https://bugs.python.org/issue36471>`_: "PEP 432, PEP 587:
Add _Py_RunMain()"
* `bpo-36763 <https://bugs.python.org/issue36763>`_: "PEP 587: Rework
initialization API to prepare second version of the PEP"
* `bpo-36775 <https://bugs.python.org/issue36775>`_: "Rework filesystem
codec implementation"
* `bpo-36900 <https://bugs.python.org/issue36900>`_: "Use _PyCoreConfig
rather than global configuration variables"
Issues related to this PEP:
* `bpo-12598 <https://bugs.python.org/issue12598>`_: "Move sys variable
initialization from import.c to sysmodule.c"
* `bpo-15577 <https://bugs.python.org/issue15577>`_: "Real argc and argv
in embedded interpreter"
* `bpo-16202 <https://bugs.python.org/issue16202>`_: "sys.path[0]
security issues"
* `bpo-18309 <https://bugs.python.org/issue18309>`_: "Make python
slightly more relocatable"
* `bpo-25631 <https://bugs.python.org/issue25631>`_: "Segmentation fault
with invalid Unicode command-line arguments in embedded Python"
* `bpo-26007 <https://bugs.python.org/issue26007>`_: "Support embedding
the standard library in an executable"
* `bpo-31210 <https://bugs.python.org/issue31210>`_: "Can not import
modules if sys.prefix contains DELIM".
* `bpo-31349 <https://bugs.python.org/issue31349>`_: "Embedded
initialization ignores Py_SetProgramName()"
* `bpo-33919 <https://bugs.python.org/issue33919>`_: "Expose
_PyCoreConfig structure to Python"
* `bpo-35173 <https://bugs.python.org/issue35173>`_: "Re-use already
existing functionality to allow Python 2.7.x (both embedded and
standalone) to locate the module path according to the shared library"
Version History
===============
* Version 3:
* ``PyConfig``: Add ``configure_c_stdio`` and ``parse_argv``,
rename ``_frozen`` to ``pathconfig_warnings``.
* Rename functions using bytes strings and wide character strings. For
example, ``Py_PreInitializeFromWideArgs`` becomes
``Py_PreInitializeFromArgs``, and ``PyConfig_SetArgv`` becomes
``PyConfig_SetBytesArgv``.
* Add ``PyWideStringList_Insert()`` function.
* New "Path configuration", "Isolate Python", "Python Issues"
and "Version History" sections.
* ``PyConfig_SetString()`` and ``PyConfig_SetBytesString()`` now
requires the configuration as the first argument.
* Rename ``Py_UnixMain()`` to ``Py_BytesMain()``
* Version 2: Add ``PyConfig`` methods (ex: ``PyConfig_Read()``), add
``PyWideStringList_Append()``, rename ``PyWideCharList`` to
``PyWideStringList``.
* Version 1: Initial version.
Copyright
=========
This document has been placed in the public domain.
[View Less]
5
11