Hi,
I have updated the PEP with feedback from discussions. The highlights are:
* Deprecate parser module * Keep fileinput module * Elaborate why crypt and spwd are dangerous and bad * Improve sections for cgitb, colorsys, nntplib, and smtpd modules * The colorsys, crypt, imghdr, sndhdr, and spwd sections now list suitable substitutions. * Mention that socketserver is going to stay for http.server and xmlrpc.server * The future maintenance section now states that the deprecated modules may be adopted by Python community members. https://github.com/python/peps/compare/7799178a...2d536899?diff=unified#diff...
I'll be traveling the next couple of days and will only have limited opportunities to respond on feedback.
Christian
------------------------------------------------------------------- PEP: 594 Title: Removing dead batteries from the standard library Author: Christian Heimes christian@python.org Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 20-May-2019 Post-History: 21-May-2019
Abstract ========
This PEP proposed a list of standard library modules to be removed from the standard library. The modules are mostly historic data formats and APIs that have been superseded a long time ago, e.g. Mac OS 9 and Commodore.
Rationale =========
Back in the early days of Python, the interpreter came with a large set of useful modules. This was often refrained to as "batteries included" philosophy and was one of the corner stones to Python's success story. Users didn't have to figure out how to download and install separate packages in order to write a simple web server or parse email.
Times have changed. The introduction of the cheese shop (PyPI), setuptools, and later pip, it became simple and straight forward to download and install packages. Nowadays Python has a rich and vibrant ecosystem of third party packages. It's pretty much standard to either install packages from PyPI or use one of the many Python or Linux distributions.
On the other hand, Python's standard library is piling up cruft, unnecessary duplication of functionality, and dispensable features. This is undesirable for several reasons.
* Any additional module increases the maintenance cost for the Python core development team. The team has limited resources, reduced maintenance cost frees development time for other improvements. * Modules in the standard library are generally favored and seen as the de-facto solution for a problem. A majority of users only pick 3rd party modules to replace a stdlib module, when they have a compelling reason, e.g. lxml instead of `xml`. The removal of an unmaintained stdlib module increases the chances of a community contributed module to become widely used. * A lean and mean standard library benefits platforms with limited resources like devices with just a few hundred kilobyte of storage (e.g. BBC Micro:bit). Python on mobile platforms like BeeWare or WebAssembly (e.g. pyodide) also benefit from reduced download size.
The modules in the PEP have been selected for deprecation because their removal is either least controversial or most beneficial. For example least controversial are 30 years old multimedia formats like ``sunau`` audio format, which was used on SPARC and NeXT workstations in the late 1980ties. The ``crypt`` module has fundamental flaws that are better solved outside the standard library.
This PEP also designates some modules as not scheduled for removal. Some modules have been deprecated for several releases or seem unnecessary at first glance. However it is beneficial to keep the modules in the standard library, mostly for environments where installing a package from PyPI is not an option. This can be cooperate environments or class rooms where external code is not permitted without legal approval.
* The usage of FTP is declining, but some files are still provided over the FTP protocol or hosters offer FTP to upload content. Therefore ``ftplib`` is going to stay. * The ``optparse`` and ``getopt`` module are widely used. They are mature modules with very low maintenance overhead. * According to David Beazley [5]_ the ``wave`` module is easy to teach to kids and can make crazy sounds. Making a computer generate crazy sounds is powerful and highly motivating exercise for a 9yo aspiring developer. It's a fun battery to keep.
Deprecation schedule ====================
3.8 ---
This PEP targets Python 3.8. Version 3.8.0 final is scheduled to be released a few months before Python 2.7 will reach its end of lifetime. We expect that Python 3.8 will be targeted by users that migrate to Python 3 in 2019 and 2020. To reduce churn and to allow a smooth transition from Python 2, Python 3.8 will neither raise `DeprecationWarning` nor remove any modules that have been scheduled for removal. Instead deprecated modules will just be *documented* as deprecated. Optionally modules may emit a `PendingDeprecationWarning`.
All deprecated modules will also undergo a feature freeze. No additional features should be added. Bug should still be fixed.
3.9 ---
Starting with Python 3.9, deprecated modules will start issuing `DeprecationWarning`. The `parser`_ module is removed and potentially replaced with a new module.
All other deprecated modules are fully supported and will receive security updates until Python 3.9 reaches its end of lifetime. Python 3.9.0 will be released about 18 months after 3.8.0 (April 2021?) and most likely be supported for 5 years after the release. The estimated EOL of Python 3.9 is in 2026.
3.10 ----
In 3.10 all deprecated modules will be removed from the CPython repository together with tests, documentation, and autoconf rules.
PEP acceptance process ======================
3.8.0b1 is scheduled to be release shortly after the PEP is officially submitted. Since it's improbable that the PEP will pass all stages of the PEP process in time, I propose a two step acceptance process that is analogous Python's two release deprecation process.
The first *provisionally accepted* phase targets Python 3.8.0b1. In the first phase no code is changes or removed. Modules are only documented as deprecated. The only exception is the `parser`_ module. It has been documented as deprecated since Python 2.5 and is scheduled for removal for 3.9 to make place for a more advanced parser.
The final decision, which modules will be removed and how the removed code is preserved, can be delayed for another year.
Deprecated modules ==================
The modules are grouped as data encoding, multimedia, network, OS interface, and misc modules. The majority of modules are for old data formats or old APIs. Some others are rarely useful and have better replacements on PyPI, e.g. Pillow for image processing or NumPy-based projects to deal with audio processing.
.. csv-table:: Table 1: Proposed modules deprecations :header: "Module", "Deprecated in", "To be removed", "Replacement" :widths: 1, 1, 1, 2
aifc,3.8,3.10,- asynchat,3.8,3.10,asyncio asyncore,3.8,3.10,asyncio audioop,3.8,3.10,- binhex,3.8,3.10,- cgi,3.8,3.10,- cgitb,3.8,3.10,- chunk,3.8,3.10,- colorsys,3.8,3.10,"colormath, colour, colorspacious, Pillow" crypt,3.8,3.10,"bcrypt, argon2cffi, hashlib, passlib" fileinput,-,**keep**,argparse formatter,3.4,3.10,- fpectl,**3.7**,**3.7**,- getopt,**3.2**,**keep**,"argparse, optparse" imghdr,3.8,3.10,"filetype, puremagic, python-magic" imp,**3.4**,3.10,importlib lib2to3,-,**keep**, macpath,**3.7**,**3.8**,- msilib,3.8,3.10,- nntplib,3.8,3.10,- nis,3.8,3.10,- optparse,-,**keep**,argparse ossaudiodev,3.8,3.10,- parser,**2.5**,**3.9**,"ast, lib2to3.pgen2" pipes,3.8,3.10,subprocess smtpd,"**3.4.7**, **3.5.4**",3.10,aiosmtpd sndhdr,3.8,3.10,"filetype, puremagic, python-magic" spwd,3.8,3.10,"python-pam, simplepam" sunau,3.8,3.10,- uu,3.8,3.10,- wave,-,**keep**, xdrlib,3.8,3.10,-
Data encoding modules ---------------------
binhex ~~~~~~
The `binhex https://docs.python.org/3/library/binhex.html`_ module encodes and decodes Apple Macintosh binhex4 data. It was originally developed for TSR-80. In the 1980s and early 1990s it was used on classic Mac OS 9 to encode binary email attachments.
Module type pure Python Deprecated in 3.8 To be removed in 3.10 Substitute **none**
uu ~~
The `uu https://docs.python.org/3/library/uu.html`_ module provides uuencode format, an old binary encoding format for email from 1980. The uu format has been replaced by MIME. The uu codec is provided by the binascii module.
Module type pure Python Deprecated in 3.8 To be removed in 3.10 Substitute **none**
xdrlib ~~~~~~
The `xdrlib https://docs.python.org/3/library/xdrlib.html`_ module supports the Sun External Data Representation Standard. XDR is an old binary serialization format from 1987. These days it's rarely used outside specialized domains like NFS.
Module type pure Python Deprecated in 3.8 To be removed in 3.10 Substitute **none**
Multimedia modules ------------------
aifc ~~~~
The `aifc https://docs.python.org/3/library/aifc.html`_ module provides support for reading and writing AIFF and AIFF-C files. The Audio Interchange File Format is an old audio format from 1988 based on Amiga IFF. It was most commonly used on the Apple Macintosh. These days only few specialized application use AIFF.
Module type pure Python (depends on `audioop`_ C extension) Deprecated in 3.8 To be removed in 3.10 Substitute **none**
audioop ~~~~~~~
The `audioop https://docs.python.org/3/library/audioop.html`_ module contains helper functions to manipulate raw audio data and adaptive differential pulse-code modulated audio data. The module is implemented in C without any additional dependencies. The `aifc`_, `sunau`_, and `wave`_ module depend on `audioop`_ for some operations. The byteswap operation in the `wave`_ module can be substituted with little work.
Module type C extension Deprecated in 3.8 To be removed in 3.10 Substitute **none**
colorsys ~~~~~~~~
The `colorsys https://docs.python.org/3/library/colorsys.html`_ module defines color conversion functions between RGB, YIQ, HSL, and HSV coordinate systems.
The PyPI packages *colormath*, *colour*, and *colorspacious* provide more and advanced features. The Pillow library is better suited to transform images between color systems.
Module type pure Python Deprecated in 3.8 To be removed in 3.10 Substitute `colormath https://pypi.org/project/colormath/`_, `colour https://pypi.org/project/colour/`_ `colorspacious https://pypi.org/project/colorspacious/`_, `Pillow https://pypi.org/project/Pillow/`_
chunk ~~~~~
The `chunk https://docs.python.org/3/library/chunk.html`_ module provides support for reading and writing Electronic Arts' Interchange File Format. IFF is an old audio file format originally introduced for Commodore and Amiga. The format is no longer relevant.
Module type pure Python Deprecated in 3.8 To be removed in 3.10 Substitute **none**
imghdr ~~~~~~
The `imghdr https://docs.python.org/3/library/imghdr.html`_ module is a simple tool to guess the image file format from the first 32 bytes of a file or buffer. It supports only a limited amount of formats and neither returns resolution nor color depth.
Module type pure Python Deprecated in 3.8 To be removed in 3.10 Substitute `puremagic https://pypi.org/project/puremagic/`_, `filetype https://pypi.org/project/filetype/`_, `python-magic https://pypi.org/project/python-magic/`_
ossaudiodev ~~~~~~~~~~~
The `ossaudiodev https://docs.python.org/3/library/ossaudiodev.html`_ module provides support for Open Sound System, an interface to sound playback and capture devices. OSS was initially free software, but later support for newer sound devices and improvements were proprietary. Linux community abandoned OSS in favor of ALSA [1]_. Some operation systems like OpenBSD and NetBSD provide an incomplete [2]_ emulation of OSS.
Module type C extension Deprecated in 3.8 To be removed in 3.10 Substitute **none**
sndhdr ~~~~~~
The `sndhdr https://docs.python.org/3/library/sndhdr.html`_ module is similar to the `imghdr`_ module but for audio formats. It guesses file format, channels, frame rate, and sample widths from the first 512 bytes of a file or buffer. The module only supports AU, AIFF, HCOM, VOC, WAV, and other ancient formats.
Module type pure Python (depends on `audioop`_ C extension for some operations) Deprecated in 3.8 To be removed in 3.10 Substitute `puremagic https://pypi.org/project/puremagic/`_, `filetype https://pypi.org/project/filetype/`_, `python-magic https://pypi.org/project/python-magic/`_
sunau ~~~~~
The `sunau https://docs.python.org/3/library/sunhdr.html`_ module provides support for Sun AU sound format. It's yet another old, obsolete file format.
Module type pure Python (depends on `audioop`_ C extension for some operations) Deprecated in 3.8 To be removed in 3.10 Substitute **none**
Networking modules ------------------
asynchat ~~~~~~~~
The `asynchat https://docs.python.org/3/library/asynchat.html`_ module is build on top of `asyncore`_ and has been deprecated since Python 3.6.
Module type pure Python Deprecated in 3.6 Removed in 3.10 Substitute asyncio
asyncore ~~~~~~~~
The `asyncore https://docs.python.org/3/library/asyncore.html`_ module was the first module for asynchronous socket service clients and servers. It has been replaced by asyncio and is deprecated since Python 3.6.
The ``asyncore`` module is also used in stdlib tests. The tests for ``ftplib``, ``logging``, ``smptd``, ``smtplib``, and ``ssl`` are partly based on ``asyncore``. These tests must be updated to use asyncio or threading.
Module type pure Python Deprecated in 3.6 Removed in 3.10 Substitute asyncio
cgi ~~~
The `cgi https://docs.python.org/3/library/cgi.html`_ module is a support module for Common Gateway Interface (CGI) scripts. CGI is deemed as inefficient because every incoming request is handled in a new process. PEP 206 considers the module as *designed poorly and are now near-impossible to fix*.
Several people proposed to either keep the cgi module for features like `cgi.parse_qs()` or move `cgi.escape()` to a different module. The functions `cgi.parse_qs` and `cgi.parse_qsl` have been deprecated for a while and are actually aliases for `urllib.parse.parse_qs` and `urllib.parse.parse_qsl`. The function `cgi.quote` has been deprecated in favor of `html.quote` with secure default values.
Module type pure Python Deprecated in 3.8 To be removed in 3.10 Substitute **none**
cgitb ~~~~~
The `cgitb https://docs.python.org/3/library/cgitb.html`_ module is a helper for the cgi module for configurable tracebacks.
The ``cgitb`` module is not used by any major Python web framework (Django, Pyramid, Plone, Flask, CherryPy, or Bottle). Only Paste uses it in an optional debugging middleware.
Module type pure Python Deprecated in 3.8 To be removed in 3.10 Substitute **none**
smtpd ~~~~~
The `smtpd https://docs.python.org/3/library/smtpd.html`_ module provides a simple implementation of a SMTP mail server. The module documentation marks the module as deprecated and recommends ``aiosmtpd`` instead. The deprecation message was added in releases 3.4.7, 3.5.4, and 3.6.1.
Module type pure Python Deprecated in **3.7** To be removed in 3.10 Substitute aiosmtpd
nntplib ~~~~~~~
The `nntplib https://docs.python.org/3/library/nntplib.html`_ module implements the client side of the Network News Transfer Protocol (nntp). News groups used to be a dominant platform for online discussions. Over the last two decades, news has been slowly but steadily replaced with mailing lists and web-based discussion platforms. Twisted is also `planning https://twistedmatrix.com/trac/ticket/9405`_ to deprecate NNTP support and `pynnt https://github.com/greenbender/pynntp`_ hasn't seen any activity since 2014. This is a good indicator that the public interest in NNTP support is declining.
The ``nntplib`` tests have been the cause of additional work in the recent past. Python only contains client side of NNTP. The tests connect to external news server. The servers are sometimes unavailble, too slow, or do not work correctly over IPv6. The situation causes flaky test runs on buildbots.
Module type pure Python Deprecated in 3.8 To be removed in 3.10 Substitute **none**
Operating system interface --------------------------
crypt ~~~~~
The `crypt https://docs.python.org/3/library/crypt.html`_ module implements password hashing based on ``crypt(3)`` function from ``libcrypt`` or ``libxcrypt`` on Unix-like platform. The algorithms are mostly old, of poor quality and insecure. Users are discouraged to use them.
* The module is not available on Windows. Cross-platform application need an alternative implementation any way. * Only DES encryption is guarenteed to be available. DES has an extremely limited key space of 2**56. * MD5, salted SHA256, salted SHA512, and Blowfish are optional extension. SSHA256 and SSHA512 are glibc extensions. Blowfish (bcrypt) is the only algorithm that is still secure. However it's in glibc and therefore not commonly available on Linux. * Depending on the platform, the ``crypt`` module is not thread safe. Only implementations with ``crypt_r(3)`` are thread safe. * The module was never useful to interact with system user and password databases. On BSD, macOS, and Linux, all user authentication and password modification operations must go through PAM (pluggable authentication module), see `spwd`_ deprecation.
Module type C extension + Python module Deprecated in 3.8 To be removed in 3.10 Substitute `bcrypt https://pypi.org/project/bcrypt/`_, `passlib https://pypi.org/project/passlib/`_, `argon2cffi https://pypi.org/project/argon2-cffi/`_, hashlib module (PBKDF2, scrypt)
macpath ~~~~~~~
The `macpath https://docs.python.org/3/library/macpath.html`_ module provides Mac OS 9 implementation of os.path routines. Mac OS 9 is no longer supported
Module type pure Python Deprecated in 3.7 Removed in 3.8 Substitute **none**
nis ~~~
The `nis https://docs.python.org/3/library/nis.html`_ module provides NIS/YP support. Network Information Service / Yellow Pages is an old and deprecated directory service protocol developed by Sun Microsystems. It's designed successor NIS+ from 1992 never took off. For a long time, libc's Name Service Switch, LDAP, and Kerberos/GSSAPI are considered a more powerful and more secure replacement of NIS.
Module type C extension Deprecated in 3.8 To be removed in 3.10 Substitute **none**
spwd ~~~~
The `spwd https://docs.python.org/3/library/spwd.html`_ module provides direct access to Unix shadow password database using non-standard APIs.
In general it's a bad idea to use the spwd. The spwd circumvents system security policies, it does not use the PAM stack, and is only compatible with local user accounts, because it ignores NSS. The use of the ``spwd`` module for access control must be consider a *security bug*, as it bypasses PAM's access control.
Further more the ``spwd`` module uses the `shadow(3) http://man7.org/linux/man-pages/man3/shadow.3.html`_ APIs. Functions like ``getspnam(3)`` access the ``/etc/shadow`` file directly. This is dangerous and even forbidden for confined services on systems with a security engine like SELinux or AppArmor.
Module type C extension Deprecated in 3.8 To be removed in 3.10 Substitute `python-pam https://pypi.org/project/python-pam/`_, `simpleplam https://pypi.org/project/simplepam/`_
Misc modules ------------
formatter ~~~~~~~~~
The `formatter https://docs.python.org/3/library/formatter.html`_ module is an old text formatting module which has been deprecated since Python 3.4.
Module type pure Python Deprecated in 3.4 To be removed in 3.10 Substitute *n/a*
imp ~~~
The `imp https://docs.python.org/3/library/imp.html`_ module is the predecessor of the `importlib https://docs.python.org/3/library/importlib.html`_ module. Most functions have been deprecated since Python 3.3 and the module since Python 3.4.
Module type C extension Deprecated in 3.4 To be removed in 3.10 Substitute importlib
msilib ~~~~~~
The `msilib https://docs.python.org/3/library/msilib.html`_ package is a Windows-only package. It supports the creation of Microsoft Installers (MSI). The package also exposes additional APIs to create cabinet files (CAB). The module is used to facilitate distutils to create MSI installers with ``bdist_msi`` command. In the past it was used to create CPython's official Windows installer, too.
Microsoft is slowly moving away from MSI in favor of Windows 10 Apps (AppX) as new deployment model [3]_.
Module type C extension + Python code Deprecated in 3.8 To be removed in 3.10 Substitute **none**
parser ~~~~~~
The `parser https://docs.python.org/3/library/parser.html`_ module provides an interface to Python’s internal parser and byte-code compiler. The stdlib has superior ways to interact with the parse tree. From Python 2.5 onward, it's much more convenient to cut in at the Abstract Syntax Tree (AST) generation and compilation stage.
The ``parser`` module causes additional work. It's C code that must be kept in sync with any change to Python's grammar and internal parser. Pablo wants to remove the parser module and promote lib2to3's pgen2 instead [6]_.
Most importantly the presence of the ``parser`` module makes it harder to switch to something more powerful than a LL(1) parser [7]_. Since the ``parser`` module is documented as deprecated since Python 2.5 and a new parsing technology is planned for 3.9, the ``parser`` module is scheduled for removal in 3.9.
Module type C extension Deprecated in 3.8, documented as deprecated since **2.5** To be removed in **3.9** Substitute ast, lib2to3.pgen2
pipes ~~~~~
The `pipes https://docs.python.org/3/library/pipes.html`_ module provides helpers to pipe the input of one command into the output of another command. The module is built on top of ``os.popen``. Users are encouraged to use the subprocess module instead.
Module type pure Python Deprecated in 3.8 To be removed in 3.10 Substitute subprocess module
Removed modules ===============
fpectl ------
The `fpectl https://docs.python.org/3.6/library/fpectl.html`_ module was never built by default, its usage was discouraged and considered dangerous. It also required a configure flag that caused an ABI incompatibility. The module was removed in 3.7 by Nathaniel J. Smith in `bpo-29137 https://bugs.python.org/issue29137`_.
Module type C extension + CAPI Deprecated in 3.7 Removed in 3.7 Substitute **none**
Modules to keep ===============
Some modules were originally proposed for deprecation.
fileinput ---------
The `fileinput https://docs.python.org/3/library/fileinput.html`_ module implements a helpers to iterate over a list of files from ``sys.argv``. The module predates the optparser and argparser module. The same functionality can be implemented with the argparser module.
Several core developers expressed their interest to keep the module in the standard library, as it is handy for quick scripts.
Module type pure Python
lib2to3 -------
The `lib2to3 https://docs.python.org/3/library/2to3.html`_ package provides the ``2to3`` command to transpile Python 2 code to Python 3 code.
The package is useful for other tasks besides porting code from Python 2 to 3. For example `black`_ uses it for code reformatting.
Module type pure Python
getopt ------
The `getopt https://docs.python.org/3/library/getopt.html`_ module mimics C's getopt() option parser.
Although users are encouraged to use argparse instead, the getopt module is still widely used. The module is small, simple, and handy for C developers to write simple Python scripts.
Module type pure Python Substitute argparse
optparse --------
The `optparse https://docs.python.org/3/library/optparse.html`_ module is the predecessor of the argparse module.
Although it has been deprecated for many years, it's still too widely used to remove it.
Module type pure Python Deprecated in 3.2 Substitute argparse
wave ----
The `wave https://docs.python.org/3/library/wave.html`_ module provides support for the WAV sound format.
The module is not deprecated, because The WAV format is still relevant these days. The ``wave`` module is also used in education, e.g. to show kids how to make noise with a computer.
The module uses one simple function from the `audioop`_ module to perform byte swapping between little and big endian formats. Before 24 bit WAV support was added, byte swap used to be implemented with the ``array`` module. To remove ``wave``'s dependency on the ``audioop``, the byte swap function could be either be moved to another module (e.g. ``operator``) or the ``array`` module could gain support for 24 bit (3 byte) arrays.
Module type pure Python (depends on *byteswap* from `audioop`_ C extension) Deprecated in 3.8 To be removed in 3.10 Substitute *n/a*
Future maintenance of removed modules =====================================
The main goal of the PEP is to reduce the burden and workload on the Python core developer team. Therefore removed modules will not be maintained by the core team as separate PyPI packages. However the removed code, tests and documentation may be moved into a new git repository, so community members have a place from which they can pick up and fork code.
A first draft of a `legacylib https://github.com/tiran/legacylib`_ repository is available on my private Github account. The modules could be made available on PyPI. The Python core team will not publish or maintain the packages. It is my hope that members of the Python community will adopt, maintain, and perhaps improve the deprecated modules.
It's my hope that some of the deprecated modules will be picked up and adopted by users that actually care about them. For example ``colorsys`` and ``imghdr`` are useful modules, but have limited feature set. A fork of ``imghdr`` can add new features and support for more image formats, without being constrained by Python's release cycle.
Most of the modules are in pure Python and can be easily packaged. Some depend on a simple C module, e.g. `audioop`_ and `crypt`_. Since `audioop`_ does not depend on any external libraries, it can be shipped in as binary wheels with some effort. Other C modules can be replaced with ctypes or cffi. For example I created `legacycrypt https://github.com/tiran/legacycrypt`_ with ``_crypt`` extension reimplemented with a few lines of ctypes code.
Discussions ===========
* Elana Hashman and Nick Coghlan suggested to keep the *getopt* module. * Berker Peksag proposed to deprecate and removed *msilib*. * Brett Cannon recommended to delay active deprecation warnings and removal of modules like *imp* until Python 3.10. Version 3.8 will be released shortly before Python 2 reaches end of lifetime. A delay reduced churn for users that migrate from Python 2 to 3.8. * Brett also came up with the idea to keep lib2to3. The package is useful for other purposes, e.g. `black https://pypi.org/project/black/`_ uses it to reformat Python code. * At one point, distutils was mentioned in the same sentence as this PEP. To avoid lengthy discussion and delay of the PEP, I decided against dealing with distutils. Deprecation of the distutils package will be handled by another PEP. * Multiple people (Gregory P. Smith, David Beazley, Nick Coghlan, ...) convinced me to keep the `wave`_ module. [4]_ * Gregory P. Smith proposed to deprecate `nntplib`_. [4]_ * Andrew Svetlov mentioned the ``socketserver`` module is questionable. However it's used to implement ``http.server`` and ``xmlrpc.server``. The stdlib doesn't have a replacement for the servers, yet.
Update history ==============
Update 1 --------
* Deprecate `parser`_ module * Keep `fileinput`_ module * Elaborate why `crypt`_ and `spwd`_ are dangerous and bad * Improve sections for `cgitb`_, `colorsys`_, `nntplib`_, and `smtpd`_ modules * The `colorsys`_, `crypt`_, `imghdr`_, `sndhdr`_, and `spwd`_ sections now list suitable substitutions. * Mention that ``socketserver`` is going to stay for ``http.server`` and ``xmlrpc.server`` * The future maintenance section now states that the deprecated modules may be adopted by Python community members.
References ==========
.. [1] https://en.wikipedia.org/wiki/Open_Sound_System#Free,_proprietary,_free .. [2] https://man.openbsd.org/ossaudio .. [3] https://blogs.msmvps.com/installsite/blog/2015/05/03/the-future-of-windows-i... .. [4] https://twitter.com/ChristianHeimes/status/1130257799475335169 .. [5] https://twitter.com/dabeaz/status/1130278844479545351 .. [6] https://mail.python.org/pipermail/python-dev/2019-May/157464.html .. [7] https://discuss.python.org/t/switch-pythons-parsing-tech-to-something-more-p...
Copyright =========
This document has been placed in the public domain.
.. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:
As I said, if the main annoyance with nntplib is the sporadic test failures, then the relevant tests can be disabled on CI.
NNTP itself is still used, even if less and less.
Regards
Antoine.
On Tue, 21 May 2019 16:12:42 +0200 Christian Heimes christian@python.org wrote:
Hi,
I have updated the PEP with feedback from discussions. The highlights are:
- Deprecate parser module
- Keep fileinput module
- Elaborate why crypt and spwd are dangerous and bad
- Improve sections for cgitb, colorsys, nntplib, and smtpd modules
- The colorsys, crypt, imghdr, sndhdr, and spwd sections now list suitable substitutions.
- Mention that socketserver is going to stay for http.server and xmlrpc.server
- The future maintenance section now states that the deprecated modules may be adopted by Python community members.
https://github.com/python/peps/compare/7799178a...2d536899?diff=unified#diff...
I'll be traveling the next couple of days and will only have limited opportunities to respond on feedback.
Christian
PEP: 594 Title: Removing dead batteries from the standard library Author: Christian Heimes christian@python.org Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 20-May-2019 Post-History: 21-May-2019
Abstract
This PEP proposed a list of standard library modules to be removed from the standard library. The modules are mostly historic data formats and APIs that have been superseded a long time ago, e.g. Mac OS 9 and Commodore.
Rationale
Back in the early days of Python, the interpreter came with a large set of useful modules. This was often refrained to as "batteries included" philosophy and was one of the corner stones to Python's success story. Users didn't have to figure out how to download and install separate packages in order to write a simple web server or parse email.
Times have changed. The introduction of the cheese shop (PyPI), setuptools, and later pip, it became simple and straight forward to download and install packages. Nowadays Python has a rich and vibrant ecosystem of third party packages. It's pretty much standard to either install packages from PyPI or use one of the many Python or Linux distributions.
On the other hand, Python's standard library is piling up cruft, unnecessary duplication of functionality, and dispensable features. This is undesirable for several reasons.
- Any additional module increases the maintenance cost for the Python core development team. The team has limited resources, reduced maintenance cost frees development time for other improvements.
- Modules in the standard library are generally favored and seen as the de-facto solution for a problem. A majority of users only pick 3rd party modules to replace a stdlib module, when they have a compelling reason, e.g. lxml instead of `xml`. The removal of an unmaintained stdlib module increases the chances of a community contributed module to become widely used.
- A lean and mean standard library benefits platforms with limited resources like devices with just a few hundred kilobyte of storage (e.g. BBC Micro:bit). Python on mobile platforms like BeeWare or WebAssembly (e.g. pyodide) also benefit from reduced download size.
The modules in the PEP have been selected for deprecation because their removal is either least controversial or most beneficial. For example least controversial are 30 years old multimedia formats like ``sunau`` audio format, which was used on SPARC and NeXT workstations in the late 1980ties. The ``crypt`` module has fundamental flaws that are better solved outside the standard library.
This PEP also designates some modules as not scheduled for removal. Some modules have been deprecated for several releases or seem unnecessary at first glance. However it is beneficial to keep the modules in the standard library, mostly for environments where installing a package from PyPI is not an option. This can be cooperate environments or class rooms where external code is not permitted without legal approval.
- The usage of FTP is declining, but some files are still provided over the FTP protocol or hosters offer FTP to upload content. Therefore ``ftplib`` is going to stay.
- The ``optparse`` and ``getopt`` module are widely used. They are mature modules with very low maintenance overhead.
- According to David Beazley [5]_ the ``wave`` module is easy to teach to kids and can make crazy sounds. Making a computer generate crazy sounds is powerful and highly motivating exercise for a 9yo aspiring developer. It's a fun battery to keep.
Deprecation schedule
3.8
This PEP targets Python 3.8. Version 3.8.0 final is scheduled to be released a few months before Python 2.7 will reach its end of lifetime. We expect that Python 3.8 will be targeted by users that migrate to Python 3 in 2019 and 2020. To reduce churn and to allow a smooth transition from Python 2, Python 3.8 will neither raise `DeprecationWarning` nor remove any modules that have been scheduled for removal. Instead deprecated modules will just be *documented* as deprecated. Optionally modules may emit a `PendingDeprecationWarning`.
All deprecated modules will also undergo a feature freeze. No additional features should be added. Bug should still be fixed.
3.9
Starting with Python 3.9, deprecated modules will start issuing `DeprecationWarning`. The `parser`_ module is removed and potentially replaced with a new module.
All other deprecated modules are fully supported and will receive security updates until Python 3.9 reaches its end of lifetime. Python 3.9.0 will be released about 18 months after 3.8.0 (April 2021?) and most likely be supported for 5 years after the release. The estimated EOL of Python 3.9 is in 2026.
3.10
In 3.10 all deprecated modules will be removed from the CPython repository together with tests, documentation, and autoconf rules.
PEP acceptance process
3.8.0b1 is scheduled to be release shortly after the PEP is officially submitted. Since it's improbable that the PEP will pass all stages of the PEP process in time, I propose a two step acceptance process that is analogous Python's two release deprecation process.
The first *provisionally accepted* phase targets Python 3.8.0b1. In the first phase no code is changes or removed. Modules are only documented as deprecated. The only exception is the `parser`_ module. It has been documented as deprecated since Python 2.5 and is scheduled for removal for 3.9 to make place for a more advanced parser.
The final decision, which modules will be removed and how the removed code is preserved, can be delayed for another year.
Deprecated modules
The modules are grouped as data encoding, multimedia, network, OS interface, and misc modules. The majority of modules are for old data formats or old APIs. Some others are rarely useful and have better replacements on PyPI, e.g. Pillow for image processing or NumPy-based projects to deal with audio processing.
.. csv-table:: Table 1: Proposed modules deprecations :header: "Module", "Deprecated in", "To be removed", "Replacement" :widths: 1, 1, 1, 2
aifc,3.8,3.10,\- asynchat,3.8,3.10,asyncio asyncore,3.8,3.10,asyncio audioop,3.8,3.10,\- binhex,3.8,3.10,\- cgi,3.8,3.10,\- cgitb,3.8,3.10,\- chunk,3.8,3.10,\- colorsys,3.8,3.10,"colormath, colour, colorspacious, Pillow" crypt,3.8,3.10,"bcrypt, argon2cffi, hashlib, passlib" fileinput,\-,**keep**,argparse formatter,3.4,3.10,\- fpectl,**3.7**,**3.7**,\- getopt,**3.2**,**keep**,"argparse, optparse" imghdr,3.8,3.10,"filetype, puremagic, python-magic" imp,**3.4**,3.10,importlib lib2to3,\-,**keep**, macpath,**3.7**,**3.8**,\- msilib,3.8,3.10,\- nntplib,3.8,3.10,\- nis,3.8,3.10,\- optparse,\-,**keep**,argparse ossaudiodev,3.8,3.10,\- parser,**2.5**,**3.9**,"ast, lib2to3.pgen2" pipes,3.8,3.10,subprocess smtpd,"**3.4.7**, **3.5.4**",3.10,aiosmtpd sndhdr,3.8,3.10,"filetype, puremagic, python-magic" spwd,3.8,3.10,"python-pam, simplepam" sunau,3.8,3.10,\- uu,3.8,3.10,\- wave,\-,**keep**, xdrlib,3.8,3.10,\-
Data encoding modules
binhex
The `binhex <https://docs.python.org/3/library/binhex.html>`_ module encodes and decodes Apple Macintosh binhex4 data. It was originally developed for TSR-80. In the 1980s and early 1990s it was used on classic Mac OS 9 to encode binary email attachments. Module type pure Python Deprecated in 3.8 To be removed in 3.10 Substitute **none** uu ~~ The `uu <https://docs.python.org/3/library/uu.html>`_ module provides uuencode format, an old binary encoding format for email from 1980. The uu format has been replaced by MIME. The uu codec is provided by the binascii module. Module type pure Python Deprecated in 3.8 To be removed in 3.10 Substitute **none** xdrlib
The `xdrlib https://docs.python.org/3/library/xdrlib.html`_ module supports the Sun External Data Representation Standard. XDR is an old binary serialization format from 1987. These days it's rarely used outside specialized domains like NFS.
Module type pure Python Deprecated in 3.8 To be removed in 3.10 Substitute **none**
Multimedia modules
aifc
The `aifc <https://docs.python.org/3/library/aifc.html>`_ module provides support for reading and writing AIFF and AIFF-C files. The Audio Interchange File Format is an old audio format from 1988 based on Amiga IFF. It was most commonly used on the Apple Macintosh. These days only few specialized application use AIFF. Module type pure Python (depends on `audioop`_ C extension) Deprecated in 3.8 To be removed in 3.10 Substitute **none** audioop
The `audioop https://docs.python.org/3/library/audioop.html`_ module contains helper functions to manipulate raw audio data and adaptive differential pulse-code modulated audio data. The module is implemented in C without any additional dependencies. The `aifc`_, `sunau`_, and `wave`_ module depend on `audioop`_ for some operations. The byteswap operation in the `wave`_ module can be substituted with little work.
Module type C extension Deprecated in 3.8 To be removed in 3.10 Substitute **none**
colorsys
The `colorsys <https://docs.python.org/3/library/colorsys.html>`_ module defines color conversion functions between RGB, YIQ, HSL, and HSV coordinate systems. The PyPI packages *colormath*, *colour*, and *colorspacious* provide more and advanced features. The Pillow library is better suited to transform images between color systems. Module type pure Python Deprecated in 3.8 To be removed in 3.10 Substitute `colormath <https://pypi.org/project/colormath/>`_, `colour <https://pypi.org/project/colour/>`_ `colorspacious <https://pypi.org/project/colorspacious/>`_, `Pillow <https://pypi.org/project/Pillow/>`_ chunk ~~~~~ The `chunk <https://docs.python.org/3/library/chunk.html>`_ module provides support for reading and writing Electronic Arts' Interchange File Format. IFF is an old audio file format originally introduced for Commodore and Amiga. The format is no longer relevant. Module type pure Python Deprecated in 3.8 To be removed in 3.10 Substitute **none** imghdr ~~~~~~ The `imghdr <https://docs.python.org/3/library/imghdr.html>`_ module is a simple tool to guess the image file format from the first 32 bytes of a file or buffer. It supports only a limited amount of formats and neither returns resolution nor color depth. Module type pure Python Deprecated in 3.8 To be removed in 3.10 Substitute `puremagic <https://pypi.org/project/puremagic/>`_, `filetype <https://pypi.org/project/filetype/>`_, `python-magic <https://pypi.org/project/python-magic/>`_ ossaudiodev
The `ossaudiodev https://docs.python.org/3/library/ossaudiodev.html`_ module provides support for Open Sound System, an interface to sound playback and capture devices. OSS was initially free software, but later support for newer sound devices and improvements were proprietary. Linux community abandoned OSS in favor of ALSA [1]_. Some operation systems like OpenBSD and NetBSD provide an incomplete [2]_ emulation of OSS.
Module type C extension Deprecated in 3.8 To be removed in 3.10 Substitute **none**
sndhdr
The `sndhdr <https://docs.python.org/3/library/sndhdr.html>`_ module is similar to the `imghdr`_ module but for audio formats. It guesses file format, channels, frame rate, and sample widths from the first 512 bytes of a file or buffer. The module only supports AU, AIFF, HCOM, VOC, WAV, and other ancient formats. Module type pure Python (depends on `audioop`_ C extension for some operations) Deprecated in 3.8 To be removed in 3.10 Substitute `puremagic <https://pypi.org/project/puremagic/>`_, `filetype <https://pypi.org/project/filetype/>`_, `python-magic <https://pypi.org/project/python-magic/>`_ sunau ~~~~~ The `sunau <https://docs.python.org/3/library/sunhdr.html>`_ module provides support for Sun AU sound format. It's yet another old, obsolete file format. Module type pure Python (depends on `audioop`_ C extension for some operations) Deprecated in 3.8 To be removed in 3.10 Substitute **none** Networking modules ------------------ asynchat
The `asynchat https://docs.python.org/3/library/asynchat.html`_ module is build on top of `asyncore`_ and has been deprecated since Python 3.6.
Module type pure Python Deprecated in 3.6 Removed in 3.10 Substitute asyncio
asyncore
The `asyncore <https://docs.python.org/3/library/asyncore.html>`_ module was the first module for asynchronous socket service clients and servers. It has been replaced by asyncio and is deprecated since Python 3.6. The ``asyncore`` module is also used in stdlib tests. The tests for ``ftplib``, ``logging``, ``smptd``, ``smtplib``, and ``ssl`` are partly based on ``asyncore``. These tests must be updated to use asyncio or threading. Module type pure Python Deprecated in 3.6 Removed in 3.10 Substitute asyncio cgi ~~~ The `cgi <https://docs.python.org/3/library/cgi.html>`_ module is a support module for Common Gateway Interface (CGI) scripts. CGI is deemed as inefficient because every incoming request is handled in a new process. PEP 206 considers the module as *designed poorly and are now near-impossible to fix*. Several people proposed to either keep the cgi module for features like `cgi.parse_qs()` or move `cgi.escape()` to a different module. The functions `cgi.parse_qs` and `cgi.parse_qsl` have been deprecated for a while and are actually aliases for `urllib.parse.parse_qs` and `urllib.parse.parse_qsl`. The function `cgi.quote` has been deprecated in favor of `html.quote` with secure default values. Module type pure Python Deprecated in 3.8 To be removed in 3.10 Substitute **none** cgitb ~~~~~ The `cgitb <https://docs.python.org/3/library/cgitb.html>`_ module is a helper for the cgi module for configurable tracebacks. The ``cgitb`` module is not used by any major Python web framework (Django, Pyramid, Plone, Flask, CherryPy, or Bottle). Only Paste uses it in an optional debugging middleware. Module type pure Python Deprecated in 3.8 To be removed in 3.10 Substitute **none** smtpd ~~~~~ The `smtpd <https://docs.python.org/3/library/smtpd.html>`_ module provides a simple implementation of a SMTP mail server. The module documentation marks the module as deprecated and recommends ``aiosmtpd`` instead. The deprecation message was added in releases 3.4.7, 3.5.4, and 3.6.1. Module type pure Python Deprecated in **3.7** To be removed in 3.10 Substitute aiosmtpd nntplib ~~~~~~~ The `nntplib <https://docs.python.org/3/library/nntplib.html>`_ module implements the client side of the Network News Transfer Protocol (nntp). News groups used to be a dominant platform for online discussions. Over the last two decades, news has been slowly but steadily replaced with mailing lists and web-based discussion platforms. Twisted is also `planning <https://twistedmatrix.com/trac/ticket/9405>`_ to deprecate NNTP support and `pynnt <https://github.com/greenbender/pynntp>`_ hasn't seen any activity since 2014. This is a good indicator that the public interest in NNTP support is declining. The ``nntplib`` tests have been the cause of additional work in the recent past. Python only contains client side of NNTP. The tests connect to external news server. The servers are sometimes unavailble, too slow, or do not work correctly over IPv6. The situation causes flaky test runs on buildbots. Module type pure Python Deprecated in 3.8 To be removed in 3.10 Substitute **none** Operating system interface -------------------------- crypt ~~~~~ The `crypt <https://docs.python.org/3/library/crypt.html>`_ module implements password hashing based on ``crypt(3)`` function from ``libcrypt`` or ``libxcrypt`` on Unix-like platform. The algorithms are mostly old, of poor quality and insecure. Users are discouraged to use them. * The module is not available on Windows. Cross-platform application need an alternative implementation any way. * Only DES encryption is guarenteed to be available. DES has an extremely limited key space of 2**56. * MD5, salted SHA256, salted SHA512, and Blowfish are optional extension. SSHA256 and SSHA512 are glibc extensions. Blowfish (bcrypt) is the only algorithm that is still secure. However it's in glibc and therefore not commonly available on Linux. * Depending on the platform, the ``crypt`` module is not thread safe. Only implementations with ``crypt_r(3)`` are thread safe. * The module was never useful to interact with system user and password databases. On BSD, macOS, and Linux, all user authentication and password modification operations must go through PAM (pluggable authentication module), see `spwd`_ deprecation. Module type C extension + Python module Deprecated in 3.8 To be removed in 3.10 Substitute `bcrypt <https://pypi.org/project/bcrypt/>`_, `passlib <https://pypi.org/project/passlib/>`_, `argon2cffi <https://pypi.org/project/argon2-cffi/>`_, hashlib module (PBKDF2, scrypt) macpath ~~~~~~~ The `macpath <https://docs.python.org/3/library/macpath.html>`_ module provides Mac OS 9 implementation of os.path routines. Mac OS 9 is no longer supported Module type pure Python Deprecated in 3.7 Removed in 3.8 Substitute **none** nis ~~~ The `nis <https://docs.python.org/3/library/nis.html>`_ module provides NIS/YP support. Network Information Service / Yellow Pages is an old and deprecated directory service protocol developed by Sun Microsystems. It's designed successor NIS+ from 1992 never took off. For a long time, libc's Name Service Switch, LDAP, and Kerberos/GSSAPI are considered a more powerful and more secure replacement of NIS. Module type C extension Deprecated in 3.8 To be removed in 3.10 Substitute **none** spwd ~~~~ The `spwd <https://docs.python.org/3/library/spwd.html>`_ module provides direct access to Unix shadow password database using non-standard APIs. In general it's a bad idea to use the spwd. The spwd circumvents system security policies, it does not use the PAM stack, and is only compatible with local user accounts, because it ignores NSS. The use of the ``spwd`` module for access control must be consider a *security bug*, as it bypasses PAM's access control. Further more the ``spwd`` module uses the `shadow(3) <http://man7.org/linux/man-pages/man3/shadow.3.html>`_ APIs. Functions like ``getspnam(3)`` access the ``/etc/shadow`` file directly. This is dangerous and even forbidden for confined services on systems with a security engine like SELinux or AppArmor. Module type C extension Deprecated in 3.8 To be removed in 3.10 Substitute `python-pam <https://pypi.org/project/python-pam/>`_, `simpleplam <https://pypi.org/project/simplepam/>`_ Misc modules ------------ formatter
The `formatter https://docs.python.org/3/library/formatter.html`_ module is an old text formatting module which has been deprecated since Python 3.4.
Module type pure Python Deprecated in 3.4 To be removed in 3.10 Substitute *n/a*
imp
The `imp <https://docs.python.org/3/library/imp.html>`_ module is the predecessor of the `importlib <https://docs.python.org/3/library/importlib.html>`_ module. Most functions have been deprecated since Python 3.3 and the module since Python 3.4. Module type C extension Deprecated in 3.4 To be removed in 3.10 Substitute importlib msilib
The `msilib https://docs.python.org/3/library/msilib.html`_ package is a Windows-only package. It supports the creation of Microsoft Installers (MSI). The package also exposes additional APIs to create cabinet files (CAB). The module is used to facilitate distutils to create MSI installers with ``bdist_msi`` command. In the past it was used to create CPython's official Windows installer, too.
Microsoft is slowly moving away from MSI in favor of Windows 10 Apps (AppX) as new deployment model [3]_.
Module type C extension + Python code Deprecated in 3.8 To be removed in 3.10 Substitute **none**
parser
The `parser <https://docs.python.org/3/library/parser.html>`_ module provides an interface to Python’s internal parser and byte-code compiler. The stdlib has superior ways to interact with the parse tree. From Python 2.5 onward, it's much more convenient to cut in at the Abstract Syntax Tree (AST) generation and compilation stage. The ``parser`` module causes additional work. It's C code that must be kept in sync with any change to Python's grammar and internal parser. Pablo wants to remove the parser module and promote lib2to3's pgen2 instead [6]_. Most importantly the presence of the ``parser`` module makes it harder to switch to something more powerful than a LL(1) parser [7]_. Since the ``parser`` module is documented as deprecated since Python 2.5 and a new parsing technology is planned for 3.9, the ``parser`` module is scheduled for removal in 3.9. Module type C extension Deprecated in 3.8, documented as deprecated since **2.5** To be removed in **3.9** Substitute ast, lib2to3.pgen2 pipes ~~~~~ The `pipes <https://docs.python.org/3/library/pipes.html>`_ module provides helpers to pipe the input of one command into the output of another command. The module is built on top of ``os.popen``. Users are encouraged to use the subprocess module instead. Module type pure Python Deprecated in 3.8 To be removed in 3.10 Substitute subprocess module Removed modules =============== fpectl ------ The `fpectl <https://docs.python.org/3.6/library/fpectl.html>`_ module was never built by default, its usage was discouraged and considered dangerous. It also required a configure flag that caused an ABI incompatibility. The module was removed in 3.7 by Nathaniel J. Smith in `bpo-29137 <https://bugs.python.org/issue29137>`_. Module type C extension + CAPI Deprecated in 3.7 Removed in 3.7 Substitute **none** Modules to keep =============== Some modules were originally proposed for deprecation. fileinput --------- The `fileinput <https://docs.python.org/3/library/fileinput.html>`_ module implements a helpers to iterate over a list of files from ``sys.argv``. The module predates the optparser and argparser module. The same functionality can be implemented with the argparser module. Several core developers expressed their interest to keep the module in the standard library, as it is handy for quick scripts. Module type pure Python lib2to3 ------- The `lib2to3 <https://docs.python.org/3/library/2to3.html>`_ package provides the ``2to3`` command to transpile Python 2 code to Python 3 code. The package is useful for other tasks besides porting code from Python 2 to 3. For example `black`_ uses it for code reformatting. Module type pure Python getopt ------ The `getopt <https://docs.python.org/3/library/getopt.html>`_ module mimics C's getopt() option parser. Although users are encouraged to use argparse instead, the getopt module is still widely used. The module is small, simple, and handy for C developers to write simple Python scripts. Module type pure Python Substitute argparse optparse -------- The `optparse <https://docs.python.org/3/library/optparse.html>`_ module is the predecessor of the argparse module. Although it has been deprecated for many years, it's still too widely used to remove it. Module type pure Python Deprecated in 3.2 Substitute argparse wave ---- The `wave <https://docs.python.org/3/library/wave.html>`_ module provides support for the WAV sound format. The module is not deprecated, because The WAV format is still relevant these days. The ``wave`` module is also used in education, e.g. to show kids how to make noise with a computer. The module uses one simple function from the `audioop`_ module to perform byte swapping between little and big endian formats. Before 24 bit WAV support was added, byte swap used to be implemented with the ``array`` module. To remove ``wave``'s dependency on the ``audioop``, the byte swap function could be either be moved to another module (e.g. ``operator``) or the ``array`` module could gain support for 24 bit (3 byte) arrays. Module type pure Python (depends on *byteswap* from `audioop`_ C extension) Deprecated in 3.8 To be removed in 3.10 Substitute *n/a* Future maintenance of removed modules ===================================== The main goal of the PEP is to reduce the burden and workload on the Python core developer team. Therefore removed modules will not be maintained by the core team as separate PyPI packages. However the removed code, tests and documentation may be moved into a new git repository, so community members have a place from which they can pick up and fork code. A first draft of a `legacylib <https://github.com/tiran/legacylib>`_ repository is available on my private Github account. The modules could be made available on PyPI. The Python core team will not publish or maintain the packages. It is my hope that members of the Python community will adopt, maintain, and perhaps improve the deprecated modules. It's my hope that some of the deprecated modules will be picked up and adopted by users that actually care about them. For example ``colorsys`` and ``imghdr`` are useful modules, but have limited feature set. A fork of ``imghdr`` can add new features and support for more image formats, without being constrained by Python's release cycle. Most of the modules are in pure Python and can be easily packaged. Some depend on a simple C module, e.g. `audioop`_ and `crypt`_. Since `audioop`_ does not depend on any external libraries, it can be shipped in as binary wheels with some effort. Other C modules can be replaced with ctypes or cffi. For example I created `legacycrypt <https://github.com/tiran/legacycrypt>`_ with ``_crypt`` extension reimplemented with a few lines of ctypes code. Discussions =========== * Elana Hashman and Nick Coghlan suggested to keep the *getopt* module. * Berker Peksag proposed to deprecate and removed *msilib*. * Brett Cannon recommended to delay active deprecation warnings and removal of modules like *imp* until Python 3.10. Version 3.8 will be released shortly before Python 2 reaches end of lifetime. A delay reduced churn for users that migrate from Python 2 to 3.8. * Brett also came up with the idea to keep lib2to3. The package is useful for other purposes, e.g. `black <https://pypi.org/project/black/>`_ uses it to reformat Python code. * At one point, distutils was mentioned in the same sentence as this PEP. To avoid lengthy discussion and delay of the PEP, I decided against dealing with distutils. Deprecation of the distutils package will be handled by another PEP. * Multiple people (Gregory P. Smith, David Beazley, Nick Coghlan, ...) convinced me to keep the `wave`_ module. [4]_ * Gregory P. Smith proposed to deprecate `nntplib`_. [4]_ * Andrew Svetlov mentioned the ``socketserver`` module is questionable. However it's used to implement ``http.server`` and ``xmlrpc.server``. The stdlib doesn't have a replacement for the servers, yet. Update history ============== Update 1 -------- * Deprecate `parser`_ module * Keep `fileinput`_ module * Elaborate why `crypt`_ and `spwd`_ are dangerous and bad * Improve sections for `cgitb`_, `colorsys`_, `nntplib`_, and `smtpd`_ modules * The `colorsys`_, `crypt`_, `imghdr`_, `sndhdr`_, and `spwd`_ sections now list suitable substitutions. * Mention that ``socketserver`` is going to stay for ``http.server`` and ``xmlrpc.server`` * The future maintenance section now states that the deprecated modules may be adopted by Python community members. References ========== .. [1] https://en.wikipedia.org/wiki/Open_Sound_System#Free,_proprietary,_free .. [2] https://man.openbsd.org/ossaudio .. [3] https://blogs.msmvps.com/installsite/blog/2015/05/03/the-future-of-windows-installer-msi-in-the-light-of-windows-10-and-the-universal-windows-platform/ .. [4] https://twitter.com/ChristianHeimes/status/1130257799475335169 .. [5] https://twitter.com/dabeaz/status/1130278844479545351 .. [6] https://mail.python.org/pipermail/python-dev/2019-May/157464.html .. [7] https://discuss.python.org/t/switch-pythons-parsing-tech-to-something-more-powerful-than-ll-1/379 Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/python-python-dev%40m.gmane.org
OK, sounds like nntplib can stay — this time.
On Tue, May 21, 2019 at 08:33 Antoine Pitrou solipsis@pitrou.net wrote:
As I said, if the main annoyance with nntplib is the sporadic test failures, then the relevant tests can be disabled on CI.
NNTP itself is still used, even if less and less.
Regards
Antoine.
On Tue, 21 May 2019 16:12:42 +0200 Christian Heimes christian@python.org wrote:
Hi,
I have updated the PEP with feedback from discussions. The highlights
are:
- Deprecate parser module
- Keep fileinput module
- Elaborate why crypt and spwd are dangerous and bad
- Improve sections for cgitb, colorsys, nntplib, and smtpd modules
- The colorsys, crypt, imghdr, sndhdr, and spwd sections now list
suitable substitutions.
- Mention that socketserver is going to stay for http.server and
xmlrpc.server
- The future maintenance section now states that the deprecated modules
may be adopted by Python community members.
https://github.com/python/peps/compare/7799178a...2d536899?diff=unified#diff...
I'll be traveling the next couple of days and will only have limited
opportunities to respond on feedback.
Christian
PEP: 594 Title: Removing dead batteries from the standard library Author: Christian Heimes christian@python.org Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 20-May-2019 Post-History: 21-May-2019
Abstract
This PEP proposed a list of standard library modules to be removed from
the
standard library. The modules are mostly historic data formats and APIs
that
have been superseded a long time ago, e.g. Mac OS 9 and Commodore.
Rationale
Back in the early days of Python, the interpreter came with a large set
of
useful modules. This was often refrained to as "batteries included" philosophy and was one of the corner stones to Python's success story. Users didn't have to figure out how to download and install separate packages in order to write a simple web server or parse email.
Times have changed. The introduction of the cheese shop (PyPI),
setuptools,
and later pip, it became simple and straight forward to download and
install
packages. Nowadays Python has a rich and vibrant ecosystem of third party packages. It's pretty much standard to either install packages from PyPI
or
use one of the many Python or Linux distributions.
On the other hand, Python's standard library is piling up cruft,
unnecessary
duplication of functionality, and dispensable features. This is
undesirable
for several reasons.
- Any additional module increases the maintenance cost for the Python
core
development team. The team has limited resources, reduced maintenance
cost
frees development time for other improvements.
- Modules in the standard library are generally favored and seen as the de-facto solution for a problem. A majority of users only pick 3rd
party
modules to replace a stdlib module, when they have a compelling
reason, e.g.
lxml instead of `xml`. The removal of an unmaintained stdlib module increases the chances of a community contributed module to become
widely
used.
- A lean and mean standard library benefits platforms with limited
resources
like devices with just a few hundred kilobyte of storage (e.g. BBC Micro:bit). Python on mobile platforms like BeeWare or WebAssembly (e.g. pyodide) also benefit from reduced download size.
The modules in the PEP have been selected for deprecation because their removal is either least controversial or most beneficial. For example least controversial are 30 years old multimedia formats like ``sunau`` audio format, which was used on SPARC and NeXT workstations in the late 1980ties. The ``crypt`` module has fundamental flaws that are better
solved
outside the standard library.
This PEP also designates some modules as not scheduled for removal. Some modules have been deprecated for several releases or seem unnecessary at first glance. However it is beneficial to keep the modules in the
standard
library, mostly for environments where installing a package from PyPI is
not
an option. This can be cooperate environments or class rooms where
external
code is not permitted without legal approval.
- The usage of FTP is declining, but some files are still provided over the FTP protocol or hosters offer FTP to upload content. Therefore ``ftplib`` is going to stay.
- The ``optparse`` and ``getopt`` module are widely used. They are mature modules with very low maintenance overhead.
- According to David Beazley [5]_ the ``wave`` module is easy to teach to kids and can make crazy sounds. Making a computer generate crazy
sounds is
powerful and highly motivating exercise for a 9yo aspiring developer.
It's
a fun battery to keep.
Deprecation schedule
3.8
This PEP targets Python 3.8. Version 3.8.0 final is scheduled to be
released
a few months before Python 2.7 will reach its end of lifetime. We expect
that
Python 3.8 will be targeted by users that migrate to Python 3 in 2019 and 2020. To reduce churn and to allow a smooth transition from Python 2, Python 3.8 will neither raise `DeprecationWarning` nor remove any modules that have been scheduled for removal. Instead deprecated modules
will
just be *documented* as deprecated. Optionally modules may emit a `PendingDeprecationWarning`.
All deprecated modules will also undergo a feature freeze. No additional features should be added. Bug should still be fixed.
3.9
Starting with Python 3.9, deprecated modules will start issuing `DeprecationWarning`. The `parser`_ module is removed and potentially replaced with a new module.
All other deprecated modules are fully supported and will receive
security
updates until Python 3.9 reaches its end of lifetime. Python 3.9.0 will be released about 18 months after 3.8.0 (April 2021?) and most likely be supported for 5 years after the release. The estimated EOL of Python
3.9
is in 2026.
3.10
In 3.10 all deprecated modules will be removed from the CPython
repository
together with tests, documentation, and autoconf rules.
PEP acceptance process
3.8.0b1 is scheduled to be release shortly after the PEP is officially submitted. Since it's improbable that the PEP will pass all stages of the PEP process in time, I propose a two step acceptance process that is analogous Python's two release deprecation process.
The first *provisionally accepted* phase targets Python 3.8.0b1. In the
first
phase no code is changes or removed. Modules are only documented as deprecated. The only exception is the `parser`_ module. It has been documented as deprecated since Python 2.5 and is scheduled for removal
for
3.9 to make place for a more advanced parser.
The final decision, which modules will be removed and how the removed
code
is preserved, can be delayed for another year.
Deprecated modules
The modules are grouped as data encoding, multimedia, network, OS
interface,
and misc modules. The majority of modules are for old data formats or old APIs. Some others are rarely useful and have better replacements on PyPI, e.g. Pillow for image processing or NumPy-based projects to deal
with
audio processing.
.. csv-table:: Table 1: Proposed modules deprecations :header: "Module", "Deprecated in", "To be removed", "Replacement" :widths: 1, 1, 1, 2
aifc,3.8,3.10,\- asynchat,3.8,3.10,asyncio asyncore,3.8,3.10,asyncio audioop,3.8,3.10,\- binhex,3.8,3.10,\- cgi,3.8,3.10,\- cgitb,3.8,3.10,\- chunk,3.8,3.10,\- colorsys,3.8,3.10,"colormath, colour, colorspacious, Pillow" crypt,3.8,3.10,"bcrypt, argon2cffi, hashlib, passlib" fileinput,\-,**keep**,argparse formatter,3.4,3.10,\- fpectl,**3.7**,**3.7**,\- getopt,**3.2**,**keep**,"argparse, optparse" imghdr,3.8,3.10,"filetype, puremagic, python-magic" imp,**3.4**,3.10,importlib lib2to3,\-,**keep**, macpath,**3.7**,**3.8**,\- msilib,3.8,3.10,\- nntplib,3.8,3.10,\- nis,3.8,3.10,\- optparse,\-,**keep**,argparse ossaudiodev,3.8,3.10,\- parser,**2.5**,**3.9**,"ast, lib2to3.pgen2" pipes,3.8,3.10,subprocess smtpd,"**3.4.7**, **3.5.4**",3.10,aiosmtpd sndhdr,3.8,3.10,"filetype, puremagic, python-magic" spwd,3.8,3.10,"python-pam, simplepam" sunau,3.8,3.10,\- uu,3.8,3.10,\- wave,\-,**keep**, xdrlib,3.8,3.10,\-
Data encoding modules
binhex
The `binhex <https://docs.python.org/3/library/binhex.html>`_ module
encodes
and decodes Apple Macintosh binhex4 data. It was originally developed for TSR-80. In the 1980s and early 1990s it was used on classic Mac OS 9 to encode binary email attachments.
Module type pure Python Deprecated in 3.8 To be removed in 3.10 Substitute **none**
uu ~~
The `uu https://docs.python.org/3/library/uu.html`_ module provides uuencode format, an old binary encoding format for email from 1980. The
uu
format has been replaced by MIME. The uu codec is provided by the
binascii
module.
Module type pure Python Deprecated in 3.8 To be removed in 3.10 Substitute **none**
xdrlib
The `xdrlib <https://docs.python.org/3/library/xdrlib.html>`_ module
supports
the Sun External Data Representation Standard. XDR is an old binary serialization format from 1987. These days it's rarely used outside specialized domains like NFS.
Module type pure Python Deprecated in 3.8 To be removed in 3.10 Substitute **none**
Multimedia modules
aifc
The `aifc <https://docs.python.org/3/library/aifc.html>`_ module
provides
support for reading and writing AIFF and AIFF-C files. The Audio
Interchange
File Format is an old audio format from 1988 based on Amiga IFF. It was
most
commonly used on the Apple Macintosh. These days only few specialized application use AIFF.
Module type pure Python (depends on `audioop`_ C extension) Deprecated in 3.8 To be removed in 3.10 Substitute **none**
audioop
The `audioop <https://docs.python.org/3/library/audioop.html>`_ module contains helper functions to manipulate raw audio data and adaptive differential pulse-code modulated audio data. The module is implemented
in
C without any additional dependencies. The `aifc`_, `sunau`_, and `wave`_ module depend on `audioop`_ for some operations. The byteswap operation
in
the `wave`_ module can be substituted with little work.
Module type C extension Deprecated in 3.8 To be removed in 3.10 Substitute **none**
colorsys
The `colorsys <https://docs.python.org/3/library/colorsys.html>`_ module defines color conversion functions between RGB, YIQ, HSL, and HSV
coordinate
systems.
The PyPI packages *colormath*, *colour*, and *colorspacious* provide
more and
advanced features. The Pillow library is better suited to transform
images
between color systems.
Module type pure Python Deprecated in 3.8 To be removed in 3.10 Substitute `colormath https://pypi.org/project/colormath/`_, `colour https://pypi.org/project/colour/`_ `colorspacious https://pypi.org/project/colorspacious/`_, `Pillow https://pypi.org/project/Pillow/`_
chunk
The `chunk <https://docs.python.org/3/library/chunk.html>`_ module
provides
support for reading and writing Electronic Arts' Interchange File Format. IFF is an old audio file format originally introduced for Commodore and Amiga. The format is no longer relevant.
Module type pure Python Deprecated in 3.8 To be removed in 3.10 Substitute **none**
imghdr
The `imghdr <https://docs.python.org/3/library/imghdr.html>`_ module is
a
simple tool to guess the image file format from the first 32 bytes of a file or buffer. It supports only a limited amount of formats and neither returns resolution nor color depth.
Module type pure Python Deprecated in 3.8 To be removed in 3.10 Substitute `puremagic https://pypi.org/project/puremagic/`_, `filetype https://pypi.org/project/filetype/`_, `python-magic https://pypi.org/project/python-magic/`_
ossaudiodev
The `ossaudiodev <https://docs.python.org/3/library/ossaudiodev.html>`_ module provides support for Open Sound System, an interface to sound playback and capture devices. OSS was initially free software, but later support for newer sound devices and improvements were proprietary. Linux community abandoned OSS in favor of ALSA [1]_. Some operation systems
like
OpenBSD and NetBSD provide an incomplete [2]_ emulation of OSS.
Module type C extension Deprecated in 3.8 To be removed in 3.10 Substitute **none**
sndhdr
The `sndhdr <https://docs.python.org/3/library/sndhdr.html>`_ module is similar to the `imghdr`_ module but for audio formats. It guesses file format, channels, frame rate, and sample widths from the first 512 bytes
of
a file or buffer. The module only supports AU, AIFF, HCOM, VOC, WAV, and other ancient formats.
Module type pure Python (depends on `audioop`_ C extension for some operations) Deprecated in 3.8 To be removed in 3.10 Substitute `puremagic https://pypi.org/project/puremagic/`_, `filetype https://pypi.org/project/filetype/`_, `python-magic https://pypi.org/project/python-magic/`_
sunau
The `sunau <https://docs.python.org/3/library/sunhdr.html>`_ module
provides
support for Sun AU sound format. It's yet another old, obsolete file
format.
Module type pure Python (depends on `audioop`_ C extension for some operations) Deprecated in 3.8 To be removed in 3.10 Substitute **none**
Networking modules
asynchat
The `asynchat <https://docs.python.org/3/library/asynchat.html>`_ module is build on top of `asyncore`_ and has been deprecated since Python 3.6. Module type pure Python Deprecated in 3.6 Removed in 3.10 Substitute asyncio asyncore
The `asyncore https://docs.python.org/3/library/asyncore.html`_
module was
the first module for asynchronous socket service clients and servers. It has been replaced by asyncio and is deprecated since Python 3.6.
The ``asyncore`` module is also used in stdlib tests. The tests for ``ftplib``, ``logging``, ``smptd``, ``smtplib``, and ``ssl`` are partly based on ``asyncore``. These tests must be updated to use asyncio or threading.
Module type pure Python Deprecated in 3.6 Removed in 3.10 Substitute asyncio
cgi
The `cgi <https://docs.python.org/3/library/cgi.html>`_ module is a
support
module for Common Gateway Interface (CGI) scripts. CGI is deemed as inefficient because every incoming request is handled in a new process.
PEP
206 considers the module as *designed poorly and are now near-impossible to fix*.
Several people proposed to either keep the cgi module for features like `cgi.parse_qs()` or move `cgi.escape()` to a different module. The functions `cgi.parse_qs` and `cgi.parse_qsl` have been deprecated for a while and are actually aliases for `urllib.parse.parse_qs` and `urllib.parse.parse_qsl`. The function `cgi.quote` has been deprecated in favor of `html.quote` with secure default values.
Module type pure Python Deprecated in 3.8 To be removed in 3.10 Substitute **none**
cgitb
The `cgitb <https://docs.python.org/3/library/cgitb.html>`_ module is a helper for the cgi module for configurable tracebacks. The ``cgitb`` module is not used by any major Python web framework
(Django,
Pyramid, Plone, Flask, CherryPy, or Bottle). Only Paste uses it in an optional debugging middleware.
Module type pure Python Deprecated in 3.8 To be removed in 3.10 Substitute **none**
smtpd
The `smtpd <https://docs.python.org/3/library/smtpd.html>`_ module
provides
a simple implementation of a SMTP mail server. The module documentation marks the module as deprecated and recommends ``aiosmtpd`` instead. The deprecation message was added in releases 3.4.7, 3.5.4, and 3.6.1.
Module type pure Python Deprecated in **3.7** To be removed in 3.10 Substitute aiosmtpd
nntplib
The `nntplib <https://docs.python.org/3/library/nntplib.html>`_ module implements the client side of the Network News Transfer Protocol (nntp).
News
groups used to be a dominant platform for online discussions. Over the
last
two decades, news has been slowly but steadily replaced with mailing
lists
and web-based discussion platforms. Twisted is also `planning https://twistedmatrix.com/trac/ticket/9405`_ to deprecate
NNTP
support and `pynnt https://github.com/greenbender/pynntp`_ hasn't
seen any
activity since 2014. This is a good indicator that the public interest in NNTP support is declining.
The ``nntplib`` tests have been the cause of additional work in the
recent
past. Python only contains client side of NNTP. The tests connect to external news server. The servers are sometimes unavailble, too slow, or
do
not work correctly over IPv6. The situation causes flaky test runs on buildbots.
Module type pure Python Deprecated in 3.8 To be removed in 3.10 Substitute **none**
Operating system interface
crypt
The `crypt <https://docs.python.org/3/library/crypt.html>`_ module
implements
password hashing based on ``crypt(3)`` function from ``libcrypt`` or ``libxcrypt`` on Unix-like platform. The algorithms are mostly old, of
poor
quality and insecure. Users are discouraged to use them.
- The module is not available on Windows. Cross-platform application need an alternative implementation any way.
- Only DES encryption is guarenteed to be available. DES has an extremely limited key space of 2**56.
- MD5, salted SHA256, salted SHA512, and Blowfish are optional extension. SSHA256 and SSHA512 are glibc extensions. Blowfish (bcrypt) is the only algorithm that is still secure. However it's in glibc and therefore not commonly available on Linux.
- Depending on the platform, the ``crypt`` module is not thread safe.
Only
implementations with ``crypt_r(3)`` are thread safe.
- The module was never useful to interact with system user and password databases. On BSD, macOS, and Linux, all user authentication and password modification operations must go through PAM (pluggable authentication module), see `spwd`_ deprecation.
Module type C extension + Python module Deprecated in 3.8 To be removed in 3.10 Substitute `bcrypt https://pypi.org/project/bcrypt/`_, `passlib https://pypi.org/project/passlib/`_, `argon2cffi https://pypi.org/project/argon2-cffi/`_, hashlib module (PBKDF2, scrypt)
macpath
The `macpath <https://docs.python.org/3/library/macpath.html>`_ module provides Mac OS 9 implementation of os.path routines. Mac OS 9 is no
longer
supported
Module type pure Python Deprecated in 3.7 Removed in 3.8 Substitute **none**
nis
The `nis <https://docs.python.org/3/library/nis.html>`_ module provides NIS/YP support. Network Information Service / Yellow Pages is an old and deprecated directory service protocol developed by Sun Microsystems. It's designed successor NIS+ from 1992 never took off. For a long time, libc's Name Service Switch, LDAP, and Kerberos/GSSAPI are considered a more
powerful
and more secure replacement of NIS.
Module type C extension Deprecated in 3.8 To be removed in 3.10 Substitute **none**
spwd
The `spwd <https://docs.python.org/3/library/spwd.html>`_ module
provides
direct access to Unix shadow password database using non-standard APIs.
In general it's a bad idea to use the spwd. The spwd circumvents system security policies, it does not use the PAM stack, and is only compatible with local user accounts, because it ignores NSS. The use of the ``spwd`` module for access control must be consider a *security bug*, as it
bypasses
PAM's access control.
Further more the ``spwd`` module uses the `shadow(3) http://man7.org/linux/man-pages/man3/shadow.3.html`_ APIs. Functions like ``getspnam(3)`` access the ``/etc/shadow`` file directly.
This
is dangerous and even forbidden for confined services on systems with a security engine like SELinux or AppArmor.
Module type C extension Deprecated in 3.8 To be removed in 3.10 Substitute `python-pam https://pypi.org/project/python-pam/`_, `simpleplam https://pypi.org/project/simplepam/`_
Misc modules
formatter
The `formatter <https://docs.python.org/3/library/formatter.html>`_
module
is an old text formatting module which has been deprecated since Python
3.4.
Module type pure Python Deprecated in 3.4 To be removed in 3.10 Substitute *n/a*
imp
The `imp <https://docs.python.org/3/library/imp.html>`_ module is the predecessor of the `importlib <https://docs.python.org/3/library/importlib.html>`_ module.
Most
functions have been deprecated since Python 3.3 and the module since Python 3.4.
Module type C extension Deprecated in 3.4 To be removed in 3.10 Substitute importlib
msilib
The `msilib <https://docs.python.org/3/library/msilib.html>`_ package
is a
Windows-only package. It supports the creation of Microsoft Installers
(MSI).
The package also exposes additional APIs to create cabinet files (CAB).
The
module is used to facilitate distutils to create MSI installers with ``bdist_msi`` command. In the past it was used to create CPython's
official
Windows installer, too.
Microsoft is slowly moving away from MSI in favor of Windows 10 Apps
(AppX)
as new deployment model [3]_.
Module type C extension + Python code Deprecated in 3.8 To be removed in 3.10 Substitute **none**
parser
The `parser <https://docs.python.org/3/library/parser.html>`_ module
provides
an interface to Python’s internal parser and byte-code compiler. The
stdlib
has superior ways to interact with the parse tree. From Python 2.5
onward,
it's much more convenient to cut in at the Abstract Syntax Tree (AST) generation and compilation stage.
The ``parser`` module causes additional work. It's C code that must be kept in sync with any change to Python's grammar and internal parser. Pablo wants to remove the parser module and promote lib2to3's pgen2
instead
[6]_.
Most importantly the presence of the ``parser`` module makes it harder to switch to something more powerful than a LL(1) parser [7]_. Since the ``parser`` module is documented as deprecated since Python 2.5 and a new parsing technology is planned for 3.9, the ``parser`` module is
scheduled for
removal in 3.9.
Module type C extension Deprecated in 3.8, documented as deprecated since **2.5** To be removed in **3.9** Substitute ast, lib2to3.pgen2
pipes
The `pipes <https://docs.python.org/3/library/pipes.html>`_ module
provides
helpers to pipe the input of one command into the output of another
command.
The module is built on top of ``os.popen``. Users are encouraged to use the subprocess module instead.
Module type pure Python Deprecated in 3.8 To be removed in 3.10 Substitute subprocess module
Removed modules
fpectl
The `fpectl https://docs.python.org/3.6/library/fpectl.html`_ module
was
never built by default, its usage was discouraged and considered
dangerous.
It also required a configure flag that caused an ABI incompatibility. The module was removed in 3.7 by Nathaniel J. Smith in `bpo-29137 https://bugs.python.org/issue29137`_.
Module type C extension + CAPI Deprecated in 3.7 Removed in 3.7 Substitute **none**
Modules to keep
Some modules were originally proposed for deprecation.
fileinput
The `fileinput https://docs.python.org/3/library/fileinput.html`_
module
implements a helpers to iterate over a list of files from ``sys.argv``.
The
module predates the optparser and argparser module. The same
functionality
can be implemented with the argparser module.
Several core developers expressed their interest to keep the module in
the
standard library, as it is handy for quick scripts.
Module type pure Python
lib2to3
The `lib2to3 https://docs.python.org/3/library/2to3.html`_ package
provides
the ``2to3`` command to transpile Python 2 code to Python 3 code.
The package is useful for other tasks besides porting code from Python 2
to
- For example `black`_ uses it for code reformatting.
Module type pure Python
getopt
The `getopt https://docs.python.org/3/library/getopt.html`_ module
mimics
C's getopt() option parser.
Although users are encouraged to use argparse instead, the getopt module
is
still widely used. The module is small, simple, and handy for C
developers
to write simple Python scripts.
Module type pure Python Substitute argparse
optparse
The `optparse https://docs.python.org/3/library/optparse.html`_
module is
the predecessor of the argparse module.
Although it has been deprecated for many years, it's still too widely
used
to remove it.
Module type pure Python Deprecated in 3.2 Substitute argparse
wave
The `wave https://docs.python.org/3/library/wave.html`_ module
provides
support for the WAV sound format.
The module is not deprecated, because The WAV format is still relevant
these
days. The ``wave`` module is also used in education, e.g. to show kids
how
to make noise with a computer.
The module uses one simple function from the `audioop`_ module to perform byte swapping between little and big endian formats. Before 24 bit WAV support was added, byte swap used to be implemented with the ``array`` module. To remove ``wave``'s dependency on the ``audioop``, the byte swap function could be either be moved to another module (e.g. ``operator``)
or
the ``array`` module could gain support for 24 bit (3 byte) arrays.
Module type pure Python (depends on *byteswap* from `audioop`_ C extension) Deprecated in 3.8 To be removed in 3.10 Substitute *n/a*
Future maintenance of removed modules
The main goal of the PEP is to reduce the burden and workload on the
Python
core developer team. Therefore removed modules will not be maintained by the core team as separate PyPI packages. However the removed code, tests
and
documentation may be moved into a new git repository, so community
members
have a place from which they can pick up and fork code.
A first draft of a `legacylib https://github.com/tiran/legacylib`_ repository is available on my private Github account. The modules could
be
made available on PyPI. The Python core team will not publish or maintain the packages. It is my hope that members of the Python community will adopt, maintain, and perhaps improve the deprecated modules.
It's my hope that some of the deprecated modules will be picked up and adopted by users that actually care about them. For example ``colorsys``
and
``imghdr`` are useful modules, but have limited feature set. A fork of ``imghdr`` can add new features and support for more image formats,
without
being constrained by Python's release cycle.
Most of the modules are in pure Python and can be easily packaged. Some depend on a simple C module, e.g. `audioop`_ and `crypt`_. Since
`audioop`_
does not depend on any external libraries, it can be shipped in as binary wheels with some effort. Other C modules can be replaced with ctypes or
cffi.
For example I created `legacycrypt <https://github.com/tiran/legacycrypt `_ with ``_crypt`` extension reimplemented with a few lines of ctypes code.
Discussions
- Elana Hashman and Nick Coghlan suggested to keep the *getopt* module.
- Berker Peksag proposed to deprecate and removed *msilib*.
- Brett Cannon recommended to delay active deprecation warnings and
removal
of modules like *imp* until Python 3.10. Version 3.8 will be released shortly before Python 2 reaches end of lifetime. A delay reduced churn
for
users that migrate from Python 2 to 3.8.
- Brett also came up with the idea to keep lib2to3. The package is useful for other purposes, e.g. `black https://pypi.org/project/black/`_
uses
it to reformat Python code.
- At one point, distutils was mentioned in the same sentence as this PEP. To avoid lengthy discussion and delay of the PEP, I decided against
dealing
with distutils. Deprecation of the distutils package will be handled by another PEP.
- Multiple people (Gregory P. Smith, David Beazley, Nick Coghlan, ...) convinced me to keep the `wave`_ module. [4]_
- Gregory P. Smith proposed to deprecate `nntplib`_. [4]_
- Andrew Svetlov mentioned the ``socketserver`` module is questionable. However it's used to implement ``http.server`` and ``xmlrpc.server``.
The
stdlib doesn't have a replacement for the servers, yet.
Update history
Update 1
- Deprecate `parser`_ module
- Keep `fileinput`_ module
- Elaborate why `crypt`_ and `spwd`_ are dangerous and bad
- Improve sections for `cgitb`_, `colorsys`_, `nntplib`_, and `smtpd`_
modules
- The `colorsys`_, `crypt`_, `imghdr`_, `sndhdr`_, and `spwd`_ sections
now
list suitable substitutions.
- Mention that ``socketserver`` is going to stay for ``http.server`` and ``xmlrpc.server``
- The future maintenance section now states that the deprecated modules may be adopted by Python community members.
References
.. [1]
https://en.wikipedia.org/wiki/Open_Sound_System#Free,_proprietary,_free
.. [2] https://man.openbsd.org/ossaudio .. [3]
https://blogs.msmvps.com/installsite/blog/2015/05/03/the-future-of-windows-i...
.. [4] https://twitter.com/ChristianHeimes/status/1130257799475335169 .. [5] https://twitter.com/dabeaz/status/1130278844479545351 .. [6] https://mail.python.org/pipermail/python-dev/2019-May/157464.html .. [7]
https://discuss.python.org/t/switch-pythons-parsing-tech-to-something-more-p...
Copyright
This document has been placed in the public domain.
.. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe:
https://mail.python.org/mailman/options/python-dev/python-python-dev%40m.gma...
Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org
On 21/05/2019 17.35, Guido van Rossum wrote:
OK, sounds like nntplib can stay — this time.
On Tue, May 21, 2019 at 08:33 Antoine Pitrou <solipsis@pitrou.net mailto:solipsis@pitrou.net> wrote:
As I said, if the main annoyance with nntplib is the sporadic test failures, then the relevant tests can be disabled on CI. NNTP itself is still used, even if less and less.
I'd rather have it moved outside the core and maintained by people that actually care about NNTP support.
Greg, you proposed the deprecation of nntplib. Any comments?
Christian
On 21/05/2019 17.31, Antoine Pitrou wrote:
As I said, if the main annoyance with nntplib is the sporadic test failures, then the relevant tests can be disabled on CI.
NNTP itself is still used, even if less and less.
I don't like the idea to drop a third of the test cases for nntplib -- and the 30% that actually test that Python's NNTP library can talk to an actual NNTP server. IMHO it's more beneficial for core development and for nntplib to have it maintained by somebody that cares about the library.
Even a deprecation in 3.8 means that the module has to be maintained until EOL of 3.9, which is around 2026.
Christian
On Tue., May 21, 2019, 09:10 Christian Heimes, christian@python.org wrote:
On 21/05/2019 17.31, Antoine Pitrou wrote:
As I said, if the main annoyance with nntplib is the sporadic test failures, then the relevant tests can be disabled on CI.
NNTP itself is still used, even if less and less.
I don't like the idea to drop a third of the test cases for nntplib -- and the 30% that actually test that Python's NNTP library can talk to an actual NNTP server. IMHO it's more beneficial for core development and for nntplib to have it maintained by somebody that cares about the library.
Ditto for me. If the code isn't going to be properly tested then it's already unsupported, at which point we're back to wanting to deprecate it.
-Brett
Even a deprecation in 3.8 means that the module has to be maintained until EOL of 3.9, which is around 2026.
Christian _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/brett%40python.org
Le 21/05/2019 à 20:16, Brett Cannon a écrit :
On Tue., May 21, 2019, 09:10 Christian Heimes, <christian@python.org mailto:christian@python.org> wrote:
On 21/05/2019 17.31, Antoine Pitrou wrote: > > As I said, if the main annoyance with nntplib is the sporadic test > failures, then the relevant tests can be disabled on CI. > > NNTP itself is still used, even if less and less. I don't like the idea to drop a third of the test cases for nntplib -- and the 30% that actually test that Python's NNTP library can talk to an actual NNTP server. IMHO it's more beneficial for core development and for nntplib to have it maintained by somebody that cares about the library.
Ditto for me. If the code isn't going to be properly tested then it's already unsupported, at which point we're back to wanting to deprecate it.
So I should never have added those tests and then we wouldn't be talking about removing nntplib.
Regards
Antoine.
On Tue., May 21, 2019, 11:29 Antoine Pitrou, antoine@python.org wrote:
Le 21/05/2019 à 20:16, Brett Cannon a écrit :
On Tue., May 21, 2019, 09:10 Christian Heimes, <christian@python.org mailto:christian@python.org> wrote:
On 21/05/2019 17.31, Antoine Pitrou wrote: > > As I said, if the main annoyance with nntplib is the sporadic test > failures, then the relevant tests can be disabled on CI. > > NNTP itself is still used, even if less and less. I don't like the idea to drop a third of the test cases for nntplib -- and the 30% that actually test that Python's NNTP library can talk to an actual NNTP server. IMHO it's more beneficial for core development and for nntplib to have it maintained by somebody that cares about the library.
Ditto for me. If the code isn't going to be properly tested then it's already unsupported, at which point we're back to wanting to deprecate
it.
So I should never have added those tests and then we wouldn't be talking about removing nntplib.
Not necessarily. I suspect it still would have been listed, you would have objected, and someone may have looked at the tests to point out it was only unit tests with no integration tests against a live service like e.g. the pythontest.net-related tests and so it was already neglected.
It also doesn't help that no one is listed in the experts index for the module either.
-Brett
Regards
Antoine. _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/brett%40python.org
I repeat my position: on an omnibus PEP like this, if there's this much debate about one module, it should be struck from the list.
On Tue, May 21, 2019 at 5:46 PM Brett Cannon bcannon@gmail.com wrote:
On Tue., May 21, 2019, 11:29 Antoine Pitrou, antoine@python.org wrote:
Le 21/05/2019 à 20:16, Brett Cannon a écrit :
On Tue., May 21, 2019, 09:10 Christian Heimes, <christian@python.org mailto:christian@python.org> wrote:
On 21/05/2019 17.31, Antoine Pitrou wrote: > > As I said, if the main annoyance with nntplib is the sporadic test > failures, then the relevant tests can be disabled on CI. > > NNTP itself is still used, even if less and less. I don't like the idea to drop a third of the test cases for nntplib -- and the 30% that actually test that Python's NNTP library can talk to an actual NNTP server. IMHO it's more beneficial for core development and for nntplib to have it maintained by somebody that cares about the library.
Ditto for me. If the code isn't going to be properly tested then it's already unsupported, at which point we're back to wanting to deprecate
it.
So I should never have added those tests and then we wouldn't be talking about removing nntplib.
Not necessarily. I suspect it still would have been listed, you would have objected, and someone may have looked at the tests to point out it was only unit tests with no integration tests against a live service like e.g. the pythontest.net-related tests and so it was already neglected.
It also doesn't help that no one is listed in the experts index for the module either.
-Brett
Regards
Antoine. _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/brett%40python.org
Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org
On Tue, 21 May 2019 17:44:16 -0700 Brett Cannon bcannon@gmail.com wrote:
So I should never have added those tests and then we wouldn't be talking about removing nntplib.
Not necessarily. I suspect it still would have been listed, you would have objected, and someone may have looked at the tests to point out it was only unit tests with no integration tests against a live service like e.g. the pythontest.net-related tests and so it was already neglected.
But that's moving the goalposts. The PEP talks about "cruft, unnecessary duplication of functionality, and dispensable features". Not modules which have an insufficient test suite.
It also doesn't help that no one is listed in the experts index for the module either.
Right, but that's the case for many modules.
Regards
Antoine.
On Wed., May 22, 2019, 03:13 Antoine Pitrou, solipsis@pitrou.net wrote:
On Tue, 21 May 2019 17:44:16 -0700 Brett Cannon bcannon@gmail.com wrote:
So I should never have added those tests and then we wouldn't be
talking
about removing nntplib.
Not necessarily. I suspect it still would have been listed, you would
have
objected, and someone may have looked at the tests to point out it was
only
unit tests with no integration tests against a live service like e.g. the pythontest.net-related tests and so it was already neglected.
But that's moving the goalposts.
I was just replying to your hypothetical with another hypothetical.
The PEP talks about "cruft,
unnecessary duplication of functionality, and dispensable features". Not modules which have an insufficient test suite.
To me, untested code is a potential signal of cruft.
It also doesn't help that no one is listed in the experts index for the module either.
Right, but that's the case for many modules.
I never said I didn't view this PEP as conservative. 😉
Anyway, my support of dropping nntplib is +0 so I personally support Christian if he wants to keep on the list but I'm also not that bothered if he wants to take it off the list.
-Brett
Regards
Antoine.
Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/brett%40python.org
Dear python-dev,
I'm writing to provide some feedback on PEP-594, primarily the proposed deprecation and reason for the removal of the aifc and audioop libraries.
The post production film industry continues to make heavy use of AIFFs, as completely uncompressed audio is preferred. Support for the consumer alternatives (ALAC, FLAC) is virtually non-existent, with no movement towards adoption of those formats. Even Apple's own professional editing tool Final Cut Pro does not support ALAC. Many of the applications also support WAV, but not all.
Removal of this module from the standard library is complicated by the fact that a large number of film industry facilities have extremely limited internet access for security reasons. This does not make it impossible to get a library from pypi, but speaking to those devs has made me aware of what a painful process that is for them. They have benefited greatly from aifc's inclusion in the standard library.
While development of the aifc and audioop libraries has slowed, this looks consistent with mature projects and file formats that are not undergoing constant change. Checking the bug tracker for aifc and audioop does not paint a picture of these being broken or leaky modules requiring a lot of dev time, and there are 2 community sourced pull requests in progress right now.
The film industry may not make up a large chunk of python devs, but it is python that makes tent pole films like Avengers possible. Without the python digital asset management pipelines, films of that scale wouldn't be possible.
Thank you for your time and consideration,
Sean Wallitsch Lead Technical Director, DreamWorks Animation
On Wed, May 22, 2019 at 8:21 AM Brett Cannon brett@python.org wrote:
On Wed., May 22, 2019, 03:13 Antoine Pitrou, solipsis@pitrou.net wrote:
On Tue, 21 May 2019 17:44:16 -0700 Brett Cannon bcannon@gmail.com wrote:
So I should never have added those tests and then we wouldn't be
talking
about removing nntplib.
Not necessarily. I suspect it still would have been listed, you would
have
objected, and someone may have looked at the tests to point out it was
only
unit tests with no integration tests against a live service like e.g.
the
pythontest.net-related tests and so it was already neglected.
But that's moving the goalposts.
I was just replying to your hypothetical with another hypothetical.
The PEP talks about "cruft,
unnecessary duplication of functionality, and dispensable features". Not modules which have an insufficient test suite.
To me, untested code is a potential signal of cruft.
It also doesn't help that no one is listed in the experts index for the module either.
Right, but that's the case for many modules.
I never said I didn't view this PEP as conservative. 😉
Anyway, my support of dropping nntplib is +0 so I personally support Christian if he wants to keep on the list but I'm also not that bothered if he wants to take it off the list.
-Brett
Regards
Antoine.
Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/brett%40python.org
Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/sean.wallitsch%40dreamwor...
On Wed, May 22, 2019, 12:14 Sean Wallitsch sean.wallitsch@dreamworks.com wrote:
Dear python-dev,
I'm writing to provide some feedback on PEP-594, primarily the proposed deprecation and reason for the removal of the aifc and audioop libraries.
The post production film industry continues to make heavy use of AIFFs, as completely uncompressed audio is preferred. Support for the consumer alternatives (ALAC, FLAC) is virtually non-existent, with no movement towards adoption of those formats. Even Apple's own professional editing tool Final Cut Pro does not support ALAC. Many of the applications also support WAV, but not all.
Removal of this module from the standard library is complicated by the fact that a large number of film industry facilities have extremely limited internet access for security reasons. This does not make it impossible to get a library from pypi, but speaking to those devs has made me aware of what a painful process that is for them. They have benefited greatly from aifc's inclusion in the standard library.
That's really helpful data, thank you!
Is audioop also used? You mention both aifc and audioop at the beginning and end of your message, but all the details in the middle focus on just aifc.
-n
My apologies for that oversight. My understanding is that many of the methods present in aifc depend heavily on audioop for reading and writing.
On Wed, May 22, 2019 at 12:35 PM Nathaniel Smith njs@pobox.com wrote:
On Wed, May 22, 2019, 12:14 Sean Wallitsch sean.wallitsch@dreamworks.com wrote:
Dear python-dev,
I'm writing to provide some feedback on PEP-594, primarily the proposed deprecation and reason for the removal of the aifc and audioop libraries.
The post production film industry continues to make heavy use of AIFFs, as completely uncompressed audio is preferred. Support for the consumer alternatives (ALAC, FLAC) is virtually non-existent, with no movement towards adoption of those formats. Even Apple's own professional editing tool Final Cut Pro does not support ALAC. Many of the applications also support WAV, but not all.
Removal of this module from the standard library is complicated by the fact that a large number of film industry facilities have extremely limited internet access for security reasons. This does not make it impossible to get a library from pypi, but speaking to those devs has made me aware of what a painful process that is for them. They have benefited greatly from aifc's inclusion in the standard library.
That's really helpful data, thank you!
Is audioop also used? You mention both aifc and audioop at the beginning and end of your message, but all the details in the middle focus on just aifc.
-n
On Wed, May 22, 2019 at 1:23 PM Sean Wallitsch < sean.wallitsch@dreamworks.com> wrote:
My apologies for that oversight. My understanding is that many of the methods present in aifc depend heavily on audioop for reading and writing.
But are people using audioop directly? This shifts whether we still publicly deprecate auidoop and make it an implementation detail of aifc versus a free-standing module whose API we need to continue to support.
-Brett
On Wed, May 22, 2019 at 12:35 PM Nathaniel Smith njs@pobox.com wrote:
On Wed, May 22, 2019, 12:14 Sean Wallitsch sean.wallitsch@dreamworks.com wrote:
Dear python-dev,
I'm writing to provide some feedback on PEP-594, primarily the proposed deprecation and reason for the removal of the aifc and audioop libraries.
The post production film industry continues to make heavy use of AIFFs, as completely uncompressed audio is preferred. Support for the consumer alternatives (ALAC, FLAC) is virtually non-existent, with no movement towards adoption of those formats. Even Apple's own professional editing tool Final Cut Pro does not support ALAC. Many of the applications also support WAV, but not all.
Removal of this module from the standard library is complicated by the fact that a large number of film industry facilities have extremely limited internet access for security reasons. This does not make it impossible to get a library from pypi, but speaking to those devs has made me aware of what a painful process that is for them. They have benefited greatly from aifc's inclusion in the standard library.
That's really helpful data, thank you!
Is audioop also used? You mention both aifc and audioop at the beginning and end of your message, but all the details in the middle focus on just aifc.
-n
Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/brett%40python.org
On Thu, May 23, 2019 at 02:06:13PM -0700, Brett Cannon wrote:
On Wed, May 22, 2019 at 1:23 PM Sean Wallitsch < sean.wallitsch@dreamworks.com> wrote:
My apologies for that oversight. My understanding is that many of the methods present in aifc depend heavily on audioop for reading and writing.
But are people using audioop directly?
Yes they are.
https://old.reddit.com/r/Python/comments/brgl8v/pep_594_removing_dead_batter...
Shouldn't the PEP be responsable for establishing (as well as any negative can be proven) that these modules aren't used, rather than merely assuming that they aren't?
Of course it is hard to establish that a module isn't being used. Not all code can be searched for on the Internet, there's huge amounts of non-open source code written by users that never leaves their computer, or behind corporate firewalls.
The difficulty of proving this negative requires that we be especially conservative about removal. At the very least, we should demonstrate that the package is *an active burden*.
Its not enough to say "nobody has touched this module for ages" since stable, working code doesn't rot unless we change the language (or the std lib!) and break it.
So far, the PEPs record hasn't been too good: out of the 31 modules listed in the original draft, the PEP marks four as "keep", leaving 27 to be deprecated. So far, we've found:
- the sound libraries are in heavy use by hobbyists and the professional audio-visual industry; aifc in particular is a mature, stable library that isn't break and doesn't need much attention;
- cgi and cgitb are used by people who don't need a heavyweight HTML solution (see Reddit);
- colorsys, fileinput and nntplib (at least) have been added to the "keep" list;
- the removal of spwd (and crypt?) has been controversial.
So far, nearly 30% of the so-called "dead batteries" turn out to be not so dead after all.
I may have missed some. Nor have I even talked much about modules which I personally use occasionally, like binhex, because it's not about *me* its about the community. As much as I would prefer binhex to remain, if it is removed I will be disappointed but I'll make do.
Speaking of binhex, I tried to get a sense of how much of a burden it actually is. There was a comment by Guido in *2007*:
https://github.com/python/cpython/commit/34a042d301d6ab88645046a6dfa6c38265c...
"This is the last time I fix binhex. If it breaks again it goes in the dustbin"
which of course is Guido's perogative to say Won't Fix. Its now 2019 and it hasn't broken again, or at least it hasn't broken again sufficiently to make any difference. Serhey found one issue:
https://bugs.python.org/issue29566
but I'm not sure if that's actually a bug or not. There was a documentation bug (in the source code) in 2017:
https://bugs.python.org/issue29557
and one enhancement request to clean up the code:
https://bugs.python.org/issue34063
Other than that, Serhey touched binhex.py as part of a mass patch to almost the whole stdlib to modernise file handling to use "with".
So by my count, in 12 years since Guido's frustrated comment about binhex, it has seen:
- removal of an obsolete and ambiguous comment from the source code; - addition of a few with blocks to modernise file handling; - one enhancement request still outstanding; - one possibly a bug (or maybe not) still outstanding.
I may have missed some, but we're talking about one issue per three years. How is this a maintenance burden?
On Thu, May 23, 2019 at 5:45 PM Steven D'Aprano steve@pearwood.info wrote:
On Thu, May 23, 2019 at 02:06:13PM -0700, Brett Cannon wrote:
On Wed, May 22, 2019 at 1:23 PM Sean Wallitsch < sean.wallitsch@dreamworks.com> wrote:
My apologies for that oversight. My understanding is that many of the methods present in aifc depend heavily on audioop for reading and
writing.
But are people using audioop directly?
Yes they are.
https://old.reddit.com/r/Python/comments/brgl8v/pep_594_removing_dead_batter...
Shouldn't the PEP be responsable for establishing (as well as any negative can be proven) that these modules aren't used, rather than merely assuming that they aren't?
Of course it is hard to establish that a module isn't being used. Not all code can be searched for on the Internet, there's huge amounts of non-open source code written by users that never leaves their computer, or behind corporate firewalls.
Exactly, hence why we are going through the PEP process on this and not simply deprecating everything outright without public discussion.
The difficulty of proving this negative requires that we be especially conservative about removal. At the very least, we should demonstrate that the package is *an active burden*.
Its not enough to say "nobody has touched this module for ages" since stable, working code doesn't rot unless we change the language (or the std lib!) and break it.
So far, the PEPs record hasn't been too good: out of the 31 modules listed in the original draft, the PEP marks four as "keep", leaving 27 to be deprecated. So far, we've found:
I don't think it's fair to be saying Christian isn't doing "too good" simply because he took a stab at trying to figure out some way to know which modules would make sense to remove and he got it wrong for some of them from some people's perspective. As you pointed out, there's no scientific way to do this ahead of time due to closed source code (e.g. the VFX industry is not publishing all of their asset pipeline tools so there was no way to know ahead of time without asking like we are with the PEP), so he had to start somewhere. And this initial list was discussed at the PyCon US 2018 language summit as a good one to start from so he isn't entirely guessing without some initial support to try this list out.
- the sound libraries are in heavy use by hobbyists and the professional
audio-visual industry; aifc in particular is a mature, stable library that isn't break and doesn't need much attention;
- cgi and cgitb are used by people who don't need a heavyweight HTML
solution (see Reddit);
- colorsys, fileinput and nntplib (at least) have been added to the
"keep" list;
- the removal of spwd (and crypt?) has been controversial.
So far, nearly 30% of the so-called "dead batteries" turn out to be not so dead after all.
I may have missed some. Nor have I even talked much about modules which I personally use occasionally, like binhex, because it's not about *me* its about the community. As much as I would prefer binhex to remain, if it is removed I will be disappointed but I'll make do.
I personally think it's about both us and the community. The community can and does ask for stuff all the time, but we have to balance that with what we as a team are capable of handling and in my opinion the stdlib is too big right now for us to maintain appropriately. Plus there's an asymmetric ask here when the community says they want something while we have to keep it going.
Speaking of binhex, I tried to get a sense of how much of a burden it actually is. There was a comment by Guido in *2007*:
https://github.com/python/cpython/commit/34a042d301d6ab88645046a6dfa6c38265c...
"This is the last time I fix binhex. If it breaks again it goes in the dustbin"
which of course is Guido's perogative to say Won't Fix. Its now 2019 and it hasn't broken again, or at least it hasn't broken again sufficiently to make any difference. Serhey found one issue:
https://bugs.python.org/issue29566
but I'm not sure if that's actually a bug or not. There was a documentation bug (in the source code) in 2017:
https://bugs.python.org/issue29557
and one enhancement request to clean up the code:
https://bugs.python.org/issue34063
Other than that, Serhey touched binhex.py as part of a mass patch to almost the whole stdlib to modernise file handling to use "with".
So by my count, in 12 years since Guido's frustrated comment about binhex, it has seen:
- removal of an obsolete and ambiguous comment from the source code;
- addition of a few with blocks to modernise file handling;
- one enhancement request still outstanding;
- one possibly a bug (or maybe not) still outstanding.
I may have missed some, but we're talking about one issue per three years. How is this a maintenance burden?
Then multiply that by however many modules stay in the PEP -- and which we know don't all have such small stats compared to binhex -- and it starts to become death by a thousand paper cuts. Numerically all of that seems small, but all of that required time that we all have a limited amount of (both in the open source contribution sense and in the living-and-breathing sense). My point is none of this is free no matter how minuscule we think the cost is.
You can also flip that question around and ask why are there *any* enhancement requests or open issues if maintenance is so easy for binhex? Why has no one signed up for the module in the experts index if it's such a minor load to maintain? My guess is we all have other priorities that we value more, which makes total sense, but that also says to me that other things are taking higher priority. And for me, even ignoring a module still takes effort.
For me personally, a module not being in this PEP means we consider it critical to a large portion of users of Python where being "in the box" is an important aspect as well as having us directly maintain the module. If a module is in this PEP it says to me that we consider it a nice-to-have for the community, but not at the cost of the modules not listed, or the CPython interpreter, or the language itself which is the other stuff we are planning to keep maintaining after this PEP.
Anyway, in the end we are never all going to agree to what the definition is of what is or is not enough effort in maintenance to warrant keeping something in the stdlib. The best I think we can do is try to reach consensus and then try to figure out what guidelines we want to use for what does belong in the stdlib going forward along with trying to figure out how to maintain what we do choose to keep.
On Tue, May 21, 2019 at 7:11 PM Christian Heimes christian@python.org wrote:
On 21/05/2019 17.31, Antoine Pitrou wrote:
As I said, if the main annoyance with nntplib is the sporadic test failures, then the relevant tests can be disabled on CI.
NNTP itself is still used, even if less and less.
I don't like the idea to drop a third of the test cases for nntplib -- and the 30% that actually test that Python's NNTP library can talk to an actual NNTP server. IMHO it's more beneficial for core development and for nntplib to have it maintained by somebody that cares about the library.
https://github.com/python/cpython/pull/9461 should fix most of the annoyances with test_nntplib. It uses a local server instead of an external one.
--Berker
Christian,
Please don't touch nntplib.
Also I think telnetlib should stay.
On Wed, May 22, 2019 at 5:44 AM Berker Peksağ berker.peksag@gmail.com wrote:
On Tue, May 21, 2019 at 7:11 PM Christian Heimes christian@python.org wrote:
On 21/05/2019 17.31, Antoine Pitrou wrote:
As I said, if the main annoyance with nntplib is the sporadic test failures, then the relevant tests can be disabled on CI.
NNTP itself is still used, even if less and less.
I don't like the idea to drop a third of the test cases for nntplib --
and the 30% that actually test that Python's NNTP library can talk to an actual NNTP server. IMHO it's more beneficial for core development and for nntplib to have it maintained by somebody that cares about the library.
https://github.com/python/cpython/pull/9461 should fix most of the annoyances with test_nntplib. It uses a local server instead of an external one.
--Berker _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org
On Tue, 21 May 2019 at 21:13, Christian Heimes christian@python.org wrote:
crypt
The `crypt <https://docs.python.org/3/library/crypt.html>`_ module implements password hashing based on ``crypt(3)`` function from ``libcrypt`` or ``libxcrypt`` on Unix-like platform. The algorithms are mostly old, of poor quality and insecure. Users are discouraged to use them. * The module is not available on Windows. Cross-platform application need an alternative implementation any way. * Only DES encryption is guarenteed to be available. DES has an extremely limited key space of 2**56. * MD5, salted SHA256, salted SHA512, and Blowfish are optional extension. SSHA256 and SSHA512 are glibc extensions. Blowfish (bcrypt) is the only algorithm that is still secure. However it's in glibc and therefore not commonly available on Linux. * Depending on the platform, the ``crypt`` module is not thread safe. Only implementations with ``crypt_r(3)`` are thread safe. * The module was never useful to interact with system user and password databases.
This is actually not true. Their main use case is to compare passwords against the shadowed password db: https://github.com/giampaolo/pyftpdlib/blob/ee7b36c701b78b2d36e938c42d08dbfb... A quick search on searchcode.com shows both spwd and crypt modules are used. I am no security expert (and I wasn’t aware they are insecure until now, since the doc doesn’t mention it) but I would prefer seeing these 2 fixed or improved rather than bluntly removed.
--
Giampaolo - http://grodola.blogspot.com
On 21/05/2019 18.08, Giampaolo Rodola' wrote:
On Tue, 21 May 2019 at 21:13, Christian Heimes <christian@python.org mailto:christian@python.org> wrote:
crypt ~~~~~ The `crypt <https://docs.python.org/3/library/crypt.html>`_ module implements password hashing based on ``crypt(3)`` function from ``libcrypt`` or ``libxcrypt`` on Unix-like platform. The algorithms are mostly old, of poor quality and insecure. Users are discouraged to use them. * The module is not available on Windows. Cross-platform application need an alternative implementation any way. * Only DES encryption is guarenteed to be available. DES has an extremely limited key space of 2**56. * MD5, salted SHA256, salted SHA512, and Blowfish are optional extension. SSHA256 and SSHA512 are glibc extensions. Blowfish (bcrypt) is the only algorithm that is still secure. However it's in glibc and therefore not commonly available on Linux. * Depending on the platform, the ``crypt`` module is not thread safe. Only implementations with ``crypt_r(3)`` are thread safe. * The module was never useful to interact with system user and password databases.
This is actually not true. Their main use case is to compare passwords against the shadowed password db: https://github.com/giampaolo/pyftpdlib/blob/ee7b36c701b78b2d36e938c42d08dbfb... A quick search on searchcode.com http://searchcode.com shows both spwd and crypt modules are used. I am no security expert (and I wasn’t aware they are insecure until now, since the doc doesn’t mention it) but I would prefer seeing these 2 fixed or improved rather than bluntly removed.
No, the statement is correct. I may have to explain this even further.
The approach in pyftpdlib is the wrong and IMO deserves a CVE. The crypt() + spwd() approach is flawed on multiple levels. For example it bypasses account restriction, access control, and login session. It also requires you to run the service as root and with SELinux disabled or an unconfined context -- a bad combination. There is only one correct way to perform a credential check: use PAM.
spwd can't be fixed. It could only be replaced with a completely different API that wraps PAM and Windows's authentication API.
Christian
PS: Authentication, authorization, and identity management are part of my day job at Red Hat.
On Tue, 21 May 2019 at 23:26, Christian Heimes christian@python.org wrote:
On 21/05/2019 18.08, Giampaolo Rodola' wrote:
On Tue, 21 May 2019 at 21:13, Christian Heimes <christian@python.org
mailto:christian@python.org> wrote:
crypt ~~~~~ The `crypt <https://docs.python.org/3/library/crypt.html>`_ module
implements
password hashing based on ``crypt(3)`` function from ``libcrypt`` or ``libxcrypt`` on Unix-like platform. The algorithms are mostly old,
of poor
quality and insecure. Users are discouraged to use them. * The module is not available on Windows. Cross-platform application
need
an alternative implementation any way. * Only DES encryption is guarenteed to be available. DES has an
extremely
limited key space of 2**56. * MD5, salted SHA256, salted SHA512, and Blowfish are optional
extension.
SSHA256 and SSHA512 are glibc extensions. Blowfish (bcrypt) is the
only
algorithm that is still secure. However it's in glibc and
therefore not
commonly available on Linux. * Depending on the platform, the ``crypt`` module is not thread
safe. Only
implementations with ``crypt_r(3)`` are thread safe. * The module was never useful to interact with system user and
password
databases.
This is actually not true. Their main use case is to compare passwords
against the shadowed password db:
https://github.com/giampaolo/pyftpdlib/blob/ee7b36c701b78b2d36e938c42d08dbfb...
A quick search on searchcode.com http://searchcode.com shows both
spwd and crypt modules are used. I am no security expert (and I wasn’t aware they are insecure until now, since the doc doesn’t mention it) but I would prefer seeing these 2 fixed or improved rather than bluntly removed.
No, the statement is correct. I may have to explain this even further.
The approach in pyftpdlib is the wrong and IMO deserves a CVE. The crypt()
- spwd() approach is flawed on multiple levels. For example it bypasses
account restriction, access control, and login session. It also requires you to run the service as root and with SELinux disabled or an unconfined context -- a bad combination. There is only one correct way to perform a credential check: use PAM.
spwd can't be fixed. It could only be replaced with a completely different API that wraps PAM and Windows's authentication API.
Christian
PS: Authentication, authorization, and identity management are part of my day job at Red Hat.
Got it. I had no idea. Since you mentioned the CVE it looks like spwd/crypt doc deserve a warning. This is probably out of the scope of the PEP, but I wonder if the 3 third-party alternatives mentioned in the PEP are mature enough and could be evaluated for stdlib inclusion (the part re. PAM / password-checking at least). Perhaps spwd/crypt could be deprecated in 3.8 and the alternative added in 3.9 before the 3.10 removal.
On 21/05/2019 20.18, Giampaolo Rodola' wrote:
No, the statement is correct. I may have to explain this even further. The approach in pyftpdlib is the wrong and IMO deserves a CVE. The crypt() + spwd() approach is flawed on multiple levels. For example it bypasses account restriction, access control, and login session. It also requires you to run the service as root and with SELinux disabled or an unconfined context -- a bad combination. There is only one correct way to perform a credential check: use PAM. spwd can't be fixed. It could only be replaced with a completely different API that wraps PAM and Windows's authentication API. Christian PS: Authentication, authorization, and identity management are part of my day job at Red Hat.
Got it. I had no idea. Since you mentioned the CVE it looks like spwd/crypt doc deserve a warning. This is probably out of the scope of the PEP, but I wonder if the 3 third-party alternatives mentioned in the PEP are mature enough and could be evaluated for stdlib inclusion (the part re. PAM / password-checking at least). Perhaps spwd/crypt could be deprecated in 3.8 and the alternative added in 3.9 before the 3.10 removal.
Sorry, I didn't even occur to me that anybody was still using spwd. I would have added a warning much earlier. There is now https://bugs.python.org/issue36997 to track the problem.
Could you do me a favor and open a feature request?
On 21/05/2019 18.08, Giampaolo Rodola' wrote:
On Tue, 21 May 2019 at 21:13, Christian Heimes <christian@python.org mailto:christian@python.org> wrote:
crypt ~~~~~ The `crypt <https://docs.python.org/3/library/crypt.html>`_ module implements password hashing based on ``crypt(3)`` function from ``libcrypt`` or ``libxcrypt`` on Unix-like platform. The algorithms are mostly old, of poor quality and insecure. Users are discouraged to use them. * The module is not available on Windows. Cross-platform application need an alternative implementation any way. * Only DES encryption is guarenteed to be available. DES has an extremely limited key space of 2**56. * MD5, salted SHA256, salted SHA512, and Blowfish are optional extension. SSHA256 and SSHA512 are glibc extensions. Blowfish (bcrypt) is the only algorithm that is still secure. However it's in glibc and therefore not commonly available on Linux. * Depending on the platform, the ``crypt`` module is not thread safe. Only implementations with ``crypt_r(3)`` are thread safe. * The module was never useful to interact with system user and password databases.
This is actually not true. Their main use case is to compare passwords against the shadowed password db: https://github.com/giampaolo/pyftpdlib/blob/ee7b36c701b78b2d36e938c42d08dbfb... A quick search on searchcode.com http://searchcode.com shows both spwd and crypt modules are used. I am no security expert (and I wasn’t aware they are insecure until now, since the doc doesn’t mention it) but I would prefer seeing these 2 fixed or improved rather than bluntly removed.
PS: "pw1 != pw2" on line 418 is also vulnerable to side channel attack. You must use a constant timing comparison operator like hmac.compare_digest() to compare password digests.
https://github.com/giampaolo/pyftpdlib/blob/ee7b36c701b78b2d36e938c42d08dbfb...
On 5/21/2019 10:12 AM, Christian Heimes wrote:
PEP: 594 Title: Removing dead batteries from the standard library
'dead' seems controversial. 'nearly useless' should be less so. I think 'after 2.7 end-of-life' is worth adding, as the schedule hinges around that. Result:
"Removing nearly useless batteries from the standard library after 2.7 end-of-life."
Abstract
Rationale
Part of the point of the PEP is to gather together nearly all proposals in one place so users can respond in a timely fashion.
Deprecation schedule
I think it worth adding that some modules were deprecated years ago but kept until after 2.7 end-of-life to ease 3.x transition. Removing modules is not a sudden new idea.
Also, the proposed schedule (with one exception) is a slower alternative than proposals to add a deprecation warning (if not one already) in 3.8 and remove in 3.9.
Hi, thanks for the work on this proposal, I think this is at least a tip of the iceberg and a good start for the bigger question of how the stdlib should evolve..
I think that the PEP should include an idea of what should happen if existing stdlib pieces depend on this deprecated modules. For example, email.mime.audio is part of a non-deprecated module but it depends on sndhdr which is being planned for deprecation. Should that part of the functionality be deprecated too? rewritten to not depend on the deprecated module (but keep the pieces of relevant code? depend on optional 3rd party modules and degrade if those are not available?
The sndhdr case is just an example but I can imagine there are others (optparse is no longer schedule for removal in your PEP, but would have been another case, being used by the profile module).
Best, Daniel
On Tue, 21 May 2019 at 15:15, Christian Heimes christian@python.org wrote:
Hi,
I have updated the PEP with feedback from discussions. The highlights are:
- Deprecate parser module
- Keep fileinput module
- Elaborate why crypt and spwd are dangerous and bad
- Improve sections for cgitb, colorsys, nntplib, and smtpd modules
- The colorsys, crypt, imghdr, sndhdr, and spwd sections now list suitable
substitutions.
- Mention that socketserver is going to stay for http.server and
xmlrpc.server
- The future maintenance section now states that the deprecated modules
may be adopted by Python community members.
https://github.com/python/peps/compare/7799178a...2d536899?diff=unified#diff...
I'll be traveling the next couple of days and will only have limited opportunities to respond on feedback.
Christian
PEP: 594 Title: Removing dead batteries from the standard library Author: Christian Heimes christian@python.org Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 20-May-2019 Post-History: 21-May-2019
Abstract
This PEP proposed a list of standard library modules to be removed from the standard library. The modules are mostly historic data formats and APIs that have been superseded a long time ago, e.g. Mac OS 9 and Commodore.
Rationale
Back in the early days of Python, the interpreter came with a large set of useful modules. This was often refrained to as "batteries included" philosophy and was one of the corner stones to Python's success story. Users didn't have to figure out how to download and install separate packages in order to write a simple web server or parse email.
Times have changed. The introduction of the cheese shop (PyPI), setuptools, and later pip, it became simple and straight forward to download and install packages. Nowadays Python has a rich and vibrant ecosystem of third party packages. It's pretty much standard to either install packages from PyPI or use one of the many Python or Linux distributions.
On the other hand, Python's standard library is piling up cruft, unnecessary duplication of functionality, and dispensable features. This is undesirable for several reasons.
- Any additional module increases the maintenance cost for the Python core development team. The team has limited resources, reduced maintenance
cost frees development time for other improvements.
- Modules in the standard library are generally favored and seen as the de-facto solution for a problem. A majority of users only pick 3rd party modules to replace a stdlib module, when they have a compelling reason,
e.g. lxml instead of `xml`. The removal of an unmaintained stdlib module increases the chances of a community contributed module to become widely used.
- A lean and mean standard library benefits platforms with limited
resources like devices with just a few hundred kilobyte of storage (e.g. BBC Micro:bit). Python on mobile platforms like BeeWare or WebAssembly (e.g. pyodide) also benefit from reduced download size.
The modules in the PEP have been selected for deprecation because their removal is either least controversial or most beneficial. For example least controversial are 30 years old multimedia formats like ``sunau`` audio format, which was used on SPARC and NeXT workstations in the late 1980ties. The ``crypt`` module has fundamental flaws that are better solved outside the standard library.
This PEP also designates some modules as not scheduled for removal. Some modules have been deprecated for several releases or seem unnecessary at first glance. However it is beneficial to keep the modules in the standard library, mostly for environments where installing a package from PyPI is not an option. This can be cooperate environments or class rooms where external code is not permitted without legal approval.
- The usage of FTP is declining, but some files are still provided over the FTP protocol or hosters offer FTP to upload content. Therefore ``ftplib`` is going to stay.
- The ``optparse`` and ``getopt`` module are widely used. They are mature modules with very low maintenance overhead.
- According to David Beazley [5]_ the ``wave`` module is easy to teach to kids and can make crazy sounds. Making a computer generate crazy sounds
is powerful and highly motivating exercise for a 9yo aspiring developer. It's a fun battery to keep.
Deprecation schedule
3.8
This PEP targets Python 3.8. Version 3.8.0 final is scheduled to be released a few months before Python 2.7 will reach its end of lifetime. We expect that Python 3.8 will be targeted by users that migrate to Python 3 in 2019 and 2020. To reduce churn and to allow a smooth transition from Python 2, Python 3.8 will neither raise `DeprecationWarning` nor remove any modules that have been scheduled for removal. Instead deprecated modules will just be *documented* as deprecated. Optionally modules may emit a `PendingDeprecationWarning`.
All deprecated modules will also undergo a feature freeze. No additional features should be added. Bug should still be fixed.
3.9
Starting with Python 3.9, deprecated modules will start issuing `DeprecationWarning`. The `parser`_ module is removed and potentially replaced with a new module.
All other deprecated modules are fully supported and will receive security updates until Python 3.9 reaches its end of lifetime. Python 3.9.0 will be released about 18 months after 3.8.0 (April 2021?) and most likely be supported for 5 years after the release. The estimated EOL of Python 3.9 is in 2026.
3.10
In 3.10 all deprecated modules will be removed from the CPython repository together with tests, documentation, and autoconf rules.
PEP acceptance process
3.8.0b1 is scheduled to be release shortly after the PEP is officially submitted. Since it's improbable that the PEP will pass all stages of the PEP process in time, I propose a two step acceptance process that is analogous Python's two release deprecation process.
The first *provisionally accepted* phase targets Python 3.8.0b1. In the first phase no code is changes or removed. Modules are only documented as deprecated. The only exception is the `parser`_ module. It has been documented as deprecated since Python 2.5 and is scheduled for removal for 3.9 to make place for a more advanced parser.
The final decision, which modules will be removed and how the removed code is preserved, can be delayed for another year.
Deprecated modules
The modules are grouped as data encoding, multimedia, network, OS interface, and misc modules. The majority of modules are for old data formats or old APIs. Some others are rarely useful and have better replacements on PyPI, e.g. Pillow for image processing or NumPy-based projects to deal with audio processing.
.. csv-table:: Table 1: Proposed modules deprecations :header: "Module", "Deprecated in", "To be removed", "Replacement" :widths: 1, 1, 1, 2
aifc,3.8,3.10,\- asynchat,3.8,3.10,asyncio asyncore,3.8,3.10,asyncio audioop,3.8,3.10,\- binhex,3.8,3.10,\- cgi,3.8,3.10,\- cgitb,3.8,3.10,\- chunk,3.8,3.10,\- colorsys,3.8,3.10,"colormath, colour, colorspacious, Pillow" crypt,3.8,3.10,"bcrypt, argon2cffi, hashlib, passlib" fileinput,\-,**keep**,argparse formatter,3.4,3.10,\- fpectl,**3.7**,**3.7**,\- getopt,**3.2**,**keep**,"argparse, optparse" imghdr,3.8,3.10,"filetype, puremagic, python-magic" imp,**3.4**,3.10,importlib lib2to3,\-,**keep**, macpath,**3.7**,**3.8**,\- msilib,3.8,3.10,\- nntplib,3.8,3.10,\- nis,3.8,3.10,\- optparse,\-,**keep**,argparse ossaudiodev,3.8,3.10,\- parser,**2.5**,**3.9**,"ast, lib2to3.pgen2" pipes,3.8,3.10,subprocess smtpd,"**3.4.7**, **3.5.4**",3.10,aiosmtpd sndhdr,3.8,3.10,"filetype, puremagic, python-magic" spwd,3.8,3.10,"python-pam, simplepam" sunau,3.8,3.10,\- uu,3.8,3.10,\- wave,\-,**keep**, xdrlib,3.8,3.10,\-
Data encoding modules
binhex
The `binhex <https://docs.python.org/3/library/binhex.html>`_ module encodes and decodes Apple Macintosh binhex4 data. It was originally developed for TSR-80. In the 1980s and early 1990s it was used on classic Mac OS 9 to encode binary email attachments. Module type pure Python Deprecated in 3.8 To be removed in 3.10 Substitute **none** uu ~~ The `uu <https://docs.python.org/3/library/uu.html>`_ module provides uuencode format, an old binary encoding format for email from 1980. The uu format has been replaced by MIME. The uu codec is provided by the binascii module. Module type pure Python Deprecated in 3.8 To be removed in 3.10 Substitute **none** xdrlib
The `xdrlib https://docs.python.org/3/library/xdrlib.html`_ module supports the Sun External Data Representation Standard. XDR is an old binary serialization format from 1987. These days it's rarely used outside specialized domains like NFS.
Module type pure Python Deprecated in 3.8 To be removed in 3.10 Substitute **none**
Multimedia modules
aifc
The `aifc <https://docs.python.org/3/library/aifc.html>`_ module provides support for reading and writing AIFF and AIFF-C files. The Audio Interchange File Format is an old audio format from 1988 based on Amiga IFF. It was most commonly used on the Apple Macintosh. These days only few specialized application use AIFF. Module type pure Python (depends on `audioop`_ C extension) Deprecated in 3.8 To be removed in 3.10 Substitute **none** audioop
The `audioop https://docs.python.org/3/library/audioop.html`_ module contains helper functions to manipulate raw audio data and adaptive differential pulse-code modulated audio data. The module is implemented in C without any additional dependencies. The `aifc`_, `sunau`_, and `wave`_ module depend on `audioop`_ for some operations. The byteswap operation in the `wave`_ module can be substituted with little work.
Module type C extension Deprecated in 3.8 To be removed in 3.10 Substitute **none**
colorsys
The `colorsys <https://docs.python.org/3/library/colorsys.html>`_ module defines color conversion functions between RGB, YIQ, HSL, and HSV coordinate systems. The PyPI packages *colormath*, *colour*, and *colorspacious* provide more and advanced features. The Pillow library is better suited to transform images between color systems. Module type pure Python Deprecated in 3.8 To be removed in 3.10 Substitute `colormath <https://pypi.org/project/colormath/>`_, `colour <https://pypi.org/project/colour/>`_ `colorspacious <https://pypi.org/project/colorspacious/>`_, `Pillow <https://pypi.org/project/Pillow/>`_ chunk ~~~~~ The `chunk <https://docs.python.org/3/library/chunk.html>`_ module provides support for reading and writing Electronic Arts' Interchange File Format. IFF is an old audio file format originally introduced for Commodore and Amiga. The format is no longer relevant. Module type pure Python Deprecated in 3.8 To be removed in 3.10 Substitute **none** imghdr ~~~~~~ The `imghdr <https://docs.python.org/3/library/imghdr.html>`_ module is a simple tool to guess the image file format from the first 32 bytes of a file or buffer. It supports only a limited amount of formats and neither returns resolution nor color depth. Module type pure Python Deprecated in 3.8 To be removed in 3.10 Substitute `puremagic <https://pypi.org/project/puremagic/>`_, `filetype <https://pypi.org/project/filetype/>`_, `python-magic <https://pypi.org/project/python-magic/>`_ ossaudiodev
The `ossaudiodev https://docs.python.org/3/library/ossaudiodev.html`_ module provides support for Open Sound System, an interface to sound playback and capture devices. OSS was initially free software, but later support for newer sound devices and improvements were proprietary. Linux community abandoned OSS in favor of ALSA [1]_. Some operation systems like OpenBSD and NetBSD provide an incomplete [2]_ emulation of OSS.
Module type C extension Deprecated in 3.8 To be removed in 3.10 Substitute **none**
sndhdr
The `sndhdr <https://docs.python.org/3/library/sndhdr.html>`_ module is similar to the `imghdr`_ module but for audio formats. It guesses file format, channels, frame rate, and sample widths from the first 512 bytes of a file or buffer. The module only supports AU, AIFF, HCOM, VOC, WAV, and other ancient formats. Module type pure Python (depends on `audioop`_ C extension for some operations) Deprecated in 3.8 To be removed in 3.10 Substitute `puremagic <https://pypi.org/project/puremagic/>`_, `filetype <https://pypi.org/project/filetype/>`_, `python-magic <https://pypi.org/project/python-magic/>`_ sunau ~~~~~ The `sunau <https://docs.python.org/3/library/sunhdr.html>`_ module provides support for Sun AU sound format. It's yet another old, obsolete file format. Module type pure Python (depends on `audioop`_ C extension for some operations) Deprecated in 3.8 To be removed in 3.10 Substitute **none** Networking modules ------------------ asynchat
The `asynchat https://docs.python.org/3/library/asynchat.html`_ module is build on top of `asyncore`_ and has been deprecated since Python 3.6.
Module type pure Python Deprecated in 3.6 Removed in 3.10 Substitute asyncio
asyncore
The `asyncore <https://docs.python.org/3/library/asyncore.html>`_ module was the first module for asynchronous socket service clients and servers. It has been replaced by asyncio and is deprecated since Python 3.6. The ``asyncore`` module is also used in stdlib tests. The tests for ``ftplib``, ``logging``, ``smptd``, ``smtplib``, and ``ssl`` are partly based on ``asyncore``. These tests must be updated to use asyncio or threading. Module type pure Python Deprecated in 3.6 Removed in 3.10 Substitute asyncio cgi ~~~ The `cgi <https://docs.python.org/3/library/cgi.html>`_ module is a support module for Common Gateway Interface (CGI) scripts. CGI is deemed as inefficient because every incoming request is handled in a new process. PEP 206 considers the module as *designed poorly and are now near-impossible to fix*. Several people proposed to either keep the cgi module for features like `cgi.parse_qs()` or move `cgi.escape()` to a different module. The functions `cgi.parse_qs` and `cgi.parse_qsl` have been deprecated for a while and are actually aliases for `urllib.parse.parse_qs` and `urllib.parse.parse_qsl`. The function `cgi.quote` has been deprecated in favor of `html.quote` with secure default values. Module type pure Python Deprecated in 3.8 To be removed in 3.10 Substitute **none** cgitb ~~~~~ The `cgitb <https://docs.python.org/3/library/cgitb.html>`_ module is a helper for the cgi module for configurable tracebacks. The ``cgitb`` module is not used by any major Python web framework (Django, Pyramid, Plone, Flask, CherryPy, or Bottle). Only Paste uses it in an optional debugging middleware. Module type pure Python Deprecated in 3.8 To be removed in 3.10 Substitute **none** smtpd ~~~~~ The `smtpd <https://docs.python.org/3/library/smtpd.html>`_ module provides a simple implementation of a SMTP mail server. The module documentation marks the module as deprecated and recommends ``aiosmtpd`` instead. The deprecation message was added in releases 3.4.7, 3.5.4, and 3.6.1. Module type pure Python Deprecated in **3.7** To be removed in 3.10 Substitute aiosmtpd nntplib ~~~~~~~ The `nntplib <https://docs.python.org/3/library/nntplib.html>`_ module implements the client side of the Network News Transfer Protocol (nntp). News groups used to be a dominant platform for online discussions. Over the last two decades, news has been slowly but steadily replaced with mailing lists and web-based discussion platforms. Twisted is also `planning <https://twistedmatrix.com/trac/ticket/9405>`_ to deprecate NNTP support and `pynnt <https://github.com/greenbender/pynntp>`_ hasn't seen any activity since 2014. This is a good indicator that the public interest in NNTP support is declining. The ``nntplib`` tests have been the cause of additional work in the recent past. Python only contains client side of NNTP. The tests connect to external news server. The servers are sometimes unavailble, too slow, or do not work correctly over IPv6. The situation causes flaky test runs on buildbots. Module type pure Python Deprecated in 3.8 To be removed in 3.10 Substitute **none** Operating system interface -------------------------- crypt ~~~~~ The `crypt <https://docs.python.org/3/library/crypt.html>`_ module implements password hashing based on ``crypt(3)`` function from ``libcrypt`` or ``libxcrypt`` on Unix-like platform. The algorithms are mostly old, of poor quality and insecure. Users are discouraged to use them. * The module is not available on Windows. Cross-platform application need an alternative implementation any way. * Only DES encryption is guarenteed to be available. DES has an extremely limited key space of 2**56. * MD5, salted SHA256, salted SHA512, and Blowfish are optional extension. SSHA256 and SSHA512 are glibc extensions. Blowfish (bcrypt) is the only algorithm that is still secure. However it's in glibc and therefore not commonly available on Linux. * Depending on the platform, the ``crypt`` module is not thread safe. Only implementations with ``crypt_r(3)`` are thread safe. * The module was never useful to interact with system user and password databases. On BSD, macOS, and Linux, all user authentication and password modification operations must go through PAM (pluggable authentication module), see `spwd`_ deprecation. Module type C extension + Python module Deprecated in 3.8 To be removed in 3.10 Substitute `bcrypt <https://pypi.org/project/bcrypt/>`_, `passlib <https://pypi.org/project/passlib/>`_, `argon2cffi <https://pypi.org/project/argon2-cffi/>`_, hashlib module (PBKDF2, scrypt) macpath ~~~~~~~ The `macpath <https://docs.python.org/3/library/macpath.html>`_ module provides Mac OS 9 implementation of os.path routines. Mac OS 9 is no longer supported Module type pure Python Deprecated in 3.7 Removed in 3.8 Substitute **none** nis ~~~ The `nis <https://docs.python.org/3/library/nis.html>`_ module provides NIS/YP support. Network Information Service / Yellow Pages is an old and deprecated directory service protocol developed by Sun Microsystems. It's designed successor NIS+ from 1992 never took off. For a long time, libc's Name Service Switch, LDAP, and Kerberos/GSSAPI are considered a more powerful and more secure replacement of NIS. Module type C extension Deprecated in 3.8 To be removed in 3.10 Substitute **none** spwd ~~~~ The `spwd <https://docs.python.org/3/library/spwd.html>`_ module provides direct access to Unix shadow password database using non-standard APIs. In general it's a bad idea to use the spwd. The spwd circumvents system security policies, it does not use the PAM stack, and is only compatible with local user accounts, because it ignores NSS. The use of the ``spwd`` module for access control must be consider a *security bug*, as it bypasses PAM's access control. Further more the ``spwd`` module uses the `shadow(3) <http://man7.org/linux/man-pages/man3/shadow.3.html>`_ APIs. Functions like ``getspnam(3)`` access the ``/etc/shadow`` file directly. This is dangerous and even forbidden for confined services on systems with a security engine like SELinux or AppArmor. Module type C extension Deprecated in 3.8 To be removed in 3.10 Substitute `python-pam <https://pypi.org/project/python-pam/>`_, `simpleplam <https://pypi.org/project/simplepam/>`_ Misc modules ------------ formatter
The `formatter https://docs.python.org/3/library/formatter.html`_ module is an old text formatting module which has been deprecated since Python 3.4.
Module type pure Python Deprecated in 3.4 To be removed in 3.10 Substitute *n/a*
imp
The `imp <https://docs.python.org/3/library/imp.html>`_ module is the predecessor of the `importlib <https://docs.python.org/3/library/importlib.html>`_ module. Most functions have been deprecated since Python 3.3 and the module since Python 3.4. Module type C extension Deprecated in 3.4 To be removed in 3.10 Substitute importlib msilib
The `msilib https://docs.python.org/3/library/msilib.html`_ package is a Windows-only package. It supports the creation of Microsoft Installers (MSI). The package also exposes additional APIs to create cabinet files (CAB). The module is used to facilitate distutils to create MSI installers with ``bdist_msi`` command. In the past it was used to create CPython's official Windows installer, too.
Microsoft is slowly moving away from MSI in favor of Windows 10 Apps (AppX) as new deployment model [3]_.
Module type C extension + Python code Deprecated in 3.8 To be removed in 3.10 Substitute **none**
parser
The `parser <https://docs.python.org/3/library/parser.html>`_ module provides an interface to Python’s internal parser and byte-code compiler. The stdlib has superior ways to interact with the parse tree. From Python 2.5 onward, it's much more convenient to cut in at the Abstract Syntax Tree (AST) generation and compilation stage. The ``parser`` module causes additional work. It's C code that must be kept in sync with any change to Python's grammar and internal parser. Pablo wants to remove the parser module and promote lib2to3's pgen2 instead [6]_. Most importantly the presence of the ``parser`` module makes it harder to switch to something more powerful than a LL(1) parser [7]_. Since the ``parser`` module is documented as deprecated since Python 2.5 and a new parsing technology is planned for 3.9, the ``parser`` module is scheduled for removal in 3.9. Module type C extension Deprecated in 3.8, documented as deprecated since **2.5** To be removed in **3.9** Substitute ast, lib2to3.pgen2 pipes ~~~~~ The `pipes <https://docs.python.org/3/library/pipes.html>`_ module provides helpers to pipe the input of one command into the output of another command. The module is built on top of ``os.popen``. Users are encouraged to use the subprocess module instead. Module type pure Python Deprecated in 3.8 To be removed in 3.10 Substitute subprocess module Removed modules =============== fpectl ------ The `fpectl <https://docs.python.org/3.6/library/fpectl.html>`_ module was never built by default, its usage was discouraged and considered dangerous. It also required a configure flag that caused an ABI incompatibility. The module was removed in 3.7 by Nathaniel J. Smith in `bpo-29137 <https://bugs.python.org/issue29137>`_. Module type C extension + CAPI Deprecated in 3.7 Removed in 3.7 Substitute **none** Modules to keep =============== Some modules were originally proposed for deprecation. fileinput --------- The `fileinput <https://docs.python.org/3/library/fileinput.html>`_ module implements a helpers to iterate over a list of files from ``sys.argv``. The module predates the optparser and argparser module. The same functionality can be implemented with the argparser module. Several core developers expressed their interest to keep the module in the standard library, as it is handy for quick scripts. Module type pure Python lib2to3 ------- The `lib2to3 <https://docs.python.org/3/library/2to3.html>`_ package provides the ``2to3`` command to transpile Python 2 code to Python 3 code. The package is useful for other tasks besides porting code from Python 2 to 3. For example `black`_ uses it for code reformatting. Module type pure Python getopt ------ The `getopt <https://docs.python.org/3/library/getopt.html>`_ module mimics C's getopt() option parser. Although users are encouraged to use argparse instead, the getopt module is still widely used. The module is small, simple, and handy for C developers to write simple Python scripts. Module type pure Python Substitute argparse optparse -------- The `optparse <https://docs.python.org/3/library/optparse.html>`_ module is the predecessor of the argparse module. Although it has been deprecated for many years, it's still too widely used to remove it. Module type pure Python Deprecated in 3.2 Substitute argparse wave ---- The `wave <https://docs.python.org/3/library/wave.html>`_ module provides support for the WAV sound format. The module is not deprecated, because The WAV format is still relevant these days. The ``wave`` module is also used in education, e.g. to show kids how to make noise with a computer. The module uses one simple function from the `audioop`_ module to perform byte swapping between little and big endian formats. Before 24 bit WAV support was added, byte swap used to be implemented with the ``array`` module. To remove ``wave``'s dependency on the ``audioop``, the byte swap function could be either be moved to another module (e.g. ``operator``) or the ``array`` module could gain support for 24 bit (3 byte) arrays. Module type pure Python (depends on *byteswap* from `audioop`_ C extension) Deprecated in 3.8 To be removed in 3.10 Substitute *n/a* Future maintenance of removed modules ===================================== The main goal of the PEP is to reduce the burden and workload on the Python core developer team. Therefore removed modules will not be maintained by the core team as separate PyPI packages. However the removed code, tests and documentation may be moved into a new git repository, so community members have a place from which they can pick up and fork code. A first draft of a `legacylib <https://github.com/tiran/legacylib>`_ repository is available on my private Github account. The modules could be made available on PyPI. The Python core team will not publish or maintain the packages. It is my hope that members of the Python community will adopt, maintain, and perhaps improve the deprecated modules. It's my hope that some of the deprecated modules will be picked up and adopted by users that actually care about them. For example ``colorsys`` and ``imghdr`` are useful modules, but have limited feature set. A fork of ``imghdr`` can add new features and support for more image formats, without being constrained by Python's release cycle. Most of the modules are in pure Python and can be easily packaged. Some depend on a simple C module, e.g. `audioop`_ and `crypt`_. Since `audioop`_ does not depend on any external libraries, it can be shipped in as binary wheels with some effort. Other C modules can be replaced with ctypes or cffi. For example I created `legacycrypt <https://github.com/tiran/legacycrypt >`_ with ``_crypt`` extension reimplemented with a few lines of ctypes code. Discussions =========== * Elana Hashman and Nick Coghlan suggested to keep the *getopt* module. * Berker Peksag proposed to deprecate and removed *msilib*. * Brett Cannon recommended to delay active deprecation warnings and removal of modules like *imp* until Python 3.10. Version 3.8 will be released shortly before Python 2 reaches end of lifetime. A delay reduced churn for users that migrate from Python 2 to 3.8. * Brett also came up with the idea to keep lib2to3. The package is useful for other purposes, e.g. `black <https://pypi.org/project/black/>`_ uses it to reformat Python code. * At one point, distutils was mentioned in the same sentence as this PEP. To avoid lengthy discussion and delay of the PEP, I decided against dealing with distutils. Deprecation of the distutils package will be handled by another PEP. * Multiple people (Gregory P. Smith, David Beazley, Nick Coghlan, ...) convinced me to keep the `wave`_ module. [4]_ * Gregory P. Smith proposed to deprecate `nntplib`_. [4]_ * Andrew Svetlov mentioned the ``socketserver`` module is questionable. However it's used to implement ``http.server`` and ``xmlrpc.server``. The stdlib doesn't have a replacement for the servers, yet. Update history ============== Update 1 -------- * Deprecate `parser`_ module * Keep `fileinput`_ module * Elaborate why `crypt`_ and `spwd`_ are dangerous and bad * Improve sections for `cgitb`_, `colorsys`_, `nntplib`_, and `smtpd`_ modules * The `colorsys`_, `crypt`_, `imghdr`_, `sndhdr`_, and `spwd`_ sections now list suitable substitutions. * Mention that ``socketserver`` is going to stay for ``http.server`` and ``xmlrpc.server`` * The future maintenance section now states that the deprecated modules may be adopted by Python community members. References ========== .. [1] https://en.wikipedia.org/wiki/Open_Sound_System#Free,_proprietary,_free .. [2] https://man.openbsd.org/ossaudio .. [3] https://blogs.msmvps.com/installsite/blog/2015/05/03/the-future-of-windows-installer-msi-in-the-light-of-windows-10-and-the-universal-windows-platform/ .. [4] https://twitter.com/ChristianHeimes/status/1130257799475335169 .. [5] https://twitter.com/dabeaz/status/1130278844479545351 .. [6] https://mail.python.org/pipermail/python-dev/2019-May/157464.html .. [7] https://discuss.python.org/t/switch-pythons-parsing-tech-to-something-more-powerful-than-ll-1/379 Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/dfmoisset%40gmail.com
On Sat, May 25, 2019 at 8:00 PM Daniel Moisset dfmoisset@gmail.com wrote:
Hi, thanks for the work on this proposal, I think this is at least a tip of the iceberg and a good start for the bigger question of how the stdlib should evolve..
I think that the PEP should include an idea of what should happen if existing stdlib pieces depend on this deprecated modules. For example, email.mime.audio is part of a non-deprecated module but it depends on sndhdr which is being planned for deprecation. Should that part of the functionality be deprecated too? rewritten to not depend on the deprecated module (but keep the pieces of relevant code? depend on optional 3rd party modules and degrade if those are not available?
It's a case-by-case situation. In some instances we may copy the relevant code over and keep it private. In other situations some functionality may get deprecated. It all depends and thus why you can't really write a general solution that applies uniformly.
-Brett
The sndhdr case is just an example but I can imagine there are others (optparse is no longer schedule for removal in your PEP, but would have been another case, being used by the profile module).
Best, Daniel
On Tue, 21 May 2019 at 15:15, Christian Heimes christian@python.org wrote:
Hi,
I have updated the PEP with feedback from discussions. The highlights are:
- Deprecate parser module
- Keep fileinput module
- Elaborate why crypt and spwd are dangerous and bad
- Improve sections for cgitb, colorsys, nntplib, and smtpd modules
- The colorsys, crypt, imghdr, sndhdr, and spwd sections now list
suitable substitutions.
- Mention that socketserver is going to stay for http.server and
xmlrpc.server
- The future maintenance section now states that the deprecated modules
may be adopted by Python community members.
https://github.com/python/peps/compare/7799178a...2d536899?diff=unified#diff...
I'll be traveling the next couple of days and will only have limited opportunities to respond on feedback.
Christian
PEP: 594 Title: Removing dead batteries from the standard library Author: Christian Heimes christian@python.org Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 20-May-2019 Post-History: 21-May-2019
Abstract
This PEP proposed a list of standard library modules to be removed from the standard library. The modules are mostly historic data formats and APIs that have been superseded a long time ago, e.g. Mac OS 9 and Commodore.
Rationale
Back in the early days of Python, the interpreter came with a large set of useful modules. This was often refrained to as "batteries included" philosophy and was one of the corner stones to Python's success story. Users didn't have to figure out how to download and install separate packages in order to write a simple web server or parse email.
Times have changed. The introduction of the cheese shop (PyPI), setuptools, and later pip, it became simple and straight forward to download and install packages. Nowadays Python has a rich and vibrant ecosystem of third party packages. It's pretty much standard to either install packages from PyPI or use one of the many Python or Linux distributions.
On the other hand, Python's standard library is piling up cruft, unnecessary duplication of functionality, and dispensable features. This is undesirable for several reasons.
- Any additional module increases the maintenance cost for the Python core development team. The team has limited resources, reduced maintenance
cost frees development time for other improvements.
- Modules in the standard library are generally favored and seen as the de-facto solution for a problem. A majority of users only pick 3rd party modules to replace a stdlib module, when they have a compelling reason,
e.g. lxml instead of `xml`. The removal of an unmaintained stdlib module increases the chances of a community contributed module to become widely used.
- A lean and mean standard library benefits platforms with limited
resources like devices with just a few hundred kilobyte of storage (e.g. BBC Micro:bit). Python on mobile platforms like BeeWare or WebAssembly (e.g. pyodide) also benefit from reduced download size.
The modules in the PEP have been selected for deprecation because their removal is either least controversial or most beneficial. For example least controversial are 30 years old multimedia formats like ``sunau`` audio format, which was used on SPARC and NeXT workstations in the late 1980ties. The ``crypt`` module has fundamental flaws that are better solved outside the standard library.
This PEP also designates some modules as not scheduled for removal. Some modules have been deprecated for several releases or seem unnecessary at first glance. However it is beneficial to keep the modules in the standard library, mostly for environments where installing a package from PyPI is not an option. This can be cooperate environments or class rooms where external code is not permitted without legal approval.
- The usage of FTP is declining, but some files are still provided over the FTP protocol or hosters offer FTP to upload content. Therefore ``ftplib`` is going to stay.
- The ``optparse`` and ``getopt`` module are widely used. They are mature modules with very low maintenance overhead.
- According to David Beazley [5]_ the ``wave`` module is easy to teach to kids and can make crazy sounds. Making a computer generate crazy sounds
is powerful and highly motivating exercise for a 9yo aspiring developer. It's a fun battery to keep.
Deprecation schedule
3.8
This PEP targets Python 3.8. Version 3.8.0 final is scheduled to be released a few months before Python 2.7 will reach its end of lifetime. We expect that Python 3.8 will be targeted by users that migrate to Python 3 in 2019 and 2020. To reduce churn and to allow a smooth transition from Python 2, Python 3.8 will neither raise `DeprecationWarning` nor remove any modules that have been scheduled for removal. Instead deprecated modules will just be *documented* as deprecated. Optionally modules may emit a `PendingDeprecationWarning`.
All deprecated modules will also undergo a feature freeze. No additional features should be added. Bug should still be fixed.
3.9
Starting with Python 3.9, deprecated modules will start issuing `DeprecationWarning`. The `parser`_ module is removed and potentially replaced with a new module.
All other deprecated modules are fully supported and will receive security updates until Python 3.9 reaches its end of lifetime. Python 3.9.0 will be released about 18 months after 3.8.0 (April 2021?) and most likely be supported for 5 years after the release. The estimated EOL of Python 3.9 is in 2026.
3.10
In 3.10 all deprecated modules will be removed from the CPython repository together with tests, documentation, and autoconf rules.
PEP acceptance process
3.8.0b1 is scheduled to be release shortly after the PEP is officially submitted. Since it's improbable that the PEP will pass all stages of the PEP process in time, I propose a two step acceptance process that is analogous Python's two release deprecation process.
The first *provisionally accepted* phase targets Python 3.8.0b1. In the first phase no code is changes or removed. Modules are only documented as deprecated. The only exception is the `parser`_ module. It has been documented as deprecated since Python 2.5 and is scheduled for removal for 3.9 to make place for a more advanced parser.
The final decision, which modules will be removed and how the removed code is preserved, can be delayed for another year.
Deprecated modules
The modules are grouped as data encoding, multimedia, network, OS interface, and misc modules. The majority of modules are for old data formats or old APIs. Some others are rarely useful and have better replacements on PyPI, e.g. Pillow for image processing or NumPy-based projects to deal with audio processing.
.. csv-table:: Table 1: Proposed modules deprecations :header: "Module", "Deprecated in", "To be removed", "Replacement" :widths: 1, 1, 1, 2
aifc,3.8,3.10,\- asynchat,3.8,3.10,asyncio asyncore,3.8,3.10,asyncio audioop,3.8,3.10,\- binhex,3.8,3.10,\- cgi,3.8,3.10,\- cgitb,3.8,3.10,\- chunk,3.8,3.10,\- colorsys,3.8,3.10,"colormath, colour, colorspacious, Pillow" crypt,3.8,3.10,"bcrypt, argon2cffi, hashlib, passlib" fileinput,\-,**keep**,argparse formatter,3.4,3.10,\- fpectl,**3.7**,**3.7**,\- getopt,**3.2**,**keep**,"argparse, optparse" imghdr,3.8,3.10,"filetype, puremagic, python-magic" imp,**3.4**,3.10,importlib lib2to3,\-,**keep**, macpath,**3.7**,**3.8**,\- msilib,3.8,3.10,\- nntplib,3.8,3.10,\- nis,3.8,3.10,\- optparse,\-,**keep**,argparse ossaudiodev,3.8,3.10,\- parser,**2.5**,**3.9**,"ast, lib2to3.pgen2" pipes,3.8,3.10,subprocess smtpd,"**3.4.7**, **3.5.4**",3.10,aiosmtpd sndhdr,3.8,3.10,"filetype, puremagic, python-magic" spwd,3.8,3.10,"python-pam, simplepam" sunau,3.8,3.10,\- uu,3.8,3.10,\- wave,\-,**keep**, xdrlib,3.8,3.10,\-
Data encoding modules
binhex
The `binhex <https://docs.python.org/3/library/binhex.html>`_ module encodes and decodes Apple Macintosh binhex4 data. It was originally developed for TSR-80. In the 1980s and early 1990s it was used on classic Mac OS 9 to encode binary email attachments. Module type pure Python Deprecated in 3.8 To be removed in 3.10 Substitute **none** uu ~~ The `uu <https://docs.python.org/3/library/uu.html>`_ module provides uuencode format, an old binary encoding format for email from 1980. The uu format has been replaced by MIME. The uu codec is provided by the binascii module. Module type pure Python Deprecated in 3.8 To be removed in 3.10 Substitute **none** xdrlib
The `xdrlib https://docs.python.org/3/library/xdrlib.html`_ module supports the Sun External Data Representation Standard. XDR is an old binary serialization format from 1987. These days it's rarely used outside specialized domains like NFS.
Module type pure Python Deprecated in 3.8 To be removed in 3.10 Substitute **none**
Multimedia modules
aifc
The `aifc <https://docs.python.org/3/library/aifc.html>`_ module provides support for reading and writing AIFF and AIFF-C files. The Audio Interchange File Format is an old audio format from 1988 based on Amiga IFF. It was most commonly used on the Apple Macintosh. These days only few specialized application use AIFF. Module type pure Python (depends on `audioop`_ C extension) Deprecated in 3.8 To be removed in 3.10 Substitute **none** audioop
The `audioop https://docs.python.org/3/library/audioop.html`_ module contains helper functions to manipulate raw audio data and adaptive differential pulse-code modulated audio data. The module is implemented in C without any additional dependencies. The `aifc`_, `sunau`_, and `wave`_ module depend on `audioop`_ for some operations. The byteswap operation in the `wave`_ module can be substituted with little work.
Module type C extension Deprecated in 3.8 To be removed in 3.10 Substitute **none**
colorsys
The `colorsys <https://docs.python.org/3/library/colorsys.html>`_ module defines color conversion functions between RGB, YIQ, HSL, and HSV coordinate systems. The PyPI packages *colormath*, *colour*, and *colorspacious* provide more and advanced features. The Pillow library is better suited to transform images between color systems. Module type pure Python Deprecated in 3.8 To be removed in 3.10 Substitute `colormath <https://pypi.org/project/colormath/>`_, `colour <https://pypi.org/project/colour/>`_ `colorspacious <https://pypi.org/project/colorspacious/>`_, `Pillow <https://pypi.org/project/Pillow/>`_ chunk ~~~~~ The `chunk <https://docs.python.org/3/library/chunk.html>`_ module provides support for reading and writing Electronic Arts' Interchange File Format. IFF is an old audio file format originally introduced for Commodore and Amiga. The format is no longer relevant. Module type pure Python Deprecated in 3.8 To be removed in 3.10 Substitute **none** imghdr ~~~~~~ The `imghdr <https://docs.python.org/3/library/imghdr.html>`_ module is a simple tool to guess the image file format from the first 32 bytes of a file or buffer. It supports only a limited amount of formats and neither returns resolution nor color depth. Module type pure Python Deprecated in 3.8 To be removed in 3.10 Substitute `puremagic <https://pypi.org/project/puremagic/>`_, `filetype <https://pypi.org/project/filetype/>`_, `python-magic <https://pypi.org/project/python-magic/>`_ ossaudiodev
The `ossaudiodev https://docs.python.org/3/library/ossaudiodev.html`_ module provides support for Open Sound System, an interface to sound playback and capture devices. OSS was initially free software, but later support for newer sound devices and improvements were proprietary. Linux community abandoned OSS in favor of ALSA [1]_. Some operation systems like OpenBSD and NetBSD provide an incomplete [2]_ emulation of OSS.
Module type C extension Deprecated in 3.8 To be removed in 3.10 Substitute **none**
sndhdr
The `sndhdr <https://docs.python.org/3/library/sndhdr.html>`_ module is similar to the `imghdr`_ module but for audio formats. It guesses file format, channels, frame rate, and sample widths from the first 512 bytes of a file or buffer. The module only supports AU, AIFF, HCOM, VOC, WAV, and other ancient formats. Module type pure Python (depends on `audioop`_ C extension for some operations) Deprecated in 3.8 To be removed in 3.10 Substitute `puremagic <https://pypi.org/project/puremagic/>`_, `filetype <https://pypi.org/project/filetype/>`_, `python-magic <https://pypi.org/project/python-magic/>`_ sunau ~~~~~ The `sunau <https://docs.python.org/3/library/sunhdr.html>`_ module provides support for Sun AU sound format. It's yet another old, obsolete file format. Module type pure Python (depends on `audioop`_ C extension for some operations) Deprecated in 3.8 To be removed in 3.10 Substitute **none** Networking modules ------------------ asynchat
The `asynchat https://docs.python.org/3/library/asynchat.html`_ module is build on top of `asyncore`_ and has been deprecated since Python 3.6.
Module type pure Python Deprecated in 3.6 Removed in 3.10 Substitute asyncio
asyncore
The `asyncore <https://docs.python.org/3/library/asyncore.html>`_ module was the first module for asynchronous socket service clients and servers. It has been replaced by asyncio and is deprecated since Python 3.6. The ``asyncore`` module is also used in stdlib tests. The tests for ``ftplib``, ``logging``, ``smptd``, ``smtplib``, and ``ssl`` are partly based on ``asyncore``. These tests must be updated to use asyncio or threading. Module type pure Python Deprecated in 3.6 Removed in 3.10 Substitute asyncio cgi ~~~ The `cgi <https://docs.python.org/3/library/cgi.html>`_ module is a support module for Common Gateway Interface (CGI) scripts. CGI is deemed as inefficient because every incoming request is handled in a new process. PEP 206 considers the module as *designed poorly and are now near-impossible to fix*. Several people proposed to either keep the cgi module for features like `cgi.parse_qs()` or move `cgi.escape()` to a different module. The functions `cgi.parse_qs` and `cgi.parse_qsl` have been deprecated for a while and are actually aliases for `urllib.parse.parse_qs` and `urllib.parse.parse_qsl`. The function `cgi.quote` has been deprecated in favor of `html.quote` with secure default values. Module type pure Python Deprecated in 3.8 To be removed in 3.10 Substitute **none** cgitb ~~~~~ The `cgitb <https://docs.python.org/3/library/cgitb.html>`_ module is a helper for the cgi module for configurable tracebacks. The ``cgitb`` module is not used by any major Python web framework (Django, Pyramid, Plone, Flask, CherryPy, or Bottle). Only Paste uses it in an optional debugging middleware. Module type pure Python Deprecated in 3.8 To be removed in 3.10 Substitute **none** smtpd ~~~~~ The `smtpd <https://docs.python.org/3/library/smtpd.html>`_ module provides a simple implementation of a SMTP mail server. The module documentation marks the module as deprecated and recommends ``aiosmtpd`` instead. The deprecation message was added in releases 3.4.7, 3.5.4, and 3.6.1. Module type pure Python Deprecated in **3.7** To be removed in 3.10 Substitute aiosmtpd nntplib ~~~~~~~ The `nntplib <https://docs.python.org/3/library/nntplib.html>`_ module implements the client side of the Network News Transfer Protocol (nntp). News groups used to be a dominant platform for online discussions. Over the last two decades, news has been slowly but steadily replaced with mailing lists and web-based discussion platforms. Twisted is also `planning <https://twistedmatrix.com/trac/ticket/9405>`_ to deprecate NNTP support and `pynnt <https://github.com/greenbender/pynntp>`_ hasn't seen any activity since 2014. This is a good indicator that the public interest in NNTP support is declining. The ``nntplib`` tests have been the cause of additional work in the recent past. Python only contains client side of NNTP. The tests connect to external news server. The servers are sometimes unavailble, too slow, or do not work correctly over IPv6. The situation causes flaky test runs on buildbots. Module type pure Python Deprecated in 3.8 To be removed in 3.10 Substitute **none** Operating system interface -------------------------- crypt ~~~~~ The `crypt <https://docs.python.org/3/library/crypt.html>`_ module implements password hashing based on ``crypt(3)`` function from ``libcrypt`` or ``libxcrypt`` on Unix-like platform. The algorithms are mostly old, of poor quality and insecure. Users are discouraged to use them. * The module is not available on Windows. Cross-platform application need an alternative implementation any way. * Only DES encryption is guarenteed to be available. DES has an extremely limited key space of 2**56. * MD5, salted SHA256, salted SHA512, and Blowfish are optional extension. SSHA256 and SSHA512 are glibc extensions. Blowfish (bcrypt) is the only algorithm that is still secure. However it's in glibc and therefore not commonly available on Linux. * Depending on the platform, the ``crypt`` module is not thread safe. Only implementations with ``crypt_r(3)`` are thread safe. * The module was never useful to interact with system user and password databases. On BSD, macOS, and Linux, all user authentication and password modification operations must go through PAM (pluggable authentication module), see `spwd`_ deprecation. Module type C extension + Python module Deprecated in 3.8 To be removed in 3.10 Substitute `bcrypt <https://pypi.org/project/bcrypt/>`_, `passlib <https://pypi.org/project/passlib/>`_, `argon2cffi <https://pypi.org/project/argon2-cffi/>`_, hashlib module (PBKDF2, scrypt) macpath ~~~~~~~ The `macpath <https://docs.python.org/3/library/macpath.html>`_ module provides Mac OS 9 implementation of os.path routines. Mac OS 9 is no longer supported Module type pure Python Deprecated in 3.7 Removed in 3.8 Substitute **none** nis ~~~ The `nis <https://docs.python.org/3/library/nis.html>`_ module provides NIS/YP support. Network Information Service / Yellow Pages is an old and deprecated directory service protocol developed by Sun Microsystems. It's designed successor NIS+ from 1992 never took off. For a long time, libc's Name Service Switch, LDAP, and Kerberos/GSSAPI are considered a more powerful and more secure replacement of NIS. Module type C extension Deprecated in 3.8 To be removed in 3.10 Substitute **none** spwd ~~~~ The `spwd <https://docs.python.org/3/library/spwd.html>`_ module provides direct access to Unix shadow password database using non-standard APIs. In general it's a bad idea to use the spwd. The spwd circumvents system security policies, it does not use the PAM stack, and is only compatible with local user accounts, because it ignores NSS. The use of the ``spwd`` module for access control must be consider a *security bug*, as it bypasses PAM's access control. Further more the ``spwd`` module uses the `shadow(3) <http://man7.org/linux/man-pages/man3/shadow.3.html>`_ APIs. Functions like ``getspnam(3)`` access the ``/etc/shadow`` file directly. This is dangerous and even forbidden for confined services on systems with a security engine like SELinux or AppArmor. Module type C extension Deprecated in 3.8 To be removed in 3.10 Substitute `python-pam <https://pypi.org/project/python-pam/>`_, `simpleplam <https://pypi.org/project/simplepam/>`_ Misc modules ------------ formatter
The `formatter https://docs.python.org/3/library/formatter.html`_ module is an old text formatting module which has been deprecated since Python 3.4.
Module type pure Python Deprecated in 3.4 To be removed in 3.10 Substitute *n/a*
imp
The `imp <https://docs.python.org/3/library/imp.html>`_ module is the predecessor of the `importlib <https://docs.python.org/3/library/importlib.html>`_ module. Most functions have been deprecated since Python 3.3 and the module since Python 3.4. Module type C extension Deprecated in 3.4 To be removed in 3.10 Substitute importlib msilib
The `msilib https://docs.python.org/3/library/msilib.html`_ package is a Windows-only package. It supports the creation of Microsoft Installers (MSI). The package also exposes additional APIs to create cabinet files (CAB). The module is used to facilitate distutils to create MSI installers with ``bdist_msi`` command. In the past it was used to create CPython's official Windows installer, too.
Microsoft is slowly moving away from MSI in favor of Windows 10 Apps (AppX) as new deployment model [3]_.
Module type C extension + Python code Deprecated in 3.8 To be removed in 3.10 Substitute **none**
parser
The `parser <https://docs.python.org/3/library/parser.html>`_ module provides an interface to Python’s internal parser and byte-code compiler. The stdlib has superior ways to interact with the parse tree. From Python 2.5 onward, it's much more convenient to cut in at the Abstract Syntax Tree (AST) generation and compilation stage. The ``parser`` module causes additional work. It's C code that must be kept in sync with any change to Python's grammar and internal parser. Pablo wants to remove the parser module and promote lib2to3's pgen2 instead [6]_. Most importantly the presence of the ``parser`` module makes it harder to switch to something more powerful than a LL(1) parser [7]_. Since the ``parser`` module is documented as deprecated since Python 2.5 and a new parsing technology is planned for 3.9, the ``parser`` module is scheduled for removal in 3.9. Module type C extension Deprecated in 3.8, documented as deprecated since **2.5** To be removed in **3.9** Substitute ast, lib2to3.pgen2 pipes ~~~~~ The `pipes <https://docs.python.org/3/library/pipes.html>`_ module provides helpers to pipe the input of one command into the output of another command. The module is built on top of ``os.popen``. Users are encouraged to use the subprocess module instead. Module type pure Python Deprecated in 3.8 To be removed in 3.10 Substitute subprocess module Removed modules =============== fpectl ------ The `fpectl <https://docs.python.org/3.6/library/fpectl.html>`_ module was never built by default, its usage was discouraged and considered dangerous. It also required a configure flag that caused an ABI incompatibility. The module was removed in 3.7 by Nathaniel J. Smith in `bpo-29137 <https://bugs.python.org/issue29137>`_. Module type C extension + CAPI Deprecated in 3.7 Removed in 3.7 Substitute **none** Modules to keep =============== Some modules were originally proposed for deprecation. fileinput --------- The `fileinput <https://docs.python.org/3/library/fileinput.html>`_ module implements a helpers to iterate over a list of files from ``sys.argv``. The module predates the optparser and argparser module. The same functionality can be implemented with the argparser module. Several core developers expressed their interest to keep the module in the standard library, as it is handy for quick scripts. Module type pure Python lib2to3 ------- The `lib2to3 <https://docs.python.org/3/library/2to3.html>`_ package provides the ``2to3`` command to transpile Python 2 code to Python 3 code. The package is useful for other tasks besides porting code from Python 2 to 3. For example `black`_ uses it for code reformatting. Module type pure Python getopt ------ The `getopt <https://docs.python.org/3/library/getopt.html>`_ module mimics C's getopt() option parser. Although users are encouraged to use argparse instead, the getopt module is still widely used. The module is small, simple, and handy for C developers to write simple Python scripts. Module type pure Python Substitute argparse optparse -------- The `optparse <https://docs.python.org/3/library/optparse.html>`_ module is the predecessor of the argparse module. Although it has been deprecated for many years, it's still too widely used to remove it. Module type pure Python Deprecated in 3.2 Substitute argparse wave ---- The `wave <https://docs.python.org/3/library/wave.html>`_ module provides support for the WAV sound format. The module is not deprecated, because The WAV format is still relevant these days. The ``wave`` module is also used in education, e.g. to show kids how to make noise with a computer. The module uses one simple function from the `audioop`_ module to perform byte swapping between little and big endian formats. Before 24 bit WAV support was added, byte swap used to be implemented with the ``array`` module. To remove ``wave``'s dependency on the ``audioop``, the byte swap function could be either be moved to another module (e.g. ``operator``) or the ``array`` module could gain support for 24 bit (3 byte) arrays. Module type pure Python (depends on *byteswap* from `audioop`_ C extension) Deprecated in 3.8 To be removed in 3.10 Substitute *n/a* Future maintenance of removed modules ===================================== The main goal of the PEP is to reduce the burden and workload on the Python core developer team. Therefore removed modules will not be maintained by the core team as separate PyPI packages. However the removed code, tests and documentation may be moved into a new git repository, so community members have a place from which they can pick up and fork code. A first draft of a `legacylib <https://github.com/tiran/legacylib>`_ repository is available on my private Github account. The modules could be made available on PyPI. The Python core team will not publish or maintain the packages. It is my hope that members of the Python community will adopt, maintain, and perhaps improve the deprecated modules. It's my hope that some of the deprecated modules will be picked up and adopted by users that actually care about them. For example ``colorsys`` and ``imghdr`` are useful modules, but have limited feature set. A fork of ``imghdr`` can add new features and support for more image formats, without being constrained by Python's release cycle. Most of the modules are in pure Python and can be easily packaged. Some depend on a simple C module, e.g. `audioop`_ and `crypt`_. Since `audioop`_ does not depend on any external libraries, it can be shipped in as binary wheels with some effort. Other C modules can be replaced with ctypes or cffi. For example I created `legacycrypt <https://github.com/tiran/legacycrypt >`_ with ``_crypt`` extension reimplemented with a few lines of ctypes code. Discussions =========== * Elana Hashman and Nick Coghlan suggested to keep the *getopt* module. * Berker Peksag proposed to deprecate and removed *msilib*. * Brett Cannon recommended to delay active deprecation warnings and removal of modules like *imp* until Python 3.10. Version 3.8 will be released shortly before Python 2 reaches end of lifetime. A delay reduced churn for users that migrate from Python 2 to 3.8. * Brett also came up with the idea to keep lib2to3. The package is useful for other purposes, e.g. `black <https://pypi.org/project/black/>`_ uses it to reformat Python code. * At one point, distutils was mentioned in the same sentence as this PEP. To avoid lengthy discussion and delay of the PEP, I decided against dealing with distutils. Deprecation of the distutils package will be handled by another PEP. * Multiple people (Gregory P. Smith, David Beazley, Nick Coghlan, ...) convinced me to keep the `wave`_ module. [4]_ * Gregory P. Smith proposed to deprecate `nntplib`_. [4]_ * Andrew Svetlov mentioned the ``socketserver`` module is questionable. However it's used to implement ``http.server`` and ``xmlrpc.server``. The stdlib doesn't have a replacement for the servers, yet. Update history ============== Update 1 -------- * Deprecate `parser`_ module * Keep `fileinput`_ module * Elaborate why `crypt`_ and `spwd`_ are dangerous and bad * Improve sections for `cgitb`_, `colorsys`_, `nntplib`_, and `smtpd`_ modules * The `colorsys`_, `crypt`_, `imghdr`_, `sndhdr`_, and `spwd`_ sections now list suitable substitutions. * Mention that ``socketserver`` is going to stay for ``http.server`` and ``xmlrpc.server`` * The future maintenance section now states that the deprecated modules may be adopted by Python community members. References ========== .. [1] https://en.wikipedia.org/wiki/Open_Sound_System#Free,_proprietary,_free .. [2] https://man.openbsd.org/ossaudio .. [3] https://blogs.msmvps.com/installsite/blog/2015/05/03/the-future-of-windows-installer-msi-in-the-light-of-windows-10-and-the-universal-windows-platform/ .. [4] https://twitter.com/ChristianHeimes/status/1130257799475335169 .. [5] https://twitter.com/dabeaz/status/1130278844479545351 .. [6] https://mail.python.org/pipermail/python-dev/2019-May/157464.html .. [7] https://discuss.python.org/t/switch-pythons-parsing-tech-to-something-more-powerful-than-ll-1/379 Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/dfmoisset%40gmail.com
Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/brett%40python.org
So what is happening for this PEP since Python 3.8 beta1 has been released? Is it too late for Python 3.8 or not?
It seems like most people are confused by the intent of the PEP. IMHO it would be better to rewrite "Remove packages from the stdlib" as "Move some stdlib modules to PyPI". But that would require to rewrite some parts of the PEP to explain how modules are moved, who become the new maintainers, how to support modules both in stdlib (old Python versions) and in PyPI (new Python), etc.
Victor
On 6/4/2019 8:21 PM, Victor Stinner wrote:
So what is happening for this PEP since Python 3.8 beta1 has been released? Is it too late for Python 3.8 or not?
The only action proposed for 3.8 was soft deprecation in the docs, which I presume can be done later in the beta process.
It seems like most people are confused by the intent of the PEP. IMHO it would be better to rewrite "Remove packages from the stdlib" as "Move some stdlib modules to PyPI". But that would require to rewrite some parts of the PEP to explain how modules are moved, who become the new maintainers, how to support modules both in stdlib (old Python versions) and in PyPI (new Python), etc.
Victor
On Tue, Jun 4, 2019 at 5:22 PM Victor Stinner vstinner@redhat.com wrote:
So what is happening for this PEP since Python 3.8 beta1 has been released? Is it too late for Python 3.8 or not?
It seems like most people are confused by the intent of the PEP. IMHO it would be better to rewrite "Remove packages from the stdlib" as "Move some stdlib modules to PyPI". But that would require to rewrite some parts of the PEP to explain how modules are moved, who become the new maintainers, how to support modules both in stdlib (old Python versions) and in PyPI (new Python), etc.
Correct, that is more than a title change but a shift in what the PEP's final result is. And the title as it currently stands as the modules would still be removed from the stdlib regardless of whether they end up on PyPI or not.
Perhaps if PEP 594 is seen to be moving ahead towards a slimmer Python (4?) stdlib, it might encourage the development of a PEP to take over maintenance of dead parrots. They might be recruited by the offer of some way to at least publish a supported bundle via the same (python.org) site that Python itself comes from. It would seem unfair to burden the PEP with promising an afterlife. That would raise quality assurance issues, though.
If no such PEP emerges then the dead parrots will be buried and forgotten. I see little reason why we (Python users) shouldn't trust the collective python-dev wisdom on what ultimately belongs in stdlib, and heaven knows these aren't simple discussions.
On Wed, Jun 5, 2019 at 6:56 PM Brett Cannon brett@python.org wrote:
On Tue, Jun 4, 2019 at 5:22 PM Victor Stinner vstinner@redhat.com wrote:
So what is happening for this PEP since Python 3.8 beta1 has been released? Is it too late for Python 3.8 or not?
It seems like most people are confused by the intent of the PEP. IMHO it would be better to rewrite "Remove packages from the stdlib" as "Move some stdlib modules to PyPI". But that would require to rewrite some parts of the PEP to explain how modules are moved, who become the new maintainers, how to support modules both in stdlib (old Python versions) and in PyPI (new Python), etc.
Correct, that is more than a title change but a shift in what the PEP's final result is. And the title as it currently stands as the modules would still be removed from the stdlib regardless of whether they end up on PyPI or not. Python-Dev mailing list -- python-dev(a)python.org To unsubscribe send an email to python-dev-leave(a)python.org Unsubscribe: https://mail.python.org/mailman3/lists/python-dev.python.org/
On 05.06.19 02:21, Victor Stinner wrote:
So what is happening for this PEP since Python 3.8 beta1 has been released? Is it too late for Python 3.8 or not?
It seems like most people are confused by the intent of the PEP. IMHO it would be better to rewrite "Remove packages from the stdlib" as "Move some stdlib modules to PyPI". But that would require to rewrite some parts of the PEP to explain how modules are moved, who become the new maintainers, how to support modules both in stdlib (old Python versions) and in PyPI (new Python), etc.
And I would like to add something as well:
The stdlib has been a set of well-known modules. Maybe not the latest and greatest, but you knew for quite sure that these modules are guaranteed to be stable and quite persistent.
With the move to PyPI, I am missing this promise, partially:
PyPI has very many good modules, but also some less good ones. With the stdlib, you had almost one choice to choose from. With PyPI, you have way too many modules, and you have no longer the feeling "this seems to be right in BDFL mind".
I think what is missing is replacement of this feature: The set of modules in the stdlib has exactly that being in the stdlib as a quality indicator. I need now a structure that replaces that quality, like
"This one is eligible to go into stdlib"
Do we have such a replacement implemented, already?
On Thu, Jun 6, 2019 at 12:25 AM Christian Tismer tismer@stackless.com wrote:
On 05.06.19 02:21, Victor Stinner wrote:
So what is happening for this PEP since Python 3.8 beta1 has been released? Is it too late for Python 3.8 or not?
It seems like most people are confused by the intent of the PEP. IMHO it would be better to rewrite "Remove packages from the stdlib" as "Move some stdlib modules to PyPI". But that would require to rewrite some parts of the PEP to explain how modules are moved, who become the new maintainers, how to support modules both in stdlib (old Python versions) and in PyPI (new Python), etc.
And I would like to add something as well:
The stdlib has been a set of well-known modules. Maybe not the latest and greatest, but you knew for quite sure that these modules are guaranteed to be stable and quite persistent.
With the move to PyPI, I am missing this promise, partially:
PyPI has very many good modules, but also some less good ones. With the stdlib, you had almost one choice to choose from. With PyPI, you have way too many modules, and you have no longer the feeling "this seems to be right in BDFL mind".
I think what is missing is replacement of this feature: The set of modules in the stdlib has exactly that being in the stdlib as a quality indicator. I need now a structure that replaces that quality, like
"This one is eligible to go into stdlib"
Do we have such a replacement implemented, already?
Are you asking for us to bless packages on PyPI as of a quality that the core devs approve of it? Or something else? If it's the former we do have links pointing to other projects already (e.g. linking to 'requests' from https://docs.python.org/3/library/urllib.request.html#module-urllib.request).
On 06.06.19 21:27, Brett Cannon wrote:
On Thu, Jun 6, 2019 at 12:25 AM Christian Tismer <tismer@stackless.com mailto:tismer@stackless.com> wrote:
On 05.06.19 02:21, Victor Stinner wrote: > So what is happening for this PEP since Python 3.8 beta1 has been > released? Is it too late for Python 3.8 or not? > > It seems like most people are confused by the intent of the PEP. IMHO > it would be better to rewrite "Remove packages from the stdlib" as > "Move some stdlib modules to PyPI". But that would require to rewrite > some parts of the PEP to explain how modules are moved, who become the > new maintainers, how to support modules both in stdlib (old Python > versions) and in PyPI (new Python), etc. And I would like to add something as well: The stdlib has been a set of well-known modules. Maybe not the latest and greatest, but you knew for quite sure that these modules are guaranteed to be stable and quite persistent. With the move to PyPI, I am missing this promise, partially: PyPI has very many good modules, but also some less good ones. With the stdlib, you had almost one choice to choose from. With PyPI, you have way too many modules, and you have no longer the feeling "this seems to be right in BDFL mind". I think what is missing is replacement of this feature: The set of modules in the stdlib has exactly that being in the stdlib as a quality indicator. I need now a structure that replaces that quality, like "This one is eligible to go into stdlib" Do we have such a replacement implemented, already?
Are you asking for us to bless packages on PyPI as of a quality that the core devs approve of it? Or something else? If it's the former we do have links pointing to other projects already (e.g. linking to 'requests' from https://docs.python.org/3/library/urllib.request.html#module-urllib.request).
Yes, I'm asking for blessing some packages. And I have not spent much time these days with the topic, so please ignore my uninformed question.
On 04Jun2019 1721, Victor Stinner wrote:
So what is happening for this PEP since Python 3.8 beta1 has been released? Is it too late for Python 3.8 or not?
It seems like most people are confused by the intent of the PEP. IMHO it would be better to rewrite "Remove packages from the stdlib" as "Move some stdlib modules to PyPI". But that would require to rewrite some parts of the PEP to explain how modules are moved, who become the new maintainers, how to support modules both in stdlib (old Python versions) and in PyPI (new Python), etc.
I think the problem with this is it sounds like the longer-term plan to maintain some stdlib packages independently of the CPython repo, *but still include them in the distro* (ensurepip style). This is a much broader set of packages that would still be available by default (assuming that distributors care about their users) but would also benefit from a single code base (rather than 3-4 branches) and being independently upgradeable (for backports and/or security fixes). They would probably still be managed by core developers, and it may be easier to attract new contributors for them since they don't need to manage the entire CPython repo.
By contrast, the packages in this PEP are just being deprecated and removed. We don't actually have to explain what happens next, and we don't have to block the PEP on those discussions.
But perhaps we should go for the big refactor now? Then at least we have an explanation for unmaintained packages too - they still show up in distros but are not part of the CPython source tree.
Cheers, Steve