It's finally time to schedule the last releases in Python 2's life. There will be two more releases of Python 2.7: Python 2.7.17 and Python 2.7.18.
Python 2.7.17 release candidate 1 will happen on October 5th followed by the final release on October 19th.
I'm going to time Python 2.7.18 to coincide with PyCon 2020 in April, so attendees can enjoy some collective catharsis. We'll still say January 1st is the official EOL date.
Thanks to Sumana Harihareswara, there's now a FAQ about the Python 2 sunset on the website: https://www.python.org/doc/sunset-python-2/
Read in the browser: https://www.python.org/dev/peps/pep-0585/ <https://www.python.org/dev/peps/pep-0585/>
Read the source: https://raw.githubusercontent.com/python/peps/master/pep-0585.rst <https://raw.githubusercontent.com/python/peps/master/pep-0585.rst>
The following PEP has been discussed on typing-sig already and a prototype implementation exists for it. I'm extending it now for wider feedback on python-dev, with the intent to present the final version for the Steering Council's consideration by mid-March.
Title: Type Hinting Generics In Standard Collections
Author: Łukasz Langa <lukasz(a)python.org>
Discussions-To: Typing-Sig <typing-sig(a)python.org>
Type: Standards Track
Static typing as defined by PEPs 484, 526, 544, 560, and 563 was built incrementally on top of the existing Python runtime and constrained by existing syntax and runtime behavior. This led to the existence of a duplicated collection hierarchy in the typing module due to generics (for example typing.List and the built-in list).
This PEP proposes to enable support for the generics syntax in all standard collections currently available in the typing module.
Rationale and Goals
This change removes the necessity for a parallel type hierarchy in the typing module, making it easier for users to annotate their programs and easier for teachers to teach Python.
Generic (n.) -- a type that can be parametrized, typically a container. Also known as a parametric type or a generic type. For example: dict.
Parametrized generic -- a specific instance of a generic with the expected types for container elements provided. Also known as a parametrized type. For example: dict[str, int].
Tooling, including type checkers and linters, will have to be adapted to recognize standard collections as generics.
On the source level, the newly described functionality requires Python 3.9. For use cases restricted to type annotations, Python files with the "annotations" future-import (available since Python 3.7) can parametrize standard collections, including builtins. To reiterate, that depends on the external tools understanding that this is valid.
Starting with Python 3.7, when from __future__ import annotations is used, function and variable annotations can parametrize standard collections directly. Example:
from __future__ import annotations
def find(haystack: dict[str, list[int]]) -> int:
Usefulness of this syntax before PEP 585 <https://www.python.org/dev/peps/pep-0585> is limited as external tooling like Mypy does not recognize standard collections as generic. Moreover, certain features of typing like type aliases or casting require putting types outside of annotations, in runtime context. While these are relatively less common than type annotations, it's important to allow using the same type syntax in all contexts. This is why starting with Python 3.9, the following collections become generic using __class_getitem__() to parametrize contained types:
tuple # typing.Tuple
list # typing.List
dict # typing.Dict
set # typing.Set
frozenset # typing.FrozenSet
type # typing.Type
collections.abc.Set # typing.AbstractSet
contextlib.AbstractContextManager # typing.ContextManager
contextlib.AbstractAsyncContextManager # typing.AsyncContextManager
re.Pattern # typing.Pattern, typing.re.Pattern
re.Match # typing.Match, typing.re.Match
Importing those from typing is deprecated. Due to PEP 563 <https://www.python.org/dev/peps/pep-0563> and the intention to minimize the runtime impact of typing, this deprecation will not generate DeprecationWarnings. Instead, type checkers may warn about such deprecated usage when the target version of the checked program is signalled to be Python 3.9 or newer. It's recommended to allow for those warnings to be silenced on a project-wide basis.
The deprecated functionality will be removed from the typing module in the first Python version released 5 years after the release of Python 3.9.0.
Parameters to generics are available at runtime
Preserving the generic type at runtime enables introspection of the type which can be used for API generation or runtime type checking. Such usage is already present in the wild.
Just like with the typing module today, the parametrized generic types listed in the previous section all preserve their type parameters at runtime:
>>> tuple[int, ...]
>>> ChainMap[str, list[str]]
This is implemented using a thin proxy type that forwards all method calls and attribute accesses to the bare origin type with the following exceptions:
the __repr__ shows the parametrized type;
the __origin__ attribute points at the non-parametrized generic class;
the __args__ attribute is a tuple (possibly of length 1) of generic types passed to the original __class_getitem__;
the __parameters__ attribute is a lazily computed tuple (possibly empty) of unique type variables found in __args__;
the __getitem__ raises an exception to disallow mistakes like dict[str][str]. However it allows e.g. dict[str, T][int] and in that case returns dict[str, int].
This design means that it is possible to create instances of parametrized collections, like:
>>> l = list[str]()
>>> list is list[str]
>>> list == list[str]
>>> list[str] == list[str]
>>> list[str] == list[int]
>>> isinstance([1, 2, 3], list[str])
TypeError: isinstance() arg 2 cannot be a parametrized generic
>>> issubclass(list, list[str])
TypeError: issubclass() arg 2 cannot be a parametrized generic
>>> isinstance(list[str], types.GenericAlias)
Objects created with bare types and parametrized types are exactly the same. The generic parameters are not preserved in instances created with parametrized types, in other words generic types erase type parameters during object creation.
One important consequence of this is that the interpreter does not attempt to type check operations on the collection created with a parametrized type. This provides symmetry between:
l: list[str] = 
l = list[str]()
For accessing the proxy type from Python code, it will be exported from the types module as GenericAlias.
Pickling or (shallow- or deep-) copying a GenericAlias instance will preserve the type, origin, attributes and parameters.
Future standard collections must implement the same behavior.
A proof-of-concept or prototype implementation <https://bugs.python.org/issue39481> exists.
Keeping the status quo forces Python programmers to perform book-keeping of imports from the typing module for standard collections, making all but the simplest annotations cumbersome to maintain. The existence of parallel types is confusing to newcomers (why is there both list and List?).
The above problems also don't exist in user-built generic classes which share runtime functionality and the ability to use them as generic type annotations. Making standard collections harder to use in type hinting from user classes hindered typing adoption and usability.
It would be easier to implement __class_getitem__ on the listed standard collections in a way that doesn't preserve the generic type, in other words:
>>> tuple[int, ...]
>>> collections.ChainMap[str, list[str]]
This is problematic as it breaks backwards compatibility: current equivalents of those types in the typing module do preserve the generic type:
>>> from typing import List, Tuple, ChainMap
>>> Tuple[int, ...]
>>> ChainMap[str, List[str]]
As mentioned in the "Implementation" section, preserving the generic type at runtime enables runtime introspection of the type which can be used for API generation or runtime type checking. Such usage is already present in the wild.
Additionally, implementing subscripts as identity functions would make Python less friendly to beginners. Say, if a user is mistakenly passing a list type instead of a list object to a function, and that function is indexing the received object, the code would no longer raise an error.
>>> l = list
TypeError: 'type' object is not subscriptable
With __class_getitem__ as an identity function:
>>> l = list
The indexing being successful here would likely end up raising an exception at a distance, confusing the user.
Disallowing instantiation of parametrized types
Given that the proxy type which preserves __origin__ and __args__ is mostly useful for runtime introspection purposes, we might have disallowed instantiation of parametrized types.
In fact, forbidding instantiation of parametrized types is what the typing module does today for types which parallel builtin collections (instantiation of other parametrized types is allowed).
The original reason for this decision was to discourage spurious parametrization which made object creation up to two orders of magnitude slower compared to the special syntax available for those builtin collections.
This rationale is not strong enough to allow the exceptional treatment of builtins. All other parametrized types can be instantiated, including parallels of collections in the standard library. Moreover, Python allows for instantiation of lists using list() and some builtin collections don't provide special syntax for instantiation.
Making isinstance(obj, list[str]) perform a check ignoring generics
An earlier version of this PEP suggested treating parametrized generics like list[str] as equivalent to their non-parametrized variants like list for purposes of isinstance() and issubclass(). This would be symmetrical to how list[str]() creates a regular list.
This design was rejected because isinstance() and issubclass() checks with parametrized generics would read like element-by-element runtime type checks. The result of those checks would be surprising, for example:
>>> isinstance([1, 2, 3], list[str])
Note the object doesn't match the provided generic type but isinstance() still returns True because it only checks whether the object is a list.
If a library is faced with a parametrized generic and would like to perform an isinstance() check using the base type, that type can be retrieved using the __origin__ attribute on the parametrized generic.
Making isinstance(obj, list[str]) perform a runtime type check
This functionality requires iterating over the collection which is a destructive operation in some of them. This functionality would have been useful, however implementing the type checker within Python that would deal with complex types, nested type checking, type variables, string forward references, and so on is out of scope for this PEP.
Note on the initial draft
An early version of this PEP discussed matters beyond generics in standard collections. Those unrelated topics were removed for clarity.
This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive.
I think that PEP 573 is ready to be accepted, to greatly improve the state
of extension modules in CPython 3.9.
It has come a long way since the original proposal and went through several
iterations and discussions by various interested people, effectively
reducing its scope quite a bit. So this is the last call for comments on
the latest version of the PEP, before I will pronounce on it. Please keep
the discussion in this thread.
An unfortunate side-effect of making PyInterpreterState in Python 3.8 opaque is it removed [PEP 523](https://www.python.org/dev/peps/pep-0523/) support. https://www.python.org/dev/peps/pep-0523/ was opened to try and fix this, but there seems to be a stalemate in the issue.
A key question is at what API level should setting the frame evaluation function live at? No one is suggesting the stable ABI, but there isn't agreement between CPython or the internal API (there's also seems to be a suggestion to potentially remove PEP 523 support entirely).
And regardless of either, there also seems to be a disagreement about providing getters and setters to continue to try and hide PyInterpreterState regardless of which API level support is provided at (if any).
If you have an opinion please weight in on the issue.
Python 3.9 introduces many small incompatible changes which broke tons
of Python projects, including popular projects, some of them being
unmaintained but still widely used (like nose, last release in 2015).
Miro and me consider that Python 3.9 is pushing too much pressure on
projects maintainers to either abandon Python 2.7 right now (need to
update the CI, the documentation, warn users, etc.), or to introduce a
*new* compatibility layer to support Python 3.9: layer which would be
dropped as soon as Python 2.7 support will be dropped (soon-ish).
Python 3.9 is too early to accumulate so many incompatible changes on
purpose, some incompatible changes like the removal of collections
aliases to ABC should be reverted, and reapplied on Python 3.10.
Python 3.9 would be the last release which still contain compatibility
layers for Python 2.7.
Said differently, we request to maintain the small compatibility layer
in Python for one more cycle, instead of requesting every single
project maintainer to maintain it on their side. We consider that the
general maintenance burden is lower if it's kept in Python for now.
== Fedora COPR notify packages broken by Python 3.9 ==
In Python 3.9, Victor introduced tons of incompatible changes at the
beginning of the devcycle. His plan was to push as many as possible,
and later decide what to do... This time has come :-) He wrote PEP 606
"Python Compatibility Version" and we wrote PEP 608 "Coordinated
Python release", but both have been rejected. At least, it seems like
most people agreed that having a CI to get notified of broken projects
We are updating the future Fedora 33 to replace Python 3.8 with Python
3.9. We are using a tool called "COPR" which is like a sandbox and can
be seen as the CI discussed previously. It rebuilds Fedora using
Python 3.9 as /usr/bin/python3 (and /usr/bin/python !). We now have a
good overview of broken packages and which incompatible changes broke
- Describes the Fedora change.
- Has package failures. Some packages fail because of broken dependencies.
- Has open Python 3.9 bug reports for Fedora packages. Some problems
have been fixed upstream already before reaching Fedora, most are only
fixed when the Fedora maintainers report the problems back to upstream
Right now, there are 150+ packages broken by Python 3.9 incompatible changes.
== Maintenance burden ==
Many Python projects have not yet removed Python 2 support and Python
2 CI. It's not that they would be in the "we will not drop Python 2
support ever" camp, it's just that they have not done it yet. Removing
Python 2 compatibility code from the codebase and removing it from the
documentation and metadata and CI is a boring task, doesn't bring
anything to users, and it might take a new major release of the
library. At this point, we are very early in 2020 to expect most
projects to have already done this.
At the same time, we would like them to support Python 3.9 as soon in
the release cycle as possible. By removing Python 2 compatibility
layers from the 3.9 standard library, we are forcing the projects
maintainers to re-invent their own compatibility layers and copy-paste
stuff like this around. Example:
from collections.abc import Sequence
# Python 2.7 doesn't have collections.abc
from collections import Sequence
While if we remove collections.Sequence in 3.10, they will face this
decision early in 2021 and they will simply fix their code by adding
".abc" at the proper places, not needing any more compatibility
layers. Of course, there will be projects that will still have
declared Python 2 support in 2021, but it will arguably not be that
While it's certainly tempting to have "more pure" code in the standard
library, maintaining the compatibility shims for one more release
isn't really that big of a maintenance burden, especially when
comparing with dozens (hundreds?) of third party libraries essentially
maintaining their own.
An good example of a broken package is the nose project which is no
longer maintained (according to their website): the last release was
in 2015. It remains a very popular test runner. According to
libraries.io, it has with 3 million downloads per month, 41.7K
dependent repositories and 325 dependent packages. We patched nose in
Fedora to fix Python 3.5, 3.6, 3.8 and now 3.9 compatibility issues.
People installing nose from PyPI with "pip install" get the
unmaintained flavor which is currently completely broken on Python
Someone should take over the nose project and maintain it again, or
every single project using nose should pick another tool (unittest,
nose2, pytest, whatever else). Both options will take a lot of time.
== Request to revert some incompatible changes ==
Incompatible changes which require "if <python3>: (...) else: (...)"
or "try: <python3 code> except (...): <python2 code>":
* Removed tostring/fromstring methods in array.array and base64 modules
* Removed collections aliases to ABC classes
* Removed fractions.gcd() function (which is similar to math.gcd())
* Remove "U" mode of open(): having to use io.open() just for Python 2
makes the code uglier
* Removed old plistlib API: 2.7 doesn't have the new API
== Kept incompatible changes ==
Ok-ish incompatible changes (mostly only affects compatiblity
libraries like six):
* _dummy_thread and dummy_threading modules removed: broke six, nine
and future projects. six and nine are already fixed for 3.9.
OK incompatible changes (can be replaced by the same code on 2.7 and 3.9):
* isAlive() method of threading.Thread has been removed:
Thread.is_alive() method is available in 2.7 and 3.9
* xml.etree.ElementTree.getchildren() and
xml.etree.ElementTree.getiterator() methods are removed from 3.9, but
list()/iter() works in 2.7 and 3.9
== Call to check for DeprecationWarning in your own projects ==
You must pay attention to DeprecationWarning in Python 3.9: it will be
the last "compatibility layer" release, incompatible changes will be
reapplied to Python 3.10.
For example, you can use the development mode to see
DeprecationWarning and ResourceWarning: use the "-X dev" command line
option or set the PYTHONDEVMODE=1 environment variable. Or you can use
the PYTHONWARNINGS=default environment variable to see
You might even want to treat all warnings as errors to ensure that you
don't miss any when you run your test suite in your CI. You can use
PYTHONWARNINGS=error, and combine it with PYTHONDEVMODE=1.
Warnings filters can be used to ignore warnings in third party code,
see the documentation:
-- Victor Stinner and Miro Hrončok for Fedora
I have successfully built Python 3.8.1 on QNX, but ran into a problem when using 'make -j4'. The build process invariably hangs with multiple single-threaded child processes stuck indefinitely waiting on semaphores. These semaphores will clearly never be posted to, as the processes are all single threaded and the semaphores are not shared with any other process.
A backtrace shows that the the offending calls come from run_at_forkers(), which is not surprising. I consider a multi-threaded fork() to be an ill-defined operation and my arch-nemesis... The problem here is that the value of the semaphore inherited from the parent shows the semaphore to be unavailable, even though the semaphore *object* itself is not the same as the one used by the parent (they share the same virtual address, but in different address spaces with different backing memory).
A few (noob) questions:
1. Is there a way to correlate the C backtrace from the core dump to the Python code that resulted in this hang?
2. It is well-known that a multi-threaded fork() is inherently unsafe, and POSIX says that no non-async-signal-safe operations are allowed between the fork() and exec() calls. I even saw comments to this effect in the Python source code ;-) So why is it done?
3. Any reason not to use posix_spawn() instead of fork()/exec()? While some systems implement posix_spawn() with fork()/exec() others (at least QNX) implements it without first creating a duplicate of the parent, making it both more efficient and safer in a multi-threaded parent.
4. thread_pthread.h seems to go to great lengths to implement locks without using native mutexes. I found one reference in the code dating to 1994 as to why that is done. Is it still applicable? Contrary to the claim in that comment the semantics for trying to lock an already-locked mutex and for unlocking an unowned mutex are well-defined.
In a recently opened typo fixing PR , an issue came up regarding the
lack of a signed CLA, where the author specifically mentioned they did not
want to sign it for privacy concerns.
In the past, I've seen several PRs with similarly minimal  changes (such
as typo fixes, grammar fixes, link fixes, etc) merged without having the
CLA signed, so it was my assumption that this was acceptable. For a full
list of merged PRs to CPython with a "CLA not signed" label, see the
However, I was informed by Pablo Galindo that there are legal issues
involved with merging *any* PRs without the CLA signed, including typo
fixes. Personally, I have no strong opinion one way or the other, as I
don't have an adequate understanding from a legal/licensing perspective.
But, I think think there's definitely an issue with the lack of consistency
regarding this policy.
*To require a signed CLA for some minimal PRs but not others, solely based
on who happens to be reviewing the PR, seems rather unfair to potential
contributors.* From my perspective, the solution seems to be clearly
defining a more explicit stance on this policy, and having it apply as
universally as possible to *all* PRs made to CPython.
For example, if the CLA should be required for all PRs, the policy might
state something like this: "The author of any PR made to the CPython
repository must have signed the CLA before their PR can be merged. Any PR
opened by an author that has not signed the CLA can't be merged until it
has been signed."
OTOH, if it's okay for minimal PRs to not have a signed CLA: "The author of
any PR made to the CPython repository must have signed the CLA before their
PR can be merged, except for minimal PRs. Some examples of minimal PRs
Currently, the policy seems to be learning more towards the former based on
the devguide , where it states "To accept your change we must have your
formal approval for distributing your work under the PSF license.
Therefore, you need to sign a contributor agreement which allows the Python
Software Foundation to license your code for use with Python (you retain
However, it seems apparent to me that either this policy isn't explicit
enough, has a lack of visibility, or simply isn't followed consistently.
What might be a viable solution to this problem?
 - https://github.com/python/cpython/pull/18603
 - The term "minimal" can be interchanged with "trivial" for the most
part in the above context, but I tend to prefer the former. IMO, it comes
across as more respectful to the efforts made by the author, as even the
smallest of PRs can require substantial efforts from first-time
contributors that are entirely unfamiliar with the workflow; regardless of
how small the change is.
 - https://devguide.python.org/pullrequest/#licensing
Now that the last bits of discussion about PEP 584 have been settled (we
will *not* be calling the `copy` or `update` methods) I am accepting this
PEP. Congratulations Steven and Brandt!
Thanks to everyone else who gave their opinion.
Formally, I will just send my recommendation to accept the PEP to the
Steering Council -- they will then let us know whether they agree, and once
that's done we can update the PEP with the "Accepted" status and land the
implementation (https://github.com/python/cpython/pull/12088). (Hm, the PEP
should probably also link to that PR rather than to Brandt's private
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*