Hi, I created an issue: "Keep deprecated features in Python 3.9 to ease migration from Python 2.7, but remove in Python 3.10" https://bugs.python.org/issue39674 And proposed a first pull request to add again collections.Mapping: https://github.com/python/cpython/pull/18545 Victor Le jeu. 23 janv. 2020 à 16:20, Victor Stinner <vstinner@python.org> a écrit :
Hi,
Python 3.9 introduces many small incompatible changes which broke tons of Python projects, including popular projects, some of them being unmaintained but still widely used (like nose, last release in 2015).
Miro and me consider that Python 3.9 is pushing too much pressure on projects maintainers to either abandon Python 2.7 right now (need to update the CI, the documentation, warn users, etc.), or to introduce a *new* compatibility layer to support Python 3.9: layer which would be dropped as soon as Python 2.7 support will be dropped (soon-ish).
Python 3.9 is too early to accumulate so many incompatible changes on purpose, some incompatible changes like the removal of collections aliases to ABC should be reverted, and reapplied on Python 3.10. Python 3.9 would be the last release which still contain compatibility layers for Python 2.7.
Said differently, we request to maintain the small compatibility layer in Python for one more cycle, instead of requesting every single project maintainer to maintain it on their side. We consider that the general maintenance burden is lower if it's kept in Python for now.
== Fedora COPR notify packages broken by Python 3.9 ==
In Python 3.9, Victor introduced tons of incompatible changes at the beginning of the devcycle. His plan was to push as many as possible, and later decide what to do... This time has come :-) He wrote PEP 606 "Python Compatibility Version" and we wrote PEP 608 "Coordinated Python release", but both have been rejected. At least, it seems like most people agreed that having a CI to get notified of broken projects should help.
We are updating the future Fedora 33 to replace Python 3.8 with Python 3.9. We are using a tool called "COPR" which is like a sandbox and can be seen as the CI discussed previously. It rebuilds Fedora using Python 3.9 as /usr/bin/python3 (and /usr/bin/python !). We now have a good overview of broken packages and which incompatible changes broke most packages.
https://fedoraproject.org/wiki/Changes/Python3.9 - Describes the Fedora change.
https://copr.fedorainfracloud.org/coprs/g/python/python3.9/monitor/ - Has package failures. Some packages fail because of broken dependencies.
https://bugzilla.redhat.com/showdependencytree.cgi?id=PYTHON39 - Has open Python 3.9 bug reports for Fedora packages. Some problems have been fixed upstream already before reaching Fedora, most are only fixed when the Fedora maintainers report the problems back to upstream maintainers.
Right now, there are 150+ packages broken by Python 3.9 incompatible changes.
== Maintenance burden ==
Many Python projects have not yet removed Python 2 support and Python 2 CI. It's not that they would be in the "we will not drop Python 2 support ever" camp, it's just that they have not done it yet. Removing Python 2 compatibility code from the codebase and removing it from the documentation and metadata and CI is a boring task, doesn't bring anything to users, and it might take a new major release of the library. At this point, we are very early in 2020 to expect most projects to have already done this.
At the same time, we would like them to support Python 3.9 as soon in the release cycle as possible. By removing Python 2 compatibility layers from the 3.9 standard library, we are forcing the projects maintainers to re-invent their own compatibility layers and copy-paste stuff like this around. Example:
try: from collections.abc import Sequence except ImprotError: # Python 2.7 doesn't have collections.abc from collections import Sequence
While if we remove collections.Sequence in 3.10, they will face this decision early in 2021 and they will simply fix their code by adding ".abc" at the proper places, not needing any more compatibility layers. Of course, there will be projects that will still have declared Python 2 support in 2021, but it will arguably not be that many.
While it's certainly tempting to have "more pure" code in the standard library, maintaining the compatibility shims for one more release isn't really that big of a maintenance burden, especially when comparing with dozens (hundreds?) of third party libraries essentially maintaining their own.
An good example of a broken package is the nose project which is no longer maintained (according to their website): the last release was in 2015. It remains a very popular test runner. According to libraries.io, it has with 3 million downloads per month, 41.7K dependent repositories and 325 dependent packages. We patched nose in Fedora to fix Python 3.5, 3.6, 3.8 and now 3.9 compatibility issues. People installing nose from PyPI with "pip install" get the unmaintained flavor which is currently completely broken on Python 3.9.
Someone should take over the nose project and maintain it again, or every single project using nose should pick another tool (unittest, nose2, pytest, whatever else). Both options will take a lot of time.
== Request to revert some incompatible changes ==
See https://docs.python.org/dev/whatsnew/3.9.html#removed
Incompatible changes which require "if <python3>: (...) else: (...)" or "try: <python3 code> except (...): <python2 code>":
* Removed tostring/fromstring methods in array.array and base64 modules * Removed collections aliases to ABC classes * Removed fractions.gcd() function (which is similar to math.gcd()) * Remove "U" mode of open(): having to use io.open() just for Python 2 makes the code uglier * Removed old plistlib API: 2.7 doesn't have the new API
== Kept incompatible changes ==
Ok-ish incompatible changes (mostly only affects compatiblity libraries like six):
* _dummy_thread and dummy_threading modules removed: broke six, nine and future projects. six and nine are already fixed for 3.9.
OK incompatible changes (can be replaced by the same code on 2.7 and 3.9):
* isAlive() method of threading.Thread has been removed: Thread.is_alive() method is available in 2.7 and 3.9 * xml.etree.ElementTree.getchildren() and xml.etree.ElementTree.getiterator() methods are removed from 3.9, but list()/iter() works in 2.7 and 3.9
== Call to check for DeprecationWarning in your own projects ==
You must pay attention to DeprecationWarning in Python 3.9: it will be the last "compatibility layer" release, incompatible changes will be reapplied to Python 3.10.
For example, you can use the development mode to see DeprecationWarning and ResourceWarning: use the "-X dev" command line option or set the PYTHONDEVMODE=1 environment variable. Or you can use the PYTHONWARNINGS=default environment variable to see DeprecationWarning.
You might even want to treat all warnings as errors to ensure that you don't miss any when you run your test suite in your CI. You can use PYTHONWARNINGS=error, and combine it with PYTHONDEVMODE=1.
Warnings filters can be used to ignore warnings in third party code, see the documentation: https://docs.python.org/dev/library/warnings.html#the-warnings-filter
-- Victor Stinner and Miro Hrončok for Fedora
-- Night gathers, and now my watch begins. It shall not end until my death.