The proposal looks awesome to me. Victor :) It could be out of scope for this proposal. How about adding APIs about compatibility implementation in this proposal? e.g compatibility_python_implementation usage will be sys.compatibility_python_implementation(('Cpython', 'PyPy', 'XXXPython')) We can check the implementation name since platform.python_implementation() is already existed. The reason for this feature is that when the python code is running on alternative python interpreter project, user can not sure whether the alternative interpreter could run as the code author intended. This feature would help them(author, user). 2019년 10월 18일 (금) 오전 10:19, Victor Stinner 님이 작성:

Hi,

I propose the following PEP to add sys.set_python_compat_version(version) to introduce a partial compatibility with Python 3.8 in the next Python 3.9 version.

I also propose this PEP in a pull request: https://github.com/python/peps/pull/1209 (Please avoid using the temporary PEP number until this PR is merged: officially, the PEP has no number yet.)

If possible, please try to read the whole PEP before replying. I would prefer to avoid knee-jerk reactions :-) The backward compatibility is complex topic where things are not black or white: it's more a grayscale.

IMHO with the incoming end of Python 2 support, it's the right time to propose this PEP!

Victor

PEP: xxx Title: Python Compatibility Version Author: Victor Stinner Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 18-Oct-2019 Python-Version: 3.9

Abstract ========

Add ``sys.set_python_compat_version(version)`` to enable partial compatibility with requested Python version. Add ``sys.get_python_compat_version()``.

Modify a few functions of the standard library to implement a partial compatibility with Python 3.8.

Add ``sys.set_python_min_compat_version(version)`` to deny backward compatibility with Python older than *version*.

Add ``-X compat_version=VERSION`` and ``-X min_compat_version=VERSION`` command line options. Add ``PYTHONCOMPATVERSION`` and ``PYTHONCOMPATMINVERSION`` environment variables.

Rationale =========

The need to evolve frequently -----------------------------

To remain relevant and useful, Python has to evolve frequently. Some enhancements require incompatible changes. Any incompatible change can break an unknown number of Python projects. Developers can decide to not implement a feature because of that.

Users want to get the latest Python version to get new features and better performance. A few incompatible changes prevent them to use their applications on the latest Python version.

This PEP proposes to add a partial compatibility with old Python versions as a tradeoff to fit both use cases.

The main issue with the migration from Python 2 to Python 3 is not that Python 3 is backward incompatible, but how incompatible changes were introduced.

Partial compatibility to minimize the Python maintenance burden ---------------------------------------------------------------

While technically it would be possible to provide a full compatibility with old Python versions, this PEP proposes to minimize the number of functions handling backward compatibility to reduce the maintenance burden of the Python project ("CPython").

Each change introducing backport compatibility to a function should be properly discussed to estimate the maintenance cost in the long-term.

Backward compatibility code will be dropped at each Python release, on a case by case basis. Each compatibility function can be supported for a different number of Python releases depending on its maintenance cost and the estimated risk (number of broken projects) if it's removed.

The maintenance cost does not only come from the code implementing the backward compatibility, but come also from additional tests.

Cases excluded from backward compatibility ------------------------------------------

The performance overhead of a compatibility code must be low when ``sys.set_python_compat_version()`` is not called.

The C API is out of the scope of this PEP: ``Py_LIMITED_API`` macro and the stable ABI are solving this problem differently, see the `PEP 384: Defining a Stable ABI https://www.python.org/dev/peps/pep-0384/`_.

Security fixes which break the backward compatibility on purpose will not get a compatibility layer. Security matters more than compatibility. For example, ``http.client.HTTPSConnection`` was modified in Python 3.4.3 to performs all the necessary certificate and hostname checks by default. It was a deliberate change motivated by the `PEP 476: Enabling certificate verification by default for stdlib http clients https://www.python.org/dev/peps/pep-0476/`_ (`bpo-22417 https://bugs.python.org/issue22417`_).

The Python language does not provide backward compatibility.

Changes which are not clearly incompatible are not covered by this PEP. For example, Python 3.9 changed the default protocol in the ``pickle`` module to Protocol 4 which was first introduced in Python 3.4. This change is backward compatible up to Python 3.4. There is no need to use the Protocol 3 by default when compatibility with Python 3.8 is requested.

New ``DeprecationWarning`` and ``PendingDeprecatingWarning`` warnings of Python 3.9 will not be disabled in Python 3.8 compatibility mode. If a project runs its test suite using ``-Werror`` (treat any warning as an error), these warnings must be fixed, or specific deprecation warnings must be ignored on a case by case basis.

Upgrade a project to a newer Python -----------------------------------

Without backward compatibility, all incompatible changes must be fixed at once, which can be a blocker issue. It is even worse when a project is upgraded to a newer Python which is separated by multiple releases from the old Python.

Postponing an upgrade only makes things worse: each skipped release adds more incompatible changes. The technical debt is only steadily increasing.

With backward compatibility, it becomes possible to upgrade Python increamentally in a project, without having to fix all issues at once.

The "all-or-nothing" is a showstopper to port large Python 2 code bases to Python 3. The list of incompatible changes between Python 2 and Python 3 is long, and it's getting longer at each Python 3.x release.

Cleaning up Python and DeprecationWarning -----------------------------------------

One of the `Zen of Python (PEP 20) https://www.python.org/dev/peps/pep-0020/`_ motto is:

There should be one-- and preferably only one --obvious way to do it.

When Python evolves, new ways emerge inevitably. ``DeprecationWarning`` are emitted to suggest to use the new way, but many developers ignore these warnings, which are silent by default (except in the ``__main__`` module: see the `PEP 565 https://www.python.org/dev/peps/pep-0565/_`). Some developers simply ignore all warnings since they are too many warnings, and so only bother with exceptions when deprecated code is removed.

Sometimes, supporting both ways has a minor maintenance cost, but developers prefer to drop the old way to clean up the code. Such kind of change is backward incompatible.

Some developers can take the end of the Python 2 support as an opportunity to push even more incompatible changes than usual.

Adding backward compatibility as an opt-in prevents to break applications and allows developers to continue to do such cleanup.

Redistribute the maintenance burden -----------------------------------

The backward compatibility involves authors of backward incompatible changes more in the upgrade path.

Examples of backward compatibility ==================================

collections ABC aliases -----------------------

``collections.abc`` aliases to ABC classes have been removed from the ``collections`` module in Python 3.9, after being deprecated since Python 3.3. For example, ``collections.Mapping`` no longer exists.

In Python 3.6, aliases were created in ``collections/__init__.py`` by ``from _collections_abc import *``.

In Python 3.7, a ``__getattr__()`` has been added to the ``collections`` module to emit a DeprecationWarning at the first access to an attribute::

def __getattr__(name): # For backwards compatibility, continue to make the collections ABCs # through Python 3.6 available through the collections module. # Note, no new collections ABCs were added in Python 3.7 if name in _collections_abc.__all__: obj = getattr(_collections_abc, name) import warnings warnings.warn("Using or importing the ABCs from 'collections' instead " "of from 'collections.abc' is deprecated since Python 3.3, " "and in 3.9 it will stop working", DeprecationWarning, stacklevel=2) globals()[name] = obj return obj raise AttributeError(f'module {__name__!r} has no attribute {name!r}')

Compatibility with Python 3.8 can be restored in Python 3.9 by adding back the ``__getattr__()`` function, but only when backward compatibility is requested::

def __getattr__(name): if (sys.get_python_compat_version() < (3, 9) and name in _collections_abc.__all__): ... raise AttributeError(f'module {__name__!r} has no attribute {name!r}')

Deprecated open() "U" mode --------------------------

The "U" mode of ``open()`` is deprecated since Python 3.4 and emits a ``DeprecationWarning``. The `bpo-37330 https://bugs.python.org/issue37330`_ proposes to drop this mode: ``open()`` would raise an exception if ``U`` mode is used.

This change falls into the "cleanup" category: it is not required to implement a feature.

A backward compatibility mode would be trivial to implement and would be welcomed here by users.

Specification =============

sys functions -------------

Add 3 functions to the ``sys`` module:

* ``sys.set_python_compat_version(version)``: set the Python compatibility version. If it has been called previously, use the minimum of requested versions. Raise an exception if ``sys.set_python_min_compat_version(min_version)`` has been called and ``version < min_version``. *version* must be greater than or equal to ``(3, 0)``.

* ``sys.set_python_min_compat_version(min_version)``: set the **minimum** compatibility version. Raise an exception if ``sys.set_python_compat_version(old_version)`` has been called previously and ``old_version < min_version``. *min_version* must be greater than or equal to ``(3, 0)``.

* ``sys.get_python_compat_version()``: get the Python compatibility version. Return a ``tuple`` of 3 integers.

A *version* must a tuple of 2 or 3 integers. ``(major, minor)`` version is equivalent to ``(major, minor, 0)``.

By default, ``sys.get_python_compat_version()`` returns the current Python version.

Example to request compatibility with Python 3.8.0::

import collections

sys.set_python_compat_version((3, 8))

# collections.Mapping alias, removed from Python 3.9, is available # again, even if collections has been imported before calling # set_python_compat_version(). parent = collections.Mapping

Obviously, calling ``sys.set_python_compat_version(version)`` has no effect on code executed before the call. Use ``-X compat_version=VERSION`` command line option or ``PYTHONCOMPATVERSIONVERSION=VERSION`` environment variable to set the compatibility version at Python startup.

Command line ------------

Add ``-X compat_version=VERSION`` and ``-X min_compat_version=VERSION`` command line options: call respectivelly ``sys.set_python_compat_version()`` and ``sys.set_python_min_compat_version()``. ``VERSION`` is a version string with 2 or 3 numbers (``major.minor.micro`` or ``major.minor``). For example, ``-X compat_version=3.8`` calls ``sys.set_python_compat_version((3, 8))``.

Add ``PYTHONCOMPATVERSIONVERSION=VERSION`` and ``PYTHONCOMPATMINVERSION=VERSION=VERSION`` environment variables: call respectivelly ``sys.set_python_compat_version()`` and ``sys.set_python_min_compat_version()``. ``VERSION`` is a version string with the same format that the command line options.

Backwards Compatibility =======================

Introducing ``sys.set_python_compat_version()`` function means that an application will behave differently depending on the compatibility version. Moreover, since the version can be decreased multiple times, the application can behave differently depending on the import order.

Python 3.9 with ``sys.set_python_compat_version((3, 8))`` is not fully compatible with Python 3.8: the compatibility is only partial.

Security Implications =====================

``sys.set_python_compat_version()`` must not disable security fixes.

Alternatives ============

Provide a workaround for each incompatible change -------------------------------------------------

An application can works around most of the incompatible changes which impacts it.

For example, ``collections`` aliases can be added again using::

import collections.abc collections.Mapping = collections.abc.Mapping collections.Sequence = collections.abc.Sequence

Handle backward compatibility in the parser -------------------------------------------

The parser is modified to support multiple versions of the Python language (grammar).

The current Python parser cannot be easily modified for that. AST and grammar are hardcoded to a single Python version.

In Python 3.8, ``compile()`` has an undocumented ``_feature_version`` to not consider ``async`` and ``await`` as keywords.

The latest major language backward incompatible change was Python 3.7 which made ``async`` and ``await`` real keywords. It seems like Twisted was the only affected project, and Twisted had a single affected function (it used a parameter called ``async``).

Handling backward compatibility in the parser seems quite complex, not only to modify the parser, but also for developers who have to check which version of the Python language is used.

from __future__ import python38_syntax ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Add ``pythonXY_syntax`` to the ``__future__`` module. It would enable backward compatibility with Python X.Y syntax, but only for the current file.

With this option, there is no need to change ``sys.implementation.cache_tag`` to use a different ``.pyc`` filename, since the parser would always produce the same output for the same input (except of the optimization level).

Example::

from __future__ import python35_syntax

async = 1 await = 2

Update cache_tag ^^^^^^^^^^^^^^^^

Modify the parser to use ``sys.get_python_compat_version()`` to choose the version of the Python language.

``sys.set_python_compat_version()`` updates ``sys.implementation.cache_tag`` to include the compatibility version without the micro version as a suffix. For example, Python 3.9 uses ``'cpython-39'`` by default, but ``sys.set_python_compat_version((3, 7, 2))`` sets ``cache_tag`` to ``'cpython-39-37'``. Changes of the Python language are now allowed in micro releases.

One problem is that ``import asyncio`` is likely to fail if ``sys.set_python_compat_version((3, 6))`` has been called previously. The code of the ``asyncio`` module requires ``async`` and ``await`` to be real keywords (change done in Python 3.7).

Another problem is that regular users cannot write ``.pyc`` files into system directories, and so cannot create them on demand. It means that ``.pyc`` optimization cannot be used in the backward compatibility mode.

One solution for that is to modify the Python installer and Python package installers to precompile ``.pyc`` files not only for the current Python version, but also for multiple older Python versions (up to Python 3.0?).

Each ``.py`` file would have 3n ``.pyc`` files (3 optimization levels), where ``n`` is the number of supported Python versions. For example, it means 6 ``.pyc`` files, instead of 3, to support Python 3.8 and Python 3.9.

Temporary moratorium on incompatible changes --------------------------------------------

In 2009, the PEP 3003 "Python Language Moratorium" proposed to a temporary moratorium (suspension) of all changes to the Python language syntax, semantics, and built-ins for Python 3.1 and Python 3.2.

In May 2018, during PEP 572 discussions, it was also proposed to slow down Python changes: see the python-dev thread `Slow down... < https://mail.python.org/archives/list/python-dev@python.org/thread/HHKRXOMRJ...

...

`_

`Barry Warsaw's call on this < https://mail.python.org/archives/list/python-dev@python.org/message/XR7IF2OB...

...

`_:

I don’t believe that the way for Python to remain relevant and useful for the next 10 years is to cease all language evolution. Who knows what the computing landscape will look like in 5 years, let alone 10? Something as arbitrary as a 10 year moratorium is (again, IMHO) a death sentence for the language.

PEP 387 -------

`PEP 387 -- Backwards Compatibility Policy https://www.python.org/dev/peps/pep-0387/`_ proposes a process to make incompatible changes. The main point is the 4th step of the process:

See if there's any feedback. Users not involved in the original discussions may comment now after seeing the warning. Perhaps reconsider.

PEP 497 -------

`PEP 497 -- A standard mechanism for backward compatibility https://www.python.org/dev/peps/pep-0497/`_ proposes different solutions to provide backward compatibility.

Except of the ``__past__`` mechanism idea, the PEP 497 does not propose concrete solutions:

When an incompatible change to core language syntax or semantics is being made, Python-dev's policy is to prefer and expect that, wherever possible, a mechanism for backward compatibility be considered and provided for future Python versions after the breaking change is adopted by default, in addition to any mechanisms proposed for forward compatibility such as new future_statements.

Examples of incompatible changes ================================

Python 3.8 ----------

Examples of Python 3.8 incompatible changes:

* (During beta phase) ``PyCode_New()`` required a new parameter: it broke all Cython extensions (all projects distributing precompiled Cython code). This change has been reverted during the 3.8 beta phase and a new ``PyCode_NewWithPosOnlyArgs()`` function was added instead.

* ``types.CodeType`` requires an additional mandatory parameter. The ``CodeType.replace()`` function was added to help projects to no longer depend on the exact signature of the ``CodeType`` constructor.

* C extensions are no longer linked to libpython.

* ``sys.abiflags`` changed from ``'m'`` to an empty string. For example, ``python3.8m`` program is gone.

* The C structure ``PyInterpreterState`` was made opaque.

* Blender:

* https://bugzilla.redhat.com/show_bug.cgi?id=1734980#c6 * https://developer.blender.org/D6038

* XML attribute order: `bpo-34160 https://bugs.python.org/issue34160`_. Broken projects:

* `coverage https://bugs.python.org/issue34160#msg329612`_ * `docutils https://sourceforge.net/p/docutils/bugs/359/`_ * `pcs https://bugzilla.redhat.com/show_bug.cgi?id=1705475`_ * `python-glyphsLib https://bugzilla.redhat.com/show_bug.cgi?id=1705391`_

Backward compatibility cannot be added for all these changes. For example, changes in the C API and in the build system are out of the scope of this PEP.

See `What’s New In Python 3.8: API and Feature Removals https://docs.python.org/dev/whatsnew/3.8.html#api-and-feature-removals`_ for all changes.

See also the `Porting to Python 3.8 https://docs.python.org/dev/whatsnew/3.8.html#porting-to-python-3-8`_ section of What’s New In Python 3.8.

Python 3.7 ----------

Examples of Python 3.7 incompatible changes:

* ``async`` and ``await`` are now reserved keywords. * Several undocumented internal imports were removed. One example is that ``os.errno`` is no longer available; use ``import errno`` directly instead. Note that such undocumented internal imports may be removed any time without notice, even in micro version releases. * Unknown escapes consisting of ``'\'`` and an ASCII letter in replacement templates for ``re.sub()`` were deprecated in Python 3.5, and will now cause an error. * The ``asyncio.windows_utils.socketpair()`` function has been removed: it was an alias to ``socket.socketpair()``. * ``asyncio`` no longer exports the ``selectors`` and ``_overlapped`` modules as ``asyncio.selectors`` and ``asyncio._overlapped``. Replace ``from asyncio import selectors`` with ``import selectors``. * PEP 479 is enabled for all code in Python 3.7, meaning that ``StopIteration`` exceptions raised directly or indirectly in coroutines and generators are transformed into ``RuntimeError`` exceptions. * ``socketserver.ThreadingMixIn.server_close()`` now waits until all non-daemon threads complete. Set the new ``block_on_close`` class attribute to ``False`` to get the pre-3.7 behaviour. * The ``struct.Struct.format`` type is now ``str`` instead of ``bytes``. * ``repr`` for ``datetime.timedelta`` has changed to include the keyword arguments in the output. * ``tracemalloc.Traceback`` frames are now sorted from oldest to most recent to be more consistent with ``traceback``.

Adding backward compatibility for most of these changes would be easy.

See also the `Porting to Python 3.7 https://docs.python.org/dev/whatsnew/3.7.html#porting-to-python-3-7`_ section of What’s New In Python 3.7.

Micro releases --------------

Sometimes, incompatible changes are introduced in micro releases (``micro`` in ``major.minor.micro``) to fix bugs or security vulnerabilities. Examples:

* Python 3.7.2, ``compileall`` and ``py_compile`` module: the *invalidation_mode* parameter's default value is updated to ``None``; the ``SOURCE_DATE_EPOCH`` environment variable no longer overrides the value of the *invalidation_mode* argument, and determines its default value instead.

* Python 3.7.1, ``xml`` modules: the SAX parser no longer processes general external entities by default to increase security by default.

* Python 3.5.2, ``os.urandom()``: on Linux, if the ``getrandom()`` syscall blocks (the urandom entropy pool is not initialized yet), fall back on reading ``/dev/urandom``.

* Python 3.5.1, ``sys.setrecursionlimit()``: a ``RecursionError`` exception is now raised if the new limit is too low at the current recursion depth.

* Python 3.4.4, ``ssl.create_default_context()``: RC4 was dropped from the default cipher string.

* Python 3.4.3, ``http.client``: ``HTTPSConnection`` now performs all the necessary certificate and hostname checks by default.

* Python 3.4.2, ``email.message``: ``EmailMessage.is_attachment()`` is now a method instead of a property, for consistency with ``Message.is_multipart()``.

* Python 3.4.1, ``os.makedirs(name, mode=0o777, exist_ok=False)``: Before Python 3.4.1, if *exist_ok* was ``True`` and the directory existed, ``makedirs()`` would still raise an error if *mode* did not match the mode of the existing directory. Since this behavior was impossible to implement safely, it was removed in Python 3.4.1 (`bpo-21082 https://bugs.python.org/issue21082`_).

Examples of changes made in micro releases which are not backward incompatible:

* ``ssl.OP_NO_TLSv1_3`` constant was added to 2.7.15, 3.6.3 and 3.7.0 for backwards compatibility with OpenSSL 1.0.2. * ``typing.AsyncContextManager`` was added to Python 3.6.2. * The ``zipfile`` module accepts a path-like object since Python 3.6.2. * ``loop.create_future()`` was added to Python 3.5.2 in the ``asyncio`` module.

No backward compatibility code is needed for such kind of changes.

References ==========

Accepted PEPs:

* `PEP 5 -- Guidelines for Language Evolution https://www.python.org/dev/peps/pep-0005/`_ * `PEP 236 -- Back to the __future__ https://www.python.org/dev/peps/pep-0236/`_ * `PEP 411 -- Provisional packages in the Python standard library https://www.python.org/dev/peps/pep-0411/`_ * `PEP 3002 -- Procedure for Backwards-Incompatible Changes https://www.python.org/dev/peps/pep-3002/`_

Draft PEPs:

* `PEP 602 -- Annual Release Cycle for Python https://www.python.org/dev/peps/pep-0602/`_ * `PEP 605 -- A rolling feature release stream for CPython https://www.python.org/dev/peps/pep-0605/`_ * See also withdrawn `PEP 598 -- Introducing incremental feature releases https://www.python.org/dev/peps/pep-0598/`_

Copyright =========

This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive.

.. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

-- Night gathers, and now my watch begins. It shall not end until my death. _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/SETZ6U... Code of Conduct: http://python.org/psf/codeofconduct/

-- Software Development Engineer at Kakao corp. Tel: +82 010-3353-9127 Email: donghee.na92@gmail.com | denny.i@kakaocorp.com Linkedin: https://www.linkedin.com/in/dong-hee-na-2b713b49/

I re-read your proposal. Please ignore my suggestion. :) I misunderstood core concept of your proposal :) 2019년 10월 18일 (금) 오전 10:19, Victor Stinner 님이 작성:

Hi,

I propose the following PEP to add sys.set_python_compat_version(version) to introduce a partial compatibility with Python 3.8 in the next Python 3.9 version.

I also propose this PEP in a pull request: https://github.com/python/peps/pull/1209 (Please avoid using the temporary PEP number until this PR is merged: officially, the PEP has no number yet.)

If possible, please try to read the whole PEP before replying. I would prefer to avoid knee-jerk reactions :-) The backward compatibility is complex topic where things are not black or white: it's more a grayscale.

IMHO with the incoming end of Python 2 support, it's the right time to propose this PEP!

Victor

PEP: xxx Title: Python Compatibility Version Author: Victor Stinner Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 18-Oct-2019 Python-Version: 3.9

Abstract ========

Add ``sys.set_python_compat_version(version)`` to enable partial compatibility with requested Python version. Add ``sys.get_python_compat_version()``.

Modify a few functions of the standard library to implement a partial compatibility with Python 3.8.

Add ``sys.set_python_min_compat_version(version)`` to deny backward compatibility with Python older than *version*.

Add ``-X compat_version=VERSION`` and ``-X min_compat_version=VERSION`` command line options. Add ``PYTHONCOMPATVERSION`` and ``PYTHONCOMPATMINVERSION`` environment variables.

Rationale =========

The need to evolve frequently -----------------------------

To remain relevant and useful, Python has to evolve frequently. Some enhancements require incompatible changes. Any incompatible change can break an unknown number of Python projects. Developers can decide to not implement a feature because of that.

Users want to get the latest Python version to get new features and better performance. A few incompatible changes prevent them to use their applications on the latest Python version.

This PEP proposes to add a partial compatibility with old Python versions as a tradeoff to fit both use cases.

The main issue with the migration from Python 2 to Python 3 is not that Python 3 is backward incompatible, but how incompatible changes were introduced.

Partial compatibility to minimize the Python maintenance burden ---------------------------------------------------------------

While technically it would be possible to provide a full compatibility with old Python versions, this PEP proposes to minimize the number of functions handling backward compatibility to reduce the maintenance burden of the Python project ("CPython").

Each change introducing backport compatibility to a function should be properly discussed to estimate the maintenance cost in the long-term.

Backward compatibility code will be dropped at each Python release, on a case by case basis. Each compatibility function can be supported for a different number of Python releases depending on its maintenance cost and the estimated risk (number of broken projects) if it's removed.

The maintenance cost does not only come from the code implementing the backward compatibility, but come also from additional tests.

Cases excluded from backward compatibility ------------------------------------------

The performance overhead of a compatibility code must be low when ``sys.set_python_compat_version()`` is not called.

The C API is out of the scope of this PEP: ``Py_LIMITED_API`` macro and the stable ABI are solving this problem differently, see the `PEP 384: Defining a Stable ABI https://www.python.org/dev/peps/pep-0384/`_.

Security fixes which break the backward compatibility on purpose will not get a compatibility layer. Security matters more than compatibility. For example, ``http.client.HTTPSConnection`` was modified in Python 3.4.3 to performs all the necessary certificate and hostname checks by default. It was a deliberate change motivated by the `PEP 476: Enabling certificate verification by default for stdlib http clients https://www.python.org/dev/peps/pep-0476/`_ (`bpo-22417 https://bugs.python.org/issue22417`_).

The Python language does not provide backward compatibility.

Changes which are not clearly incompatible are not covered by this PEP. For example, Python 3.9 changed the default protocol in the ``pickle`` module to Protocol 4 which was first introduced in Python 3.4. This change is backward compatible up to Python 3.4. There is no need to use the Protocol 3 by default when compatibility with Python 3.8 is requested.

New ``DeprecationWarning`` and ``PendingDeprecatingWarning`` warnings of Python 3.9 will not be disabled in Python 3.8 compatibility mode. If a project runs its test suite using ``-Werror`` (treat any warning as an error), these warnings must be fixed, or specific deprecation warnings must be ignored on a case by case basis.

Upgrade a project to a newer Python -----------------------------------

Without backward compatibility, all incompatible changes must be fixed at once, which can be a blocker issue. It is even worse when a project is upgraded to a newer Python which is separated by multiple releases from the old Python.

Postponing an upgrade only makes things worse: each skipped release adds more incompatible changes. The technical debt is only steadily increasing.

With backward compatibility, it becomes possible to upgrade Python increamentally in a project, without having to fix all issues at once.

The "all-or-nothing" is a showstopper to port large Python 2 code bases to Python 3. The list of incompatible changes between Python 2 and Python 3 is long, and it's getting longer at each Python 3.x release.

Cleaning up Python and DeprecationWarning -----------------------------------------

One of the `Zen of Python (PEP 20) https://www.python.org/dev/peps/pep-0020/`_ motto is:

There should be one-- and preferably only one --obvious way to do it.

When Python evolves, new ways emerge inevitably. ``DeprecationWarning`` are emitted to suggest to use the new way, but many developers ignore these warnings, which are silent by default (except in the ``__main__`` module: see the `PEP 565 https://www.python.org/dev/peps/pep-0565/_`). Some developers simply ignore all warnings since they are too many warnings, and so only bother with exceptions when deprecated code is removed.

Sometimes, supporting both ways has a minor maintenance cost, but developers prefer to drop the old way to clean up the code. Such kind of change is backward incompatible.

Some developers can take the end of the Python 2 support as an opportunity to push even more incompatible changes than usual.

Adding backward compatibility as an opt-in prevents to break applications and allows developers to continue to do such cleanup.

Redistribute the maintenance burden -----------------------------------

The backward compatibility involves authors of backward incompatible changes more in the upgrade path.

Examples of backward compatibility ==================================

collections ABC aliases -----------------------

``collections.abc`` aliases to ABC classes have been removed from the ``collections`` module in Python 3.9, after being deprecated since Python 3.3. For example, ``collections.Mapping`` no longer exists.

In Python 3.6, aliases were created in ``collections/__init__.py`` by ``from _collections_abc import *``.

In Python 3.7, a ``__getattr__()`` has been added to the ``collections`` module to emit a DeprecationWarning at the first access to an attribute::

def __getattr__(name): # For backwards compatibility, continue to make the collections ABCs # through Python 3.6 available through the collections module. # Note, no new collections ABCs were added in Python 3.7 if name in _collections_abc.__all__: obj = getattr(_collections_abc, name) import warnings warnings.warn("Using or importing the ABCs from 'collections' instead " "of from 'collections.abc' is deprecated since Python 3.3, " "and in 3.9 it will stop working", DeprecationWarning, stacklevel=2) globals()[name] = obj return obj raise AttributeError(f'module {__name__!r} has no attribute {name!r}')

Compatibility with Python 3.8 can be restored in Python 3.9 by adding back the ``__getattr__()`` function, but only when backward compatibility is requested::

def __getattr__(name): if (sys.get_python_compat_version() < (3, 9) and name in _collections_abc.__all__): ... raise AttributeError(f'module {__name__!r} has no attribute {name!r}')

Deprecated open() "U" mode --------------------------

The "U" mode of ``open()`` is deprecated since Python 3.4 and emits a ``DeprecationWarning``. The `bpo-37330 https://bugs.python.org/issue37330`_ proposes to drop this mode: ``open()`` would raise an exception if ``U`` mode is used.

This change falls into the "cleanup" category: it is not required to implement a feature.

A backward compatibility mode would be trivial to implement and would be welcomed here by users.

Specification =============

sys functions -------------

Add 3 functions to the ``sys`` module:

* ``sys.set_python_compat_version(version)``: set the Python compatibility version. If it has been called previously, use the minimum of requested versions. Raise an exception if ``sys.set_python_min_compat_version(min_version)`` has been called and ``version < min_version``. *version* must be greater than or equal to ``(3, 0)``.

* ``sys.set_python_min_compat_version(min_version)``: set the **minimum** compatibility version. Raise an exception if ``sys.set_python_compat_version(old_version)`` has been called previously and ``old_version < min_version``. *min_version* must be greater than or equal to ``(3, 0)``.

* ``sys.get_python_compat_version()``: get the Python compatibility version. Return a ``tuple`` of 3 integers.

A *version* must a tuple of 2 or 3 integers. ``(major, minor)`` version is equivalent to ``(major, minor, 0)``.

By default, ``sys.get_python_compat_version()`` returns the current Python version.

Example to request compatibility with Python 3.8.0::

import collections

sys.set_python_compat_version((3, 8))

# collections.Mapping alias, removed from Python 3.9, is available # again, even if collections has been imported before calling # set_python_compat_version(). parent = collections.Mapping

Obviously, calling ``sys.set_python_compat_version(version)`` has no effect on code executed before the call. Use ``-X compat_version=VERSION`` command line option or ``PYTHONCOMPATVERSIONVERSION=VERSION`` environment variable to set the compatibility version at Python startup.

Command line ------------

Add ``-X compat_version=VERSION`` and ``-X min_compat_version=VERSION`` command line options: call respectivelly ``sys.set_python_compat_version()`` and ``sys.set_python_min_compat_version()``. ``VERSION`` is a version string with 2 or 3 numbers (``major.minor.micro`` or ``major.minor``). For example, ``-X compat_version=3.8`` calls ``sys.set_python_compat_version((3, 8))``.

Add ``PYTHONCOMPATVERSIONVERSION=VERSION`` and ``PYTHONCOMPATMINVERSION=VERSION=VERSION`` environment variables: call respectivelly ``sys.set_python_compat_version()`` and ``sys.set_python_min_compat_version()``. ``VERSION`` is a version string with the same format that the command line options.

Backwards Compatibility =======================

Introducing ``sys.set_python_compat_version()`` function means that an application will behave differently depending on the compatibility version. Moreover, since the version can be decreased multiple times, the application can behave differently depending on the import order.

Python 3.9 with ``sys.set_python_compat_version((3, 8))`` is not fully compatible with Python 3.8: the compatibility is only partial.

Security Implications =====================

``sys.set_python_compat_version()`` must not disable security fixes.

Alternatives ============

Provide a workaround for each incompatible change -------------------------------------------------

An application can works around most of the incompatible changes which impacts it.

For example, ``collections`` aliases can be added again using::

import collections.abc collections.Mapping = collections.abc.Mapping collections.Sequence = collections.abc.Sequence

Handle backward compatibility in the parser -------------------------------------------

The parser is modified to support multiple versions of the Python language (grammar).

The current Python parser cannot be easily modified for that. AST and grammar are hardcoded to a single Python version.

In Python 3.8, ``compile()`` has an undocumented ``_feature_version`` to not consider ``async`` and ``await`` as keywords.

The latest major language backward incompatible change was Python 3.7 which made ``async`` and ``await`` real keywords. It seems like Twisted was the only affected project, and Twisted had a single affected function (it used a parameter called ``async``).

Handling backward compatibility in the parser seems quite complex, not only to modify the parser, but also for developers who have to check which version of the Python language is used.

from __future__ import python38_syntax ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Add ``pythonXY_syntax`` to the ``__future__`` module. It would enable backward compatibility with Python X.Y syntax, but only for the current file.

With this option, there is no need to change ``sys.implementation.cache_tag`` to use a different ``.pyc`` filename, since the parser would always produce the same output for the same input (except of the optimization level).

Example::

from __future__ import python35_syntax

async = 1 await = 2

Update cache_tag ^^^^^^^^^^^^^^^^

Modify the parser to use ``sys.get_python_compat_version()`` to choose the version of the Python language.

``sys.set_python_compat_version()`` updates ``sys.implementation.cache_tag`` to include the compatibility version without the micro version as a suffix. For example, Python 3.9 uses ``'cpython-39'`` by default, but ``sys.set_python_compat_version((3, 7, 2))`` sets ``cache_tag`` to ``'cpython-39-37'``. Changes of the Python language are now allowed in micro releases.

One problem is that ``import asyncio`` is likely to fail if ``sys.set_python_compat_version((3, 6))`` has been called previously. The code of the ``asyncio`` module requires ``async`` and ``await`` to be real keywords (change done in Python 3.7).

Another problem is that regular users cannot write ``.pyc`` files into system directories, and so cannot create them on demand. It means that ``.pyc`` optimization cannot be used in the backward compatibility mode.

One solution for that is to modify the Python installer and Python package installers to precompile ``.pyc`` files not only for the current Python version, but also for multiple older Python versions (up to Python 3.0?).

Each ``.py`` file would have 3n ``.pyc`` files (3 optimization levels), where ``n`` is the number of supported Python versions. For example, it means 6 ``.pyc`` files, instead of 3, to support Python 3.8 and Python 3.9.

Temporary moratorium on incompatible changes --------------------------------------------

In 2009, the PEP 3003 "Python Language Moratorium" proposed to a temporary moratorium (suspension) of all changes to the Python language syntax, semantics, and built-ins for Python 3.1 and Python 3.2.

In May 2018, during PEP 572 discussions, it was also proposed to slow down Python changes: see the python-dev thread `Slow down... https://mail.python.org/archives/list/python-dev@python.org/thread/HHKRXOMRJ...`_

`Barry Warsaw's call on this https://mail.python.org/archives/list/python-dev@python.org/message/XR7IF2OB...`_:

I don’t believe that the way for Python to remain relevant and useful for the next 10 years is to cease all language evolution. Who knows what the computing landscape will look like in 5 years, let alone 10? Something as arbitrary as a 10 year moratorium is (again, IMHO) a death sentence for the language.

PEP 387 -------

`PEP 387 -- Backwards Compatibility Policy https://www.python.org/dev/peps/pep-0387/`_ proposes a process to make incompatible changes. The main point is the 4th step of the process:

See if there's any feedback. Users not involved in the original discussions may comment now after seeing the warning. Perhaps reconsider.

PEP 497 -------

`PEP 497 -- A standard mechanism for backward compatibility https://www.python.org/dev/peps/pep-0497/`_ proposes different solutions to provide backward compatibility.

Except of the ``__past__`` mechanism idea, the PEP 497 does not propose concrete solutions:

When an incompatible change to core language syntax or semantics is being made, Python-dev's policy is to prefer and expect that, wherever possible, a mechanism for backward compatibility be considered and provided for future Python versions after the breaking change is adopted by default, in addition to any mechanisms proposed for forward compatibility such as new future_statements.

Examples of incompatible changes ================================

Python 3.8 ----------

Examples of Python 3.8 incompatible changes:

* (During beta phase) ``PyCode_New()`` required a new parameter: it broke all Cython extensions (all projects distributing precompiled Cython code). This change has been reverted during the 3.8 beta phase and a new ``PyCode_NewWithPosOnlyArgs()`` function was added instead.

* ``types.CodeType`` requires an additional mandatory parameter. The ``CodeType.replace()`` function was added to help projects to no longer depend on the exact signature of the ``CodeType`` constructor.

* C extensions are no longer linked to libpython.

* ``sys.abiflags`` changed from ``'m'`` to an empty string. For example, ``python3.8m`` program is gone.

* The C structure ``PyInterpreterState`` was made opaque.

* Blender:

* https://bugzilla.redhat.com/show_bug.cgi?id=1734980#c6 * https://developer.blender.org/D6038

* XML attribute order: `bpo-34160 https://bugs.python.org/issue34160`_. Broken projects:

* `coverage https://bugs.python.org/issue34160#msg329612`_ * `docutils https://sourceforge.net/p/docutils/bugs/359/`_ * `pcs https://bugzilla.redhat.com/show_bug.cgi?id=1705475`_ * `python-glyphsLib https://bugzilla.redhat.com/show_bug.cgi?id=1705391`_

Backward compatibility cannot be added for all these changes. For example, changes in the C API and in the build system are out of the scope of this PEP.

See `What’s New In Python 3.8: API and Feature Removals https://docs.python.org/dev/whatsnew/3.8.html#api-and-feature-removals`_ for all changes.

See also the `Porting to Python 3.8 https://docs.python.org/dev/whatsnew/3.8.html#porting-to-python-3-8`_ section of What’s New In Python 3.8.

Python 3.7 ----------

Examples of Python 3.7 incompatible changes:

* ``async`` and ``await`` are now reserved keywords. * Several undocumented internal imports were removed. One example is that ``os.errno`` is no longer available; use ``import errno`` directly instead. Note that such undocumented internal imports may be removed any time without notice, even in micro version releases. * Unknown escapes consisting of ``'\'`` and an ASCII letter in replacement templates for ``re.sub()`` were deprecated in Python 3.5, and will now cause an error. * The ``asyncio.windows_utils.socketpair()`` function has been removed: it was an alias to ``socket.socketpair()``. * ``asyncio`` no longer exports the ``selectors`` and ``_overlapped`` modules as ``asyncio.selectors`` and ``asyncio._overlapped``. Replace ``from asyncio import selectors`` with ``import selectors``. * PEP 479 is enabled for all code in Python 3.7, meaning that ``StopIteration`` exceptions raised directly or indirectly in coroutines and generators are transformed into ``RuntimeError`` exceptions. * ``socketserver.ThreadingMixIn.server_close()`` now waits until all non-daemon threads complete. Set the new ``block_on_close`` class attribute to ``False`` to get the pre-3.7 behaviour. * The ``struct.Struct.format`` type is now ``str`` instead of ``bytes``. * ``repr`` for ``datetime.timedelta`` has changed to include the keyword arguments in the output. * ``tracemalloc.Traceback`` frames are now sorted from oldest to most recent to be more consistent with ``traceback``.

Adding backward compatibility for most of these changes would be easy.

See also the `Porting to Python 3.7 https://docs.python.org/dev/whatsnew/3.7.html#porting-to-python-3-7`_ section of What’s New In Python 3.7.

Micro releases --------------

Sometimes, incompatible changes are introduced in micro releases (``micro`` in ``major.minor.micro``) to fix bugs or security vulnerabilities. Examples:

* Python 3.7.2, ``compileall`` and ``py_compile`` module: the *invalidation_mode* parameter's default value is updated to ``None``; the ``SOURCE_DATE_EPOCH`` environment variable no longer overrides the value of the *invalidation_mode* argument, and determines its default value instead.

* Python 3.7.1, ``xml`` modules: the SAX parser no longer processes general external entities by default to increase security by default.

* Python 3.5.2, ``os.urandom()``: on Linux, if the ``getrandom()`` syscall blocks (the urandom entropy pool is not initialized yet), fall back on reading ``/dev/urandom``.

* Python 3.5.1, ``sys.setrecursionlimit()``: a ``RecursionError`` exception is now raised if the new limit is too low at the current recursion depth.

* Python 3.4.4, ``ssl.create_default_context()``: RC4 was dropped from the default cipher string.

* Python 3.4.3, ``http.client``: ``HTTPSConnection`` now performs all the necessary certificate and hostname checks by default.

* Python 3.4.2, ``email.message``: ``EmailMessage.is_attachment()`` is now a method instead of a property, for consistency with ``Message.is_multipart()``.

* Python 3.4.1, ``os.makedirs(name, mode=0o777, exist_ok=False)``: Before Python 3.4.1, if *exist_ok* was ``True`` and the directory existed, ``makedirs()`` would still raise an error if *mode* did not match the mode of the existing directory. Since this behavior was impossible to implement safely, it was removed in Python 3.4.1 (`bpo-21082 https://bugs.python.org/issue21082`_).

Examples of changes made in micro releases which are not backward incompatible:

* ``ssl.OP_NO_TLSv1_3`` constant was added to 2.7.15, 3.6.3 and 3.7.0 for backwards compatibility with OpenSSL 1.0.2. * ``typing.AsyncContextManager`` was added to Python 3.6.2. * The ``zipfile`` module accepts a path-like object since Python 3.6.2. * ``loop.create_future()`` was added to Python 3.5.2 in the ``asyncio`` module.

No backward compatibility code is needed for such kind of changes.

References ==========

Accepted PEPs:

* `PEP 5 -- Guidelines for Language Evolution https://www.python.org/dev/peps/pep-0005/`_ * `PEP 236 -- Back to the __future__ https://www.python.org/dev/peps/pep-0236/`_ * `PEP 411 -- Provisional packages in the Python standard library https://www.python.org/dev/peps/pep-0411/`_ * `PEP 3002 -- Procedure for Backwards-Incompatible Changes https://www.python.org/dev/peps/pep-3002/`_

Draft PEPs:

* `PEP 602 -- Annual Release Cycle for Python https://www.python.org/dev/peps/pep-0602/`_ * `PEP 605 -- A rolling feature release stream for CPython https://www.python.org/dev/peps/pep-0605/`_ * See also withdrawn `PEP 598 -- Introducing incremental feature releases https://www.python.org/dev/peps/pep-0598/`_

Copyright =========

This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive.

.. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

-- Night gathers, and now my watch begins. It shall not end until my death. _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/SETZ6U... Code of Conduct: http://python.org/psf/codeofconduct/

-- Software Development Engineer at Kakao corp. Tel: +82 010-3353-9127 Email: donghee.na92@gmail.com | denny.i@kakaocorp.com Linkedin: https://www.linkedin.com/in/dong-hee-na-2b713b49/

Hi Victor, I think this is a nice idea, but I do wonder how this would be used in practice: The problem is that Python projects typically use lots of 3rd party modules and if a few decide to use/rely on the feature, this may create a situation where packages then compete over which compatibility mode to enable. In such situations, I typically prefer failing early, since detecting whether a package requires an older implementation of a Python API is hard from the outside. Regardless of the outcome of the PEP, what I think would be a great addition is a place where to find the the list of incompatible changes per Python release in a single document - much like you have done in the PEP. Cheers, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts

...

...

Python Projects, Coaching and Consulting ... http://www.egenix.com/ Python Database Interfaces ... http://products.egenix.com/ Plone/Zope Database Interfaces ... http://zope.egenix.com/

::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/ On 18.10.2019 03:02, Victor Stinner wrote:

Hi,

I propose the following PEP to add sys.set_python_compat_version(version) to introduce a partial compatibility with Python 3.8 in the next Python 3.9 version.

I also propose this PEP in a pull request: https://github.com/python/peps/pull/1209 (Please avoid using the temporary PEP number until this PR is merged: officially, the PEP has no number yet.)

If possible, please try to read the whole PEP before replying. I would prefer to avoid knee-jerk reactions :-) The backward compatibility is complex topic where things are not black or white: it's more a grayscale.

IMHO with the incoming end of Python 2 support, it's the right time to propose this PEP!

Victor

PEP: xxx Title: Python Compatibility Version Author: Victor Stinner Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 18-Oct-2019 Python-Version: 3.9

Abstract ========

Add ``sys.set_python_compat_version(version)`` to enable partial compatibility with requested Python version. Add ``sys.get_python_compat_version()``.

Modify a few functions of the standard library to implement a partial compatibility with Python 3.8.

Add ``sys.set_python_min_compat_version(version)`` to deny backward compatibility with Python older than *version*.

Add ``-X compat_version=VERSION`` and ``-X min_compat_version=VERSION`` command line options. Add ``PYTHONCOMPATVERSION`` and ``PYTHONCOMPATMINVERSION`` environment variables.

Rationale =========

The need to evolve frequently -----------------------------

To remain relevant and useful, Python has to evolve frequently. Some enhancements require incompatible changes. Any incompatible change can break an unknown number of Python projects. Developers can decide to not implement a feature because of that.

Users want to get the latest Python version to get new features and better performance. A few incompatible changes prevent them to use their applications on the latest Python version.

This PEP proposes to add a partial compatibility with old Python versions as a tradeoff to fit both use cases.

The main issue with the migration from Python 2 to Python 3 is not that Python 3 is backward incompatible, but how incompatible changes were introduced.

Partial compatibility to minimize the Python maintenance burden ---------------------------------------------------------------

While technically it would be possible to provide a full compatibility with old Python versions, this PEP proposes to minimize the number of functions handling backward compatibility to reduce the maintenance burden of the Python project ("CPython").

Each change introducing backport compatibility to a function should be properly discussed to estimate the maintenance cost in the long-term.

Backward compatibility code will be dropped at each Python release, on a case by case basis. Each compatibility function can be supported for a different number of Python releases depending on its maintenance cost and the estimated risk (number of broken projects) if it's removed.

The maintenance cost does not only come from the code implementing the backward compatibility, but come also from additional tests.

Cases excluded from backward compatibility ------------------------------------------

The performance overhead of a compatibility code must be low when ``sys.set_python_compat_version()`` is not called.

The C API is out of the scope of this PEP: ``Py_LIMITED_API`` macro and the stable ABI are solving this problem differently, see the `PEP 384: Defining a Stable ABI https://www.python.org/dev/peps/pep-0384/`_.

Security fixes which break the backward compatibility on purpose will not get a compatibility layer. Security matters more than compatibility. For example, ``http.client.HTTPSConnection`` was modified in Python 3.4.3 to performs all the necessary certificate and hostname checks by default. It was a deliberate change motivated by the `PEP 476: Enabling certificate verification by default for stdlib http clients https://www.python.org/dev/peps/pep-0476/`_ (`bpo-22417 https://bugs.python.org/issue22417`_).

The Python language does not provide backward compatibility.

Changes which are not clearly incompatible are not covered by this PEP. For example, Python 3.9 changed the default protocol in the ``pickle`` module to Protocol 4 which was first introduced in Python 3.4. This change is backward compatible up to Python 3.4. There is no need to use the Protocol 3 by default when compatibility with Python 3.8 is requested.

New ``DeprecationWarning`` and ``PendingDeprecatingWarning`` warnings of Python 3.9 will not be disabled in Python 3.8 compatibility mode. If a project runs its test suite using ``-Werror`` (treat any warning as an error), these warnings must be fixed, or specific deprecation warnings must be ignored on a case by case basis.

Upgrade a project to a newer Python -----------------------------------

Without backward compatibility, all incompatible changes must be fixed at once, which can be a blocker issue. It is even worse when a project is upgraded to a newer Python which is separated by multiple releases from the old Python.

Postponing an upgrade only makes things worse: each skipped release adds more incompatible changes. The technical debt is only steadily increasing.

With backward compatibility, it becomes possible to upgrade Python increamentally in a project, without having to fix all issues at once.

The "all-or-nothing" is a showstopper to port large Python 2 code bases to Python 3. The list of incompatible changes between Python 2 and Python 3 is long, and it's getting longer at each Python 3.x release.

Cleaning up Python and DeprecationWarning -----------------------------------------

One of the `Zen of Python (PEP 20) https://www.python.org/dev/peps/pep-0020/`_ motto is:

There should be one-- and preferably only one --obvious way to do it.

When Python evolves, new ways emerge inevitably. ``DeprecationWarning`` are emitted to suggest to use the new way, but many developers ignore these warnings, which are silent by default (except in the ``__main__`` module: see the `PEP 565 https://www.python.org/dev/peps/pep-0565/_`). Some developers simply ignore all warnings since they are too many warnings, and so only bother with exceptions when deprecated code is removed.

Sometimes, supporting both ways has a minor maintenance cost, but developers prefer to drop the old way to clean up the code. Such kind of change is backward incompatible.

Some developers can take the end of the Python 2 support as an opportunity to push even more incompatible changes than usual.

Adding backward compatibility as an opt-in prevents to break applications and allows developers to continue to do such cleanup.

Redistribute the maintenance burden -----------------------------------

The backward compatibility involves authors of backward incompatible changes more in the upgrade path.

Examples of backward compatibility ==================================

collections ABC aliases -----------------------

``collections.abc`` aliases to ABC classes have been removed from the ``collections`` module in Python 3.9, after being deprecated since Python 3.3. For example, ``collections.Mapping`` no longer exists.

In Python 3.6, aliases were created in ``collections/__init__.py`` by ``from _collections_abc import *``.

In Python 3.7, a ``__getattr__()`` has been added to the ``collections`` module to emit a DeprecationWarning at the first access to an attribute::

def __getattr__(name): # For backwards compatibility, continue to make the collections ABCs # through Python 3.6 available through the collections module. # Note, no new collections ABCs were added in Python 3.7 if name in _collections_abc.__all__: obj = getattr(_collections_abc, name) import warnings warnings.warn("Using or importing the ABCs from 'collections' instead " "of from 'collections.abc' is deprecated since Python 3.3, " "and in 3.9 it will stop working", DeprecationWarning, stacklevel=2) globals()[name] = obj return obj raise AttributeError(f'module {__name__!r} has no attribute {name!r}')

Compatibility with Python 3.8 can be restored in Python 3.9 by adding back the ``__getattr__()`` function, but only when backward compatibility is requested::

def __getattr__(name): if (sys.get_python_compat_version() < (3, 9) and name in _collections_abc.__all__): ... raise AttributeError(f'module {__name__!r} has no attribute {name!r}')

Deprecated open() "U" mode --------------------------

The "U" mode of ``open()`` is deprecated since Python 3.4 and emits a ``DeprecationWarning``. The `bpo-37330 https://bugs.python.org/issue37330`_ proposes to drop this mode: ``open()`` would raise an exception if ``U`` mode is used.

This change falls into the "cleanup" category: it is not required to implement a feature.

A backward compatibility mode would be trivial to implement and would be welcomed here by users.

Specification =============

sys functions -------------

Add 3 functions to the ``sys`` module:

* ``sys.set_python_compat_version(version)``: set the Python compatibility version. If it has been called previously, use the minimum of requested versions. Raise an exception if ``sys.set_python_min_compat_version(min_version)`` has been called and ``version < min_version``. *version* must be greater than or equal to ``(3, 0)``.

* ``sys.set_python_min_compat_version(min_version)``: set the **minimum** compatibility version. Raise an exception if ``sys.set_python_compat_version(old_version)`` has been called previously and ``old_version < min_version``. *min_version* must be greater than or equal to ``(3, 0)``.

* ``sys.get_python_compat_version()``: get the Python compatibility version. Return a ``tuple`` of 3 integers.

A *version* must a tuple of 2 or 3 integers. ``(major, minor)`` version is equivalent to ``(major, minor, 0)``.

By default, ``sys.get_python_compat_version()`` returns the current Python version.

Example to request compatibility with Python 3.8.0::

import collections

sys.set_python_compat_version((3, 8))

# collections.Mapping alias, removed from Python 3.9, is available # again, even if collections has been imported before calling # set_python_compat_version(). parent = collections.Mapping

Obviously, calling ``sys.set_python_compat_version(version)`` has no effect on code executed before the call. Use ``-X compat_version=VERSION`` command line option or ``PYTHONCOMPATVERSIONVERSION=VERSION`` environment variable to set the compatibility version at Python startup.

Command line ------------

Add ``-X compat_version=VERSION`` and ``-X min_compat_version=VERSION`` command line options: call respectivelly ``sys.set_python_compat_version()`` and ``sys.set_python_min_compat_version()``. ``VERSION`` is a version string with 2 or 3 numbers (``major.minor.micro`` or ``major.minor``). For example, ``-X compat_version=3.8`` calls ``sys.set_python_compat_version((3, 8))``.

Add ``PYTHONCOMPATVERSIONVERSION=VERSION`` and ``PYTHONCOMPATMINVERSION=VERSION=VERSION`` environment variables: call respectivelly ``sys.set_python_compat_version()`` and ``sys.set_python_min_compat_version()``. ``VERSION`` is a version string with the same format that the command line options.

Backwards Compatibility =======================

Introducing ``sys.set_python_compat_version()`` function means that an application will behave differently depending on the compatibility version. Moreover, since the version can be decreased multiple times, the application can behave differently depending on the import order.

Python 3.9 with ``sys.set_python_compat_version((3, 8))`` is not fully compatible with Python 3.8: the compatibility is only partial.

Security Implications =====================

``sys.set_python_compat_version()`` must not disable security fixes.

Alternatives ============

Provide a workaround for each incompatible change -------------------------------------------------

An application can works around most of the incompatible changes which impacts it.

For example, ``collections`` aliases can be added again using::

import collections.abc collections.Mapping = collections.abc.Mapping collections.Sequence = collections.abc.Sequence

Handle backward compatibility in the parser -------------------------------------------

The parser is modified to support multiple versions of the Python language (grammar).

The current Python parser cannot be easily modified for that. AST and grammar are hardcoded to a single Python version.

In Python 3.8, ``compile()`` has an undocumented ``_feature_version`` to not consider ``async`` and ``await`` as keywords.

The latest major language backward incompatible change was Python 3.7 which made ``async`` and ``await`` real keywords. It seems like Twisted was the only affected project, and Twisted had a single affected function (it used a parameter called ``async``).

Handling backward compatibility in the parser seems quite complex, not only to modify the parser, but also for developers who have to check which version of the Python language is used.

from __future__ import python38_syntax ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Add ``pythonXY_syntax`` to the ``__future__`` module. It would enable backward compatibility with Python X.Y syntax, but only for the current file.

With this option, there is no need to change ``sys.implementation.cache_tag`` to use a different ``.pyc`` filename, since the parser would always produce the same output for the same input (except of the optimization level).

Example::

from __future__ import python35_syntax

async = 1 await = 2

Update cache_tag ^^^^^^^^^^^^^^^^

Modify the parser to use ``sys.get_python_compat_version()`` to choose the version of the Python language.

``sys.set_python_compat_version()`` updates ``sys.implementation.cache_tag`` to include the compatibility version without the micro version as a suffix. For example, Python 3.9 uses ``'cpython-39'`` by default, but ``sys.set_python_compat_version((3, 7, 2))`` sets ``cache_tag`` to ``'cpython-39-37'``. Changes of the Python language are now allowed in micro releases.

One problem is that ``import asyncio`` is likely to fail if ``sys.set_python_compat_version((3, 6))`` has been called previously. The code of the ``asyncio`` module requires ``async`` and ``await`` to be real keywords (change done in Python 3.7).

Another problem is that regular users cannot write ``.pyc`` files into system directories, and so cannot create them on demand. It means that ``.pyc`` optimization cannot be used in the backward compatibility mode.

One solution for that is to modify the Python installer and Python package installers to precompile ``.pyc`` files not only for the current Python version, but also for multiple older Python versions (up to Python 3.0?).

Each ``.py`` file would have 3n ``.pyc`` files (3 optimization levels), where ``n`` is the number of supported Python versions. For example, it means 6 ``.pyc`` files, instead of 3, to support Python 3.8 and Python 3.9.

Temporary moratorium on incompatible changes --------------------------------------------

In 2009, the PEP 3003 "Python Language Moratorium" proposed to a temporary moratorium (suspension) of all changes to the Python language syntax, semantics, and built-ins for Python 3.1 and Python 3.2.

In May 2018, during PEP 572 discussions, it was also proposed to slow down Python changes: see the python-dev thread `Slow down... https://mail.python.org/archives/list/python-dev@python.org/thread/HHKRXOMRJ...`_

`Barry Warsaw's call on this https://mail.python.org/archives/list/python-dev@python.org/message/XR7IF2OB...`_:

I don’t believe that the way for Python to remain relevant and useful for the next 10 years is to cease all language evolution. Who knows what the computing landscape will look like in 5 years, let alone 10? Something as arbitrary as a 10 year moratorium is (again, IMHO) a death sentence for the language.

PEP 387 -------

`PEP 387 -- Backwards Compatibility Policy https://www.python.org/dev/peps/pep-0387/`_ proposes a process to make incompatible changes. The main point is the 4th step of the process:

See if there's any feedback. Users not involved in the original discussions may comment now after seeing the warning. Perhaps reconsider.

PEP 497 -------

`PEP 497 -- A standard mechanism for backward compatibility https://www.python.org/dev/peps/pep-0497/`_ proposes different solutions to provide backward compatibility.

Except of the ``__past__`` mechanism idea, the PEP 497 does not propose concrete solutions:

When an incompatible change to core language syntax or semantics is being made, Python-dev's policy is to prefer and expect that, wherever possible, a mechanism for backward compatibility be considered and provided for future Python versions after the breaking change is adopted by default, in addition to any mechanisms proposed for forward compatibility such as new future_statements.

Examples of incompatible changes ================================

Python 3.8 ----------

Examples of Python 3.8 incompatible changes:

* (During beta phase) ``PyCode_New()`` required a new parameter: it broke all Cython extensions (all projects distributing precompiled Cython code). This change has been reverted during the 3.8 beta phase and a new ``PyCode_NewWithPosOnlyArgs()`` function was added instead.

* ``types.CodeType`` requires an additional mandatory parameter. The ``CodeType.replace()`` function was added to help projects to no longer depend on the exact signature of the ``CodeType`` constructor.

* C extensions are no longer linked to libpython.

* ``sys.abiflags`` changed from ``'m'`` to an empty string. For example, ``python3.8m`` program is gone.

* The C structure ``PyInterpreterState`` was made opaque.

* Blender:

* https://bugzilla.redhat.com/show_bug.cgi?id=1734980#c6 * https://developer.blender.org/D6038

* XML attribute order: `bpo-34160 https://bugs.python.org/issue34160`_. Broken projects:

* `coverage https://bugs.python.org/issue34160#msg329612`_ * `docutils https://sourceforge.net/p/docutils/bugs/359/`_ * `pcs https://bugzilla.redhat.com/show_bug.cgi?id=1705475`_ * `python-glyphsLib https://bugzilla.redhat.com/show_bug.cgi?id=1705391`_

Backward compatibility cannot be added for all these changes. For example, changes in the C API and in the build system are out of the scope of this PEP.

See `What’s New In Python 3.8: API and Feature Removals https://docs.python.org/dev/whatsnew/3.8.html#api-and-feature-removals`_ for all changes.

See also the `Porting to Python 3.8 https://docs.python.org/dev/whatsnew/3.8.html#porting-to-python-3-8`_ section of What’s New In Python 3.8.

Python 3.7 ----------

Examples of Python 3.7 incompatible changes:

* ``async`` and ``await`` are now reserved keywords. * Several undocumented internal imports were removed. One example is that ``os.errno`` is no longer available; use ``import errno`` directly instead. Note that such undocumented internal imports may be removed any time without notice, even in micro version releases. * Unknown escapes consisting of ``'\'`` and an ASCII letter in replacement templates for ``re.sub()`` were deprecated in Python 3.5, and will now cause an error. * The ``asyncio.windows_utils.socketpair()`` function has been removed: it was an alias to ``socket.socketpair()``. * ``asyncio`` no longer exports the ``selectors`` and ``_overlapped`` modules as ``asyncio.selectors`` and ``asyncio._overlapped``. Replace ``from asyncio import selectors`` with ``import selectors``. * PEP 479 is enabled for all code in Python 3.7, meaning that ``StopIteration`` exceptions raised directly or indirectly in coroutines and generators are transformed into ``RuntimeError`` exceptions. * ``socketserver.ThreadingMixIn.server_close()`` now waits until all non-daemon threads complete. Set the new ``block_on_close`` class attribute to ``False`` to get the pre-3.7 behaviour. * The ``struct.Struct.format`` type is now ``str`` instead of ``bytes``. * ``repr`` for ``datetime.timedelta`` has changed to include the keyword arguments in the output. * ``tracemalloc.Traceback`` frames are now sorted from oldest to most recent to be more consistent with ``traceback``.

Adding backward compatibility for most of these changes would be easy.

See also the `Porting to Python 3.7 https://docs.python.org/dev/whatsnew/3.7.html#porting-to-python-3-7`_ section of What’s New In Python 3.7.

Micro releases --------------

Sometimes, incompatible changes are introduced in micro releases (``micro`` in ``major.minor.micro``) to fix bugs or security vulnerabilities. Examples:

* Python 3.7.2, ``compileall`` and ``py_compile`` module: the *invalidation_mode* parameter's default value is updated to ``None``; the ``SOURCE_DATE_EPOCH`` environment variable no longer overrides the value of the *invalidation_mode* argument, and determines its default value instead.

* Python 3.7.1, ``xml`` modules: the SAX parser no longer processes general external entities by default to increase security by default.

* Python 3.5.2, ``os.urandom()``: on Linux, if the ``getrandom()`` syscall blocks (the urandom entropy pool is not initialized yet), fall back on reading ``/dev/urandom``.

* Python 3.5.1, ``sys.setrecursionlimit()``: a ``RecursionError`` exception is now raised if the new limit is too low at the current recursion depth.

* Python 3.4.4, ``ssl.create_default_context()``: RC4 was dropped from the default cipher string.

* Python 3.4.3, ``http.client``: ``HTTPSConnection`` now performs all the necessary certificate and hostname checks by default.

* Python 3.4.2, ``email.message``: ``EmailMessage.is_attachment()`` is now a method instead of a property, for consistency with ``Message.is_multipart()``.

* Python 3.4.1, ``os.makedirs(name, mode=0o777, exist_ok=False)``: Before Python 3.4.1, if *exist_ok* was ``True`` and the directory existed, ``makedirs()`` would still raise an error if *mode* did not match the mode of the existing directory. Since this behavior was impossible to implement safely, it was removed in Python 3.4.1 (`bpo-21082 https://bugs.python.org/issue21082`_).

Examples of changes made in micro releases which are not backward incompatible:

* ``ssl.OP_NO_TLSv1_3`` constant was added to 2.7.15, 3.6.3 and 3.7.0 for backwards compatibility with OpenSSL 1.0.2. * ``typing.AsyncContextManager`` was added to Python 3.6.2. * The ``zipfile`` module accepts a path-like object since Python 3.6.2. * ``loop.create_future()`` was added to Python 3.5.2 in the ``asyncio`` module.

No backward compatibility code is needed for such kind of changes.

References ==========

Accepted PEPs:

* `PEP 5 -- Guidelines for Language Evolution https://www.python.org/dev/peps/pep-0005/`_ * `PEP 236 -- Back to the __future__ https://www.python.org/dev/peps/pep-0236/`_ * `PEP 411 -- Provisional packages in the Python standard library https://www.python.org/dev/peps/pep-0411/`_ * `PEP 3002 -- Procedure for Backwards-Incompatible Changes https://www.python.org/dev/peps/pep-3002/`_

Draft PEPs:

* `PEP 602 -- Annual Release Cycle for Python https://www.python.org/dev/peps/pep-0602/`_ * `PEP 605 -- A rolling feature release stream for CPython https://www.python.org/dev/peps/pep-0605/`_ * See also withdrawn `PEP 598 -- Introducing incremental feature releases https://www.python.org/dev/peps/pep-0598/`_

Copyright =========

This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive.

.. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

Hi, Just bear in mind that I have read only a couple sentences in the PEP. But as soon as you propose to offer only "partial compatibility" with a previous version, then I think it's worse than useless. You are introducing an additional Python version, i.e. "3.9 bended towards 3.8 partial compatibility", which is neither fully 3.8 nor fully 3.9. It won't make life any easier for third-party library maintainers, on the contrary: it will make it worse. Regards Antoine. On Fri, 18 Oct 2019 03:02:50 +0200 Victor Stinner wrote:

Hi,

I propose the following PEP to add sys.set_python_compat_version(version) to introduce a partial compatibility with Python 3.8 in the next Python 3.9 version.

I also propose this PEP in a pull request: https://github.com/python/peps/pull/1209 (Please avoid using the temporary PEP number until this PR is merged: officially, the PEP has no number yet.)

If possible, please try to read the whole PEP before replying. I would prefer to avoid knee-jerk reactions :-) The backward compatibility is complex topic where things are not black or white: it's more a grayscale.

IMHO with the incoming end of Python 2 support, it's the right time to propose this PEP!

Victor

PEP: xxx Title: Python Compatibility Version Author: Victor Stinner Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 18-Oct-2019 Python-Version: 3.9

Abstract ========

Add ``sys.set_python_compat_version(version)`` to enable partial compatibility with requested Python version. Add ``sys.get_python_compat_version()``.

Modify a few functions of the standard library to implement a partial compatibility with Python 3.8.

Add ``sys.set_python_min_compat_version(version)`` to deny backward compatibility with Python older than *version*.

Add ``-X compat_version=VERSION`` and ``-X min_compat_version=VERSION`` command line options. Add ``PYTHONCOMPATVERSION`` and ``PYTHONCOMPATMINVERSION`` environment variables.

Rationale =========

The need to evolve frequently -----------------------------

To remain relevant and useful, Python has to evolve frequently. Some enhancements require incompatible changes. Any incompatible change can break an unknown number of Python projects. Developers can decide to not implement a feature because of that.

Users want to get the latest Python version to get new features and better performance. A few incompatible changes prevent them to use their applications on the latest Python version.

This PEP proposes to add a partial compatibility with old Python versions as a tradeoff to fit both use cases.

The main issue with the migration from Python 2 to Python 3 is not that Python 3 is backward incompatible, but how incompatible changes were introduced.

Partial compatibility to minimize the Python maintenance burden ---------------------------------------------------------------

While technically it would be possible to provide a full compatibility with old Python versions, this PEP proposes to minimize the number of functions handling backward compatibility to reduce the maintenance burden of the Python project ("CPython").

Each change introducing backport compatibility to a function should be properly discussed to estimate the maintenance cost in the long-term.

Backward compatibility code will be dropped at each Python release, on a case by case basis. Each compatibility function can be supported for a different number of Python releases depending on its maintenance cost and the estimated risk (number of broken projects) if it's removed.

The maintenance cost does not only come from the code implementing the backward compatibility, but come also from additional tests.

Cases excluded from backward compatibility ------------------------------------------

The performance overhead of a compatibility code must be low when ``sys.set_python_compat_version()`` is not called.

The C API is out of the scope of this PEP: ``Py_LIMITED_API`` macro and the stable ABI are solving this problem differently, see the `PEP 384: Defining a Stable ABI https://www.python.org/dev/peps/pep-0384/`_.

Security fixes which break the backward compatibility on purpose will not get a compatibility layer. Security matters more than compatibility. For example, ``http.client.HTTPSConnection`` was modified in Python 3.4.3 to performs all the necessary certificate and hostname checks by default. It was a deliberate change motivated by the `PEP 476: Enabling certificate verification by default for stdlib http clients https://www.python.org/dev/peps/pep-0476/`_ (`bpo-22417 https://bugs.python.org/issue22417`_).

The Python language does not provide backward compatibility.

Changes which are not clearly incompatible are not covered by this PEP. For example, Python 3.9 changed the default protocol in the ``pickle`` module to Protocol 4 which was first introduced in Python 3.4. This change is backward compatible up to Python 3.4. There is no need to use the Protocol 3 by default when compatibility with Python 3.8 is requested.

New ``DeprecationWarning`` and ``PendingDeprecatingWarning`` warnings of Python 3.9 will not be disabled in Python 3.8 compatibility mode. If a project runs its test suite using ``-Werror`` (treat any warning as an error), these warnings must be fixed, or specific deprecation warnings must be ignored on a case by case basis.

Upgrade a project to a newer Python -----------------------------------

Without backward compatibility, all incompatible changes must be fixed at once, which can be a blocker issue. It is even worse when a project is upgraded to a newer Python which is separated by multiple releases from the old Python.

Postponing an upgrade only makes things worse: each skipped release adds more incompatible changes. The technical debt is only steadily increasing.

With backward compatibility, it becomes possible to upgrade Python increamentally in a project, without having to fix all issues at once.

The "all-or-nothing" is a showstopper to port large Python 2 code bases to Python 3. The list of incompatible changes between Python 2 and Python 3 is long, and it's getting longer at each Python 3.x release.

Cleaning up Python and DeprecationWarning -----------------------------------------

One of the `Zen of Python (PEP 20) https://www.python.org/dev/peps/pep-0020/`_ motto is:

There should be one-- and preferably only one --obvious way to do it.

When Python evolves, new ways emerge inevitably. ``DeprecationWarning`` are emitted to suggest to use the new way, but many developers ignore these warnings, which are silent by default (except in the ``__main__`` module: see the `PEP 565 https://www.python.org/dev/peps/pep-0565/_`). Some developers simply ignore all warnings since they are too many warnings, and so only bother with exceptions when deprecated code is removed.

Sometimes, supporting both ways has a minor maintenance cost, but developers prefer to drop the old way to clean up the code. Such kind of change is backward incompatible.

Some developers can take the end of the Python 2 support as an opportunity to push even more incompatible changes than usual.

Adding backward compatibility as an opt-in prevents to break applications and allows developers to continue to do such cleanup.

Redistribute the maintenance burden -----------------------------------

The backward compatibility involves authors of backward incompatible changes more in the upgrade path.

Examples of backward compatibility ==================================

collections ABC aliases -----------------------

``collections.abc`` aliases to ABC classes have been removed from the ``collections`` module in Python 3.9, after being deprecated since Python 3.3. For example, ``collections.Mapping`` no longer exists.

In Python 3.6, aliases were created in ``collections/__init__.py`` by ``from _collections_abc import *``.

In Python 3.7, a ``__getattr__()`` has been added to the ``collections`` module to emit a DeprecationWarning at the first access to an attribute::

def __getattr__(name): # For backwards compatibility, continue to make the collections ABCs # through Python 3.6 available through the collections module. # Note, no new collections ABCs were added in Python 3.7 if name in _collections_abc.__all__: obj = getattr(_collections_abc, name) import warnings warnings.warn("Using or importing the ABCs from 'collections' instead " "of from 'collections.abc' is deprecated since Python 3.3, " "and in 3.9 it will stop working", DeprecationWarning, stacklevel=2) globals()[name] = obj return obj raise AttributeError(f'module {__name__!r} has no attribute {name!r}')

Compatibility with Python 3.8 can be restored in Python 3.9 by adding back the ``__getattr__()`` function, but only when backward compatibility is requested::

def __getattr__(name): if (sys.get_python_compat_version() < (3, 9) and name in _collections_abc.__all__): ... raise AttributeError(f'module {__name__!r} has no attribute {name!r}')

Deprecated open() "U" mode --------------------------

The "U" mode of ``open()`` is deprecated since Python 3.4 and emits a ``DeprecationWarning``. The `bpo-37330 https://bugs.python.org/issue37330`_ proposes to drop this mode: ``open()`` would raise an exception if ``U`` mode is used.

This change falls into the "cleanup" category: it is not required to implement a feature.

A backward compatibility mode would be trivial to implement and would be welcomed here by users.

Specification =============

sys functions -------------

Add 3 functions to the ``sys`` module:

* ``sys.set_python_compat_version(version)``: set the Python compatibility version. If it has been called previously, use the minimum of requested versions. Raise an exception if ``sys.set_python_min_compat_version(min_version)`` has been called and ``version < min_version``. *version* must be greater than or equal to ``(3, 0)``.

* ``sys.set_python_min_compat_version(min_version)``: set the **minimum** compatibility version. Raise an exception if ``sys.set_python_compat_version(old_version)`` has been called previously and ``old_version < min_version``. *min_version* must be greater than or equal to ``(3, 0)``.

* ``sys.get_python_compat_version()``: get the Python compatibility version. Return a ``tuple`` of 3 integers.

A *version* must a tuple of 2 or 3 integers. ``(major, minor)`` version is equivalent to ``(major, minor, 0)``.

By default, ``sys.get_python_compat_version()`` returns the current Python version.

Example to request compatibility with Python 3.8.0::

import collections

sys.set_python_compat_version((3, 8))

# collections.Mapping alias, removed from Python 3.9, is available # again, even if collections has been imported before calling # set_python_compat_version(). parent = collections.Mapping

Obviously, calling ``sys.set_python_compat_version(version)`` has no effect on code executed before the call. Use ``-X compat_version=VERSION`` command line option or ``PYTHONCOMPATVERSIONVERSION=VERSION`` environment variable to set the compatibility version at Python startup.

Command line ------------

Add ``-X compat_version=VERSION`` and ``-X min_compat_version=VERSION`` command line options: call respectivelly ``sys.set_python_compat_version()`` and ``sys.set_python_min_compat_version()``. ``VERSION`` is a version string with 2 or 3 numbers (``major.minor.micro`` or ``major.minor``). For example, ``-X compat_version=3.8`` calls ``sys.set_python_compat_version((3, 8))``.

Add ``PYTHONCOMPATVERSIONVERSION=VERSION`` and ``PYTHONCOMPATMINVERSION=VERSION=VERSION`` environment variables: call respectivelly ``sys.set_python_compat_version()`` and ``sys.set_python_min_compat_version()``. ``VERSION`` is a version string with the same format that the command line options.

Backwards Compatibility =======================

Introducing ``sys.set_python_compat_version()`` function means that an application will behave differently depending on the compatibility version. Moreover, since the version can be decreased multiple times, the application can behave differently depending on the import order.

Python 3.9 with ``sys.set_python_compat_version((3, 8))`` is not fully compatible with Python 3.8: the compatibility is only partial.

Security Implications =====================

``sys.set_python_compat_version()`` must not disable security fixes.

Alternatives ============

Provide a workaround for each incompatible change -------------------------------------------------

An application can works around most of the incompatible changes which impacts it.

For example, ``collections`` aliases can be added again using::

import collections.abc collections.Mapping = collections.abc.Mapping collections.Sequence = collections.abc.Sequence

Handle backward compatibility in the parser -------------------------------------------

The parser is modified to support multiple versions of the Python language (grammar).

The current Python parser cannot be easily modified for that. AST and grammar are hardcoded to a single Python version.

In Python 3.8, ``compile()`` has an undocumented ``_feature_version`` to not consider ``async`` and ``await`` as keywords.

The latest major language backward incompatible change was Python 3.7 which made ``async`` and ``await`` real keywords. It seems like Twisted was the only affected project, and Twisted had a single affected function (it used a parameter called ``async``).

Handling backward compatibility in the parser seems quite complex, not only to modify the parser, but also for developers who have to check which version of the Python language is used.

from __future__ import python38_syntax ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Add ``pythonXY_syntax`` to the ``__future__`` module. It would enable backward compatibility with Python X.Y syntax, but only for the current file.

With this option, there is no need to change ``sys.implementation.cache_tag`` to use a different ``.pyc`` filename, since the parser would always produce the same output for the same input (except of the optimization level).

Example::

from __future__ import python35_syntax

async = 1 await = 2

Update cache_tag ^^^^^^^^^^^^^^^^

Modify the parser to use ``sys.get_python_compat_version()`` to choose the version of the Python language.

``sys.set_python_compat_version()`` updates ``sys.implementation.cache_tag`` to include the compatibility version without the micro version as a suffix. For example, Python 3.9 uses ``'cpython-39'`` by default, but ``sys.set_python_compat_version((3, 7, 2))`` sets ``cache_tag`` to ``'cpython-39-37'``. Changes of the Python language are now allowed in micro releases.

One problem is that ``import asyncio`` is likely to fail if ``sys.set_python_compat_version((3, 6))`` has been called previously. The code of the ``asyncio`` module requires ``async`` and ``await`` to be real keywords (change done in Python 3.7).

Another problem is that regular users cannot write ``.pyc`` files into system directories, and so cannot create them on demand. It means that ``.pyc`` optimization cannot be used in the backward compatibility mode.

One solution for that is to modify the Python installer and Python package installers to precompile ``.pyc`` files not only for the current Python version, but also for multiple older Python versions (up to Python 3.0?).

Each ``.py`` file would have 3n ``.pyc`` files (3 optimization levels), where ``n`` is the number of supported Python versions. For example, it means 6 ``.pyc`` files, instead of 3, to support Python 3.8 and Python 3.9.

Temporary moratorium on incompatible changes --------------------------------------------

In 2009, the PEP 3003 "Python Language Moratorium" proposed to a temporary moratorium (suspension) of all changes to the Python language syntax, semantics, and built-ins for Python 3.1 and Python 3.2.

In May 2018, during PEP 572 discussions, it was also proposed to slow down Python changes: see the python-dev thread `Slow down... https://mail.python.org/archives/list/python-dev@python.org/thread/HHKRXOMRJ...`_

`Barry Warsaw's call on this https://mail.python.org/archives/list/python-dev@python.org/message/XR7IF2OB...`_:

I don’t believe that the way for Python to remain relevant and useful for the next 10 years is to cease all language evolution. Who knows what the computing landscape will look like in 5 years, let alone 10? Something as arbitrary as a 10 year moratorium is (again, IMHO) a death sentence for the language.

PEP 387 -------

`PEP 387 -- Backwards Compatibility Policy https://www.python.org/dev/peps/pep-0387/`_ proposes a process to make incompatible changes. The main point is the 4th step of the process:

See if there's any feedback. Users not involved in the original discussions may comment now after seeing the warning. Perhaps reconsider.

PEP 497 -------

`PEP 497 -- A standard mechanism for backward compatibility https://www.python.org/dev/peps/pep-0497/`_ proposes different solutions to provide backward compatibility.

Except of the ``__past__`` mechanism idea, the PEP 497 does not propose concrete solutions:

When an incompatible change to core language syntax or semantics is being made, Python-dev's policy is to prefer and expect that, wherever possible, a mechanism for backward compatibility be considered and provided for future Python versions after the breaking change is adopted by default, in addition to any mechanisms proposed for forward compatibility such as new future_statements.

Examples of incompatible changes ================================

Python 3.8 ----------

Examples of Python 3.8 incompatible changes:

* (During beta phase) ``PyCode_New()`` required a new parameter: it broke all Cython extensions (all projects distributing precompiled Cython code). This change has been reverted during the 3.8 beta phase and a new ``PyCode_NewWithPosOnlyArgs()`` function was added instead.

* ``types.CodeType`` requires an additional mandatory parameter. The ``CodeType.replace()`` function was added to help projects to no longer depend on the exact signature of the ``CodeType`` constructor.

* C extensions are no longer linked to libpython.

* ``sys.abiflags`` changed from ``'m'`` to an empty string. For example, ``python3.8m`` program is gone.

* The C structure ``PyInterpreterState`` was made opaque.

* Blender:

* https://bugzilla.redhat.com/show_bug.cgi?id=1734980#c6 * https://developer.blender.org/D6038

* XML attribute order: `bpo-34160 https://bugs.python.org/issue34160`_. Broken projects:

* `coverage https://bugs.python.org/issue34160#msg329612`_ * `docutils https://sourceforge.net/p/docutils/bugs/359/`_ * `pcs https://bugzilla.redhat.com/show_bug.cgi?id=1705475`_ * `python-glyphsLib https://bugzilla.redhat.com/show_bug.cgi?id=1705391`_

Backward compatibility cannot be added for all these changes. For example, changes in the C API and in the build system are out of the scope of this PEP.

See `What’s New In Python 3.8: API and Feature Removals https://docs.python.org/dev/whatsnew/3.8.html#api-and-feature-removals`_ for all changes.

See also the `Porting to Python 3.8 https://docs.python.org/dev/whatsnew/3.8.html#porting-to-python-3-8`_ section of What’s New In Python 3.8.

Python 3.7 ----------

Examples of Python 3.7 incompatible changes:

* ``async`` and ``await`` are now reserved keywords. * Several undocumented internal imports were removed. One example is that ``os.errno`` is no longer available; use ``import errno`` directly instead. Note that such undocumented internal imports may be removed any time without notice, even in micro version releases. * Unknown escapes consisting of ``'\'`` and an ASCII letter in replacement templates for ``re.sub()`` were deprecated in Python 3.5, and will now cause an error. * The ``asyncio.windows_utils.socketpair()`` function has been removed: it was an alias to ``socket.socketpair()``. * ``asyncio`` no longer exports the ``selectors`` and ``_overlapped`` modules as ``asyncio.selectors`` and ``asyncio._overlapped``. Replace ``from asyncio import selectors`` with ``import selectors``. * PEP 479 is enabled for all code in Python 3.7, meaning that ``StopIteration`` exceptions raised directly or indirectly in coroutines and generators are transformed into ``RuntimeError`` exceptions. * ``socketserver.ThreadingMixIn.server_close()`` now waits until all non-daemon threads complete. Set the new ``block_on_close`` class attribute to ``False`` to get the pre-3.7 behaviour. * The ``struct.Struct.format`` type is now ``str`` instead of ``bytes``. * ``repr`` for ``datetime.timedelta`` has changed to include the keyword arguments in the output. * ``tracemalloc.Traceback`` frames are now sorted from oldest to most recent to be more consistent with ``traceback``.

Adding backward compatibility for most of these changes would be easy.

See also the `Porting to Python 3.7 https://docs.python.org/dev/whatsnew/3.7.html#porting-to-python-3-7`_ section of What’s New In Python 3.7.

Micro releases --------------

Sometimes, incompatible changes are introduced in micro releases (``micro`` in ``major.minor.micro``) to fix bugs or security vulnerabilities. Examples:

* Python 3.7.2, ``compileall`` and ``py_compile`` module: the *invalidation_mode* parameter's default value is updated to ``None``; the ``SOURCE_DATE_EPOCH`` environment variable no longer overrides the value of the *invalidation_mode* argument, and determines its default value instead.

* Python 3.7.1, ``xml`` modules: the SAX parser no longer processes general external entities by default to increase security by default.

* Python 3.5.2, ``os.urandom()``: on Linux, if the ``getrandom()`` syscall blocks (the urandom entropy pool is not initialized yet), fall back on reading ``/dev/urandom``.

* Python 3.5.1, ``sys.setrecursionlimit()``: a ``RecursionError`` exception is now raised if the new limit is too low at the current recursion depth.

* Python 3.4.4, ``ssl.create_default_context()``: RC4 was dropped from the default cipher string.

* Python 3.4.3, ``http.client``: ``HTTPSConnection`` now performs all the necessary certificate and hostname checks by default.

* Python 3.4.2, ``email.message``: ``EmailMessage.is_attachment()`` is now a method instead of a property, for consistency with ``Message.is_multipart()``.

* Python 3.4.1, ``os.makedirs(name, mode=0o777, exist_ok=False)``: Before Python 3.4.1, if *exist_ok* was ``True`` and the directory existed, ``makedirs()`` would still raise an error if *mode* did not match the mode of the existing directory. Since this behavior was impossible to implement safely, it was removed in Python 3.4.1 (`bpo-21082 https://bugs.python.org/issue21082`_).

Examples of changes made in micro releases which are not backward incompatible:

* ``ssl.OP_NO_TLSv1_3`` constant was added to 2.7.15, 3.6.3 and 3.7.0 for backwards compatibility with OpenSSL 1.0.2. * ``typing.AsyncContextManager`` was added to Python 3.6.2. * The ``zipfile`` module accepts a path-like object since Python 3.6.2. * ``loop.create_future()`` was added to Python 3.5.2 in the ``asyncio`` module.

No backward compatibility code is needed for such kind of changes.

References ==========

Accepted PEPs:

* `PEP 5 -- Guidelines for Language Evolution https://www.python.org/dev/peps/pep-0005/`_ * `PEP 236 -- Back to the __future__ https://www.python.org/dev/peps/pep-0236/`_ * `PEP 411 -- Provisional packages in the Python standard library https://www.python.org/dev/peps/pep-0411/`_ * `PEP 3002 -- Procedure for Backwards-Incompatible Changes https://www.python.org/dev/peps/pep-3002/`_

Draft PEPs:

* `PEP 602 -- Annual Release Cycle for Python https://www.python.org/dev/peps/pep-0602/`_ * `PEP 605 -- A rolling feature release stream for CPython https://www.python.org/dev/peps/pep-0605/`_ * See also withdrawn `PEP 598 -- Introducing incremental feature releases https://www.python.org/dev/peps/pep-0598/`_

Copyright =========

This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive.

.. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: