Le mar. 23 juin 2020 à 15:56, Petr Viktorin encukou@gmail.com a écrit :
Adding or removing members of C structures is causing multiple backward compatibility issues.
Adding a new member breaks the stable ABI (PEP 384), especially for types declared statically (e.g. ``static PyTypeObject MyType = {...};``).
PyTypeObject is explicitly not part of the stable ABI, see PEP 384: https://www.python.org/dev/peps/pep-0384/#structures I don't know why Py_TPFLAGS_HAVE_FINALIZE was added, but it wasn't for the PEP 384 stable ABI.
Maybe Antoine Pitrou knows the rationale why Py_TPFLAGS_HAVE_FINALIZE flag was added. Removing the flag was discussed at:
* https://bugs.python.org/issue32388 * https://mail.python.org/pipermail/python-dev/2017-December/151328.html
Summary
- (**Completed**) Reorganize the C API header files: create
``Include/cpython/`` and ``Include/internal/`` subdirectories.
- (**Completed**) Move private functions exposing implementation
details to the internal C API.
- (**Completed**) Convert macros to static inline functions.
- (**Completed**) Add new functions ``Py_SET_TYPE()``, ``Py_SET_REFCNT()`` and ``Py_SET_SIZE()``. The ``Py_TYPE()``, ``Py_REFCNT()`` and ``Py_SIZE()`` macros become functions which cannot be used as l-value.
- (**Completed**) New C API functions must not return borrowed references.
- (**In Progress**) Provide ``pythoncapi_compat.h`` header file.
- (**In Progress**) Make structures opaque, add getter and setter functions.
- (**Not Started**) Deprecate ``PySequence_Fast_ITEMS()``.
- (**Not Started**) Convert ``PyTuple_GET_ITEM()`` and ``PyList_GET_ITEM()`` macros to static inline functions.
What does **Completed** mean? All work in that category is done, and any new changes in the category will require a new PEP (or a change to this PEP)?
The purpose of the PEP is to show an overview of the work already done towards the same goal. Changes were scattered in multiple issues and so I don't think that it was easy to follow it.
The PEP cannot be exhaustive, hiding implementation details is a work-in-progress.
Usually, once a PEP is approved, it should not be modified.
I don't think that minor incompatible changes should require a new whole PEP.
The purpose of the PEP is to define an overall process for past and future changes in this area, and also announces our willingness for this (hide implementation details).
If tomorrow, an incompatible C API change becomes very controversial, maybe another PEP will be justified.
So far, incompatible C API changes were done with no PEP. I decided to write a PEP for the Py_TYPE() change: disallow to use it as an l-value, that's one of the "most" incompatible changes since I started to work on this.
It seems that this PEP is largely asking for forgiveness, so trying to find better ways to solve the problems would be a waste of time at this point.
Most completed changes were already discussed during physical meetings, like Pycons and Core dev sprints. From what I recall, everybody (involved in the discussion) agreed on the solution.
But there is always room for enhancements! The purpose of the PEP is also to make the work already done more visible, so it's easier to reason about the overall picture.
Feel free to propose your ideas, it's exactly the role of a discussion on a PEP!
Make structures opaque
All structures of the C API should become opaque: C extensions must use getter or setter functions to get or set structure members. For example, ``tuple->ob_item[0]`` must be replaced with ``PyTuple_GET_ITEM(tuple, 0)``.
All structures? I don't think we can make PyModuleDef opaque, for example. PEP 384 lists more structures, some of which I don't think should be opaque: https://www.python.org/dev/peps/pep-0384/#structures
Oh, no, PyModuleDef is fine. I would like to make the following structures opaque:
* PyInterpreterState * PyThreadState * PyGC_Head * PyTypeObject * PyObject and PyVarObject * PyTypeObject * All types which inherit from PyObject or PyVarObject
But I'm not sure about the exact list. Let's start with this list :-)
Process to reduce the number of broken C extensions
Process to reduce the number of broken C extensions when introducing C API incompatible changes listed in this PEP:
- Estimate how many popular C extensions are affected by the incompatible change.
- Coordinate with maintainers of broken C extensions to prepare their code for the future incompatible change.
- Introduce the incompatible changes in Python. The documentation must explain how to port existing code. It is recommended to merge such changes at the beginning of a development cycle to have more time for tests.
- Changes which are the most likely to break a large number of C extensions should be announced on the capi-sig mailing list to notify C extensions maintainers to prepare their project for the next Python.
- If the change breaks too many projects, reverting the change should be discussed, taking in account the number of broken packages, their importance in the Python community, and the importance of the change.
What is "popular"? What is "too many"? Who decides?
The process is underspecified on purpose. I don't know that it's possible to define strict limits.
For me the important part is discussing incompatible changes and trying to reach a consensus.
Future incompatible changes can be announced by deprecating a function in the documentation and by annotating the function with ``Py_DEPRECATED()``.
How long should functions be deprecated before being removed?
In the whole PEP 620, there is a single function which is deprecated: PySequence_Fast_ITEMS().
Note: It seems like Brett and Benjamin wants to require to have a deprecation for 2 releases before being able to remove a function: https://discuss.python.org/t/pep-387-backwards-compatibilty-policy/4421
The PEP doesn't plan PySequence_Fast_ITEMS() removal, but I guess that it should follow PEP 387. If it's deprecated in 3.10, its removal will happen in Python 3.12.
But making a structure opaque and preventing the usage of a macro as l-value cannot be deprecated with ``Py_DEPRECATED()``.
So, how should things like this be deprecated? And for how long?
These changes don't go through any deprecation process. When you build a C extension affected by these incompatible changes, you get a compilation error.
The important part is coordination and finding a balance between CPython evolutions and backward compatibility. For example, breaking a random, old, obscure and unmaintained C extension on PyPI is less severe than breaking numpy.
How will this balance be found?
By discussing the change.
If a change is reverted, we move back to the coordination step to better prepare the change. Once more C extensions are ready, the incompatible change can be reconsidered.
When in the process should it be reverted?
Whenever you want. If there are too many complaints about an incompatible change during the beta phase and we decide to revert the change, the important part is to revert before the 3.x.0 final release.
If the complaints only come after the final release, it's usually too late and it's better to learn how to live with this change.
Do you think that these aspects should be clearly specified in the PEP?
Victor