PEP 674: Disallow using macros as l-value
Hi, I propose to disallow using macros as l-value. Read and comment the plain text below, or read the PEP 674 online, https://python.github.io/peps/pep-0674/ While I'm not sure that the proposed changes are really controversial, I decided to write a formal PEP since the incompatible changes are not following the PEP 387 deprecation process and so a format PEP 387 exception is better. Well, the list of modified macros is also quite long. Moreover, a PEP is a way to document and announce the changes ;-) The Py_TYPE() and Py_SIZE() changes are already approved the Steering Council, but I prefer to list them in the PEP: https://github.com/python/steering-council/issues/79 In practice, I'm only aware of 4 projects impacted by these changes, and I wrote the pythoncapi_compat project which updates automatically C extensions: add Python 3.11 support, without losing support for older Python versions. I already prepared major projects like Cython and numpy for these changes (in total, 14 impacted projects have already been updated). Victor --- PEP: 674 Title: Disallow using macros as l-value Author: Victor Stinner <vstinner@python.org> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 30-Oct-2021 Python-Version: 3.11 Abstract ======== Incompatible C API change disallowing using macros as l-value to allow evolving CPython internals and to ease the C API implementation on other Python implementation. In practice, the majority of projects impacted by these incompatible changes should only have to make two changes: * Replace ``Py_TYPE(obj) = new_type;`` with ``Py_SET_TYPE(obj, new_type);``. * Replace ``Py_SIZE(obj) = new_size;`` with ``Py_SET_SIZE(obj, new_size);``. Rationale ========= Using a macro as a l-value -------------------------- In the Python C API, some functions are implemented as macro because writing a macro is simpler than writing a regular function. If a macro exposes directly a struture member, it is technically possible to use this macro to not only get the structure member but also set it. Example with the Python 3.10 ``Py_TYPE()`` macro:: #define Py_TYPE(ob) (((PyObject *)(ob))->ob_type) This macro can be used as a **r-value** to **get** an object type:: type = Py_TYPE(object); It can also be used as **l-value** to **set** an object type:: Py_TYPE(object) = new_type; It is also possible to set an object reference count and an object size using ``Py_REFCNT()`` and ``Py_SIZE()`` macros. Setting directly an object attribute relies on the current exact CPython implementation. Implementing this feature in other Python implementations can make their C API implementation less efficient. CPython nogil fork ------------------ Sam Gross forked Python 3.8 to remove the GIL: the `nogil branch <https://github.com/colesbury/nogil/>`_. This fork has no ``PyObject.ob_refcnt`` member, but a more elaborated implementation for reference counting, and so the ``Py_REFCNT(obj) = new_refcnt;`` code fails with a compiler error. Merging the nogil fork into the upstream CPython main branch requires first to fix this C API compatibility issue. It is a concrete example of a Python optimization blocked indirectly by the C API. This issue was already fixed in Python 3.10: the ``Py_REFCNT()`` macro has been already modified to disallow using it as a l-value. HPy project ----------- The `HPy project <https://hpyproject.org/>`_ is a brand new C API for Python using only handles and function calls: handles are opaque, structure members cannot be accessed directly,and pointers cannot be dereferenced. Disallowing the usage of macros as l-value helps the migration of existing C extensions to HPy by reducing differences between the C API and the HPy API. PyPy cpyext module ------------------ In PyPy, when a Python object is accessed by the Python C API, the PyPy ``cpyext`` module has to convert PyPy object to a CPython object. While PyPy objects are designed to be efficient with the PyPy JIT compiler, CPython objects are less efficient and increase the memory usage. This PEP alone is not enough to get rid of the CPython objects in the PyPy ``cpyext`` module, but it is a step towards this long term goal. PyPy already supports HPy which is a better solution in the long term. Specification ============= Disallow using macros as l-value -------------------------------- PyObject and PyVarObject macros ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ * ``Py_TYPE()``: ``Py_SET_TYPE()`` must be used instead * ``Py_SIZE()``: ``Py_SET_SIZE()`` must be used instead "GET" macros ^^^^^^^^^^^^ * ``PyByteArray_GET_SIZE()`` * ``PyBytes_GET_SIZE()`` * ``PyCFunction_GET_CLASS()`` * ``PyCFunction_GET_FLAGS()`` * ``PyCFunction_GET_FUNCTION()`` * ``PyCFunction_GET_SELF()`` * ``PyCell_GET()`` * ``PyCode_GetNumFree()`` * ``PyDict_GET_SIZE()`` * ``PyFunction_GET_ANNOTATIONS()`` * ``PyFunction_GET_CLOSURE()`` * ``PyFunction_GET_CODE()`` * ``PyFunction_GET_DEFAULTS()`` * ``PyFunction_GET_GLOBALS()`` * ``PyFunction_GET_KW_DEFAULTS()`` * ``PyFunction_GET_MODULE()`` * ``PyHeapType_GET_MEMBERS()`` * ``PyInstanceMethod_GET_FUNCTION()`` * ``PyList_GET_SIZE()`` * ``PyMemoryView_GET_BASE()`` * ``PyMemoryView_GET_BUFFER()`` * ``PyMethod_GET_FUNCTION()`` * ``PyMethod_GET_SELF()`` * ``PySet_GET_SIZE()`` * ``PyTuple_GET_SIZE()`` * ``PyUnicode_GET_DATA_SIZE()`` * ``PyUnicode_GET_LENGTH()`` * ``PyUnicode_GET_LENGTH()`` * ``PyUnicode_GET_SIZE()`` * ``PyWeakref_GET_OBJECT()`` "AS" macros ^^^^^^^^^^^ * ``PyByteArray_AS_STRING()`` * ``PyBytes_AS_STRING()`` * ``PyFloat_AS_DOUBLE()`` * ``PyUnicode_AS_DATA()`` * ``PyUnicode_AS_UNICODE()`` PyUnicode macros ^^^^^^^^^^^^^^^^ * ``PyUnicode_1BYTE_DATA()`` * ``PyUnicode_2BYTE_DATA()`` * ``PyUnicode_4BYTE_DATA()`` * ``PyUnicode_DATA()`` * ``PyUnicode_IS_ASCII()`` * ``PyUnicode_IS_COMPACT()`` * ``PyUnicode_IS_READY()`` * ``PyUnicode_KIND()`` * ``PyUnicode_READ()`` * ``PyUnicode_READ_CHAR()`` PyDateTime "GET" macros ^^^^^^^^^^^^^^^^^^^^^^^ * ``PyDateTime_DATE_GET_FOLD()`` * ``PyDateTime_DATE_GET_HOUR()`` * ``PyDateTime_DATE_GET_MICROSECOND()`` * ``PyDateTime_DATE_GET_MINUTE()`` * ``PyDateTime_DATE_GET_SECOND()`` * ``PyDateTime_DATE_GET_TZINFO()`` * ``PyDateTime_DELTA_GET_DAYS()`` * ``PyDateTime_DELTA_GET_MICROSECONDS()`` * ``PyDateTime_DELTA_GET_SECONDS()`` * ``PyDateTime_GET_DAY()`` * ``PyDateTime_GET_MONTH()`` * ``PyDateTime_GET_YEAR()`` * ``PyDateTime_TIME_GET_FOLD()`` * ``PyDateTime_TIME_GET_HOUR()`` * ``PyDateTime_TIME_GET_MICROSECOND()`` * ``PyDateTime_TIME_GET_MINUTE()`` * ``PyDateTime_TIME_GET_SECOND()`` * ``PyDateTime_TIME_GET_TZINFO()`` PyDescr macros ^^^^^^^^^^^^^^ * ``PyDescr_NAME()`` * ``PyDescr_TYPE()`` Port C extensions to Python 3.11 -------------------------------- In practice, the majority of projects impacted by these PEP incompatible changes should only have to make two changes: * Replace ``Py_TYPE(obj) = new_type;`` with ``Py_SET_TYPE(obj, new_type);``. * Replace ``Py_SIZE(obj) = new_size;`` with ``Py_SET_SIZE(obj, new_size);``. The `pythoncapi_compat project <https://github.com/pythoncapi/pythoncapi_compat>`_ can be used to update automatically C extensions: add Python 3.11 support without losing support with older Python versions. The project provides a header file which provides ``Py_SET_REFCNT()``, ``Py_SET_TYPE()`` and ``Py_SET_SIZE()`` functions to Python 3.8 and older. PyTuple_GET_ITEM() and PyList_GET_ITEM() ---------------------------------------- The ``PyTuple_GET_ITEM()`` and ``PyList_GET_ITEM()`` macros are left unchanged. The code pattern ``&PyTuple_GET_ITEM(tuple, 0)`` and ``&PyList_GET_ITEM(list, 0)`` is still commonly used to get access to the inner ``PyObject**`` array. Changing these macros would require to add a new API to get access to the inner array which is out of the scope of this PEP. Backwards Compatibility ======================= The proposed C API changes are backward incompatible on purpose. In practice, only a minority of third party projects are affected (16 projects are known to be broken) and `most of them have already been updated for these changes <https://bugs.python.org/issue39573#msg401378>`__ (12 on 16). Most projects are broken by ``Py_TYPE()`` and ``Py_SIZE()`` changes. These two macros have been converted to static inline macro in Python 3.10 alpha versions, but the change had to be reverted since it broke too many projects. In the meanwhile, many projects, like Cython, have been prepared for this change by using ``Py_SET_TYPE()`` and ``Py_SET_SIZE()``. For example, projects using Cython only have to regenerate their outdated generated C code to become compatible. For the "GET" functions like ``PyDict_GET_SIZE()``, no project in the PyPI top 5000 projects use these functions as l-value. The ``PyFloat_AS_DOUBLE()`` function is not used as a l-value in the PyPI top 5000 projects. The ``PyBytes_AS_STRING()`` and ``PyByteArray_AS_STRING()`` are used as l-value but only to modify string characters, not to override the ``PyBytesObject.ob_sval`` or ``PyByteArrayObject.ob_start`` member. For example, Cython uses the following code which remains valid:: PyByteArray_AS_STRING(string)[i] = (char) v; This change does not follow the PEP 387 deprecation process. There is no known way to emit a deprecation warning when a macro is used as a l-value, but not when it's used differently (ex: r-value). Rejected Idea: Leave the macros as they are =========================================== The documentation of each function can discourage developers to use macros to modify Python objects. If these is a need to make an assignment, a setter function can be added and the macro documentation can require to use the setter function. For example, a ``Py_SET_TYPE()`` function has been added to Python 3.9 and the ``Py_TYPE()`` documentation now requires to use the ``Py_SET_TYPE()`` function to set an object type. If developers use macros as l-value, it's their responsibility when their code breaks, not the Python responsibility. We are operating under the consenting adults principle: we expect users of the Python C API to use it as documented and expect them to take care of the fallout, if things break when they don't. This idea was rejected because only few developers read the documentation, and only a minority is tracking changes of the Python C API documentation. The majority of developers are only using CPython and so are not aware of compatibility issues with other Python implementations. Moreover, continuing to allow using macros as l-value does not solve issues of the nogil, PyPy and HPy projects. Macros already modified ======================= The following C API macros have already been modified to disallow using them as l-value: * ``PyCell_SET()`` * ``PyList_SET_ITEM()`` * ``PyTuple_SET_ITEM()`` * ``Py_REFCNT()`` (Python 3.10): ``Py_SET_REFCNT()`` must be used * ``_PyGCHead_SET_FINALIZED()`` * ``_PyGCHead_SET_NEXT()`` * ``asdl_seq_GET()`` * ``asdl_seq_GET_UNTYPED()`` * ``asdl_seq_LEN()`` * ``asdl_seq_SET()`` * ``asdl_seq_SET_UNTYPED()`` For example, ``PyList_SET_ITEM(list, 0, item) < 0`` now fails with a compiler error as expected. References ========== * `Python C API: Add functions to access PyObject <https://vstinner.github.io/c-api-abstract-pyobject.html>`_ (October 2021) article by Victor Stinner * `[C API] Disallow using PyFloat_AS_DOUBLE() as l-value <https://bugs.python.org/issue45476>`_ (October 2021) * `[capi-sig] Py_TYPE() and Py_SIZE() become static inline functions <https://mail.python.org/archives/list/capi-sig@python.org/thread/WGRLTHTHC32DQTACPPX36TPR2GLJAFRB/>`_ (September 2021) * `[C API] Avoid accessing PyObject and PyVarObject members directly: add Py_SET_TYPE() and Py_IS_TYPE(), disallow Py_TYPE(obj)=type <https://bugs.python.org/issue39573>`__ (February 2020) * `bpo-30459: PyList_SET_ITEM could be safer <https://bugs.python.org/issue30459>`_ (May 2017) Copyright ========= This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive.
How about *not* asking for an exception and just following the PEP 387 process? Is that really too burdensome? On Tue, Nov 30, 2021 at 10:30 AM Victor Stinner <vstinner@python.org> wrote: > Hi, > > I propose to disallow using macros as l-value. Read and comment the > plain text below, or read the PEP 674 online, > https://python.github.io/peps/pep-0674/ > > While I'm not sure that the proposed changes are really controversial, > I decided to write a formal PEP since the incompatible changes are not > following the PEP 387 deprecation process and so a format PEP 387 > exception is better. Well, the list of modified macros is also quite > long. Moreover, a PEP is a way to document and announce the changes > ;-) > > The Py_TYPE() and Py_SIZE() changes are already approved the Steering > Council, but I prefer to list them in the PEP: > https://github.com/python/steering-council/issues/79 > > In practice, I'm only aware of 4 projects impacted by these changes, > and I wrote the pythoncapi_compat project which updates automatically > C extensions: add Python 3.11 support, without losing support for > older Python versions. I already prepared major projects like Cython > and numpy for these changes (in total, 14 impacted projects have > already been updated). > > Victor > > --- > > PEP: 674 > Title: Disallow using macros as l-value > Author: Victor Stinner <vstinner@python.org> > Status: Draft > Type: Standards Track > Content-Type: text/x-rst > Created: 30-Oct-2021 > Python-Version: 3.11 > > Abstract > ======== > > Incompatible C API change disallowing using macros as l-value to allow > evolving CPython internals and to ease the C API implementation on other > Python implementation. > > In practice, the majority of projects impacted by these incompatible > changes should only have to make two changes: > > * Replace ``Py_TYPE(obj) = new_type;`` > with ``Py_SET_TYPE(obj, new_type);``. > * Replace ``Py_SIZE(obj) = new_size;`` > with ``Py_SET_SIZE(obj, new_size);``. > > > Rationale > ========= > > Using a macro as a l-value > -------------------------- > > In the Python C API, some functions are implemented as macro because > writing a macro is simpler than writing a regular function. If a macro > exposes directly a struture member, it is technically possible to use > this macro to not only get the structure member but also set it. > > Example with the Python 3.10 ``Py_TYPE()`` macro:: > > #define Py_TYPE(ob) (((PyObject *)(ob))->ob_type) > > This macro can be used as a **r-value** to **get** an object type:: > > type = Py_TYPE(object); > > It can also be used as **l-value** to **set** an object type:: > > Py_TYPE(object) = new_type; > > It is also possible to set an object reference count and an object size > using ``Py_REFCNT()`` and ``Py_SIZE()`` macros. > > Setting directly an object attribute relies on the current exact CPython > implementation. Implementing this feature in other Python > implementations can make their C API implementation less efficient. > > CPython nogil fork > ------------------ > > Sam Gross forked Python 3.8 to remove the GIL: the `nogil branch > <https://github.com/colesbury/nogil/>`_. This fork has no > ``PyObject.ob_refcnt`` member, but a more elaborated implementation for > reference counting, and so the ``Py_REFCNT(obj) = new_refcnt;`` code > fails with a compiler error. > > Merging the nogil fork into the upstream CPython main branch requires > first to fix this C API compatibility issue. It is a concrete example of > a Python optimization blocked indirectly by the C API. > > This issue was already fixed in Python 3.10: the ``Py_REFCNT()`` macro > has been already modified to disallow using it as a l-value. > > HPy project > ----------- > > The `HPy project <https://hpyproject.org/>`_ is a brand new C API for > Python using only handles and function calls: handles are opaque, > structure members cannot be accessed directly,and pointers cannot be > dereferenced. > > Disallowing the usage of macros as l-value helps the migration of > existing C extensions to HPy by reducing differences between the C API > and the HPy API. > > PyPy cpyext module > ------------------ > > In PyPy, when a Python object is accessed by the Python C API, the PyPy > ``cpyext`` module has to convert PyPy object to a CPython object. While > PyPy objects are designed to be efficient with the PyPy JIT compiler, > CPython objects are less efficient and increase the memory usage. > > This PEP alone is not enough to get rid of the CPython objects in the > PyPy ``cpyext`` module, but it is a step towards this long term goal. > PyPy already supports HPy which is a better solution in the long term. > > > Specification > ============= > > Disallow using macros as l-value > -------------------------------- > > PyObject and PyVarObject macros > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > > * ``Py_TYPE()``: ``Py_SET_TYPE()`` must be used instead > * ``Py_SIZE()``: ``Py_SET_SIZE()`` must be used instead > > "GET" macros > ^^^^^^^^^^^^ > > * ``PyByteArray_GET_SIZE()`` > * ``PyBytes_GET_SIZE()`` > * ``PyCFunction_GET_CLASS()`` > * ``PyCFunction_GET_FLAGS()`` > * ``PyCFunction_GET_FUNCTION()`` > * ``PyCFunction_GET_SELF()`` > * ``PyCell_GET()`` > * ``PyCode_GetNumFree()`` > * ``PyDict_GET_SIZE()`` > * ``PyFunction_GET_ANNOTATIONS()`` > * ``PyFunction_GET_CLOSURE()`` > * ``PyFunction_GET_CODE()`` > * ``PyFunction_GET_DEFAULTS()`` > * ``PyFunction_GET_GLOBALS()`` > * ``PyFunction_GET_KW_DEFAULTS()`` > * ``PyFunction_GET_MODULE()`` > * ``PyHeapType_GET_MEMBERS()`` > * ``PyInstanceMethod_GET_FUNCTION()`` > * ``PyList_GET_SIZE()`` > * ``PyMemoryView_GET_BASE()`` > * ``PyMemoryView_GET_BUFFER()`` > * ``PyMethod_GET_FUNCTION()`` > * ``PyMethod_GET_SELF()`` > * ``PySet_GET_SIZE()`` > * ``PyTuple_GET_SIZE()`` > * ``PyUnicode_GET_DATA_SIZE()`` > * ``PyUnicode_GET_LENGTH()`` > * ``PyUnicode_GET_LENGTH()`` > * ``PyUnicode_GET_SIZE()`` > * ``PyWeakref_GET_OBJECT()`` > > "AS" macros > ^^^^^^^^^^^ > > * ``PyByteArray_AS_STRING()`` > * ``PyBytes_AS_STRING()`` > * ``PyFloat_AS_DOUBLE()`` > * ``PyUnicode_AS_DATA()`` > * ``PyUnicode_AS_UNICODE()`` > > PyUnicode macros > ^^^^^^^^^^^^^^^^ > > * ``PyUnicode_1BYTE_DATA()`` > * ``PyUnicode_2BYTE_DATA()`` > * ``PyUnicode_4BYTE_DATA()`` > * ``PyUnicode_DATA()`` > * ``PyUnicode_IS_ASCII()`` > * ``PyUnicode_IS_COMPACT()`` > * ``PyUnicode_IS_READY()`` > * ``PyUnicode_KIND()`` > * ``PyUnicode_READ()`` > * ``PyUnicode_READ_CHAR()`` > > PyDateTime "GET" macros > ^^^^^^^^^^^^^^^^^^^^^^^ > > * ``PyDateTime_DATE_GET_FOLD()`` > * ``PyDateTime_DATE_GET_HOUR()`` > * ``PyDateTime_DATE_GET_MICROSECOND()`` > * ``PyDateTime_DATE_GET_MINUTE()`` > * ``PyDateTime_DATE_GET_SECOND()`` > * ``PyDateTime_DATE_GET_TZINFO()`` > * ``PyDateTime_DELTA_GET_DAYS()`` > * ``PyDateTime_DELTA_GET_MICROSECONDS()`` > * ``PyDateTime_DELTA_GET_SECONDS()`` > * ``PyDateTime_GET_DAY()`` > * ``PyDateTime_GET_MONTH()`` > * ``PyDateTime_GET_YEAR()`` > * ``PyDateTime_TIME_GET_FOLD()`` > * ``PyDateTime_TIME_GET_HOUR()`` > * ``PyDateTime_TIME_GET_MICROSECOND()`` > * ``PyDateTime_TIME_GET_MINUTE()`` > * ``PyDateTime_TIME_GET_SECOND()`` > * ``PyDateTime_TIME_GET_TZINFO()`` > > PyDescr macros > ^^^^^^^^^^^^^^ > > * ``PyDescr_NAME()`` > * ``PyDescr_TYPE()`` > > Port C extensions to Python 3.11 > -------------------------------- > > In practice, the majority of projects impacted by these PEP incompatible > changes should only have to make two changes: > > * Replace ``Py_TYPE(obj) = new_type;`` > with ``Py_SET_TYPE(obj, new_type);``. > * Replace ``Py_SIZE(obj) = new_size;`` > with ``Py_SET_SIZE(obj, new_size);``. > > The `pythoncapi_compat project > <https://github.com/pythoncapi/pythoncapi_compat>`_ can be used to > update automatically C extensions: add Python 3.11 support without > losing support with older Python versions. The project provides a header > file which provides ``Py_SET_REFCNT()``, ``Py_SET_TYPE()`` and > ``Py_SET_SIZE()`` functions to Python 3.8 and older. > > PyTuple_GET_ITEM() and PyList_GET_ITEM() > ---------------------------------------- > > The ``PyTuple_GET_ITEM()`` and ``PyList_GET_ITEM()`` macros are left > unchanged. > > The code pattern ``&PyTuple_GET_ITEM(tuple, 0)`` and > ``&PyList_GET_ITEM(list, 0)`` is still commonly used to get access to > the inner ``PyObject**`` array. > > Changing these macros would require to add a new API to get access to > the inner array which is out of the scope of this PEP. > > > Backwards Compatibility > ======================= > > The proposed C API changes are backward incompatible on purpose. In > practice, only a minority of third party projects are affected (16 > projects are known to be broken) and `most of them have already been > updated for these changes > <https://bugs.python.org/issue39573#msg401378>`__ (12 on 16). > > Most projects are broken by ``Py_TYPE()`` and ``Py_SIZE()`` changes. > These two macros have been converted to static inline macro in Python > 3.10 alpha versions, but the change had to be reverted since it broke > too many projects. In the meanwhile, many projects, like Cython, have > been prepared for this change by using ``Py_SET_TYPE()`` and > ``Py_SET_SIZE()``. For example, projects using Cython only have to > regenerate their outdated generated C code to become compatible. > > For the "GET" functions like ``PyDict_GET_SIZE()``, no project in the PyPI > top 5000 projects use these functions as l-value. > > The ``PyFloat_AS_DOUBLE()`` function is not used as a l-value in the > PyPI top 5000 projects. > > The ``PyBytes_AS_STRING()`` and ``PyByteArray_AS_STRING()`` are used as > l-value but only to modify string characters, not to override the > ``PyBytesObject.ob_sval`` or ``PyByteArrayObject.ob_start`` member. > For example, Cython uses the following code which remains valid:: > > PyByteArray_AS_STRING(string)[i] = (char) v; > > This change does not follow the PEP 387 deprecation process. There is no > known way to emit a deprecation warning when a macro is used as a > l-value, but not when it's used differently (ex: r-value). > > > Rejected Idea: Leave the macros as they are > =========================================== > > The documentation of each function can discourage developers to use > macros to modify Python objects. > > If these is a need to make an assignment, a setter function can be added > and the macro documentation can require to use the setter function. For > example, a ``Py_SET_TYPE()`` function has been added to Python 3.9 and > the ``Py_TYPE()`` documentation now requires to use the > ``Py_SET_TYPE()`` function to set an object type. > > If developers use macros as l-value, it's their responsibility when > their code breaks, not the Python responsibility. We are operating under > the consenting adults principle: we expect users of the Python C API to > use it as documented and expect them to take care of the fallout, if > things break when they don't. > > This idea was rejected because only few developers read the > documentation, and only a minority is tracking changes of the Python C > API documentation. The majority of developers are only using CPython and > so are not aware of compatibility issues with other Python > implementations. > > Moreover, continuing to allow using macros as l-value does not solve > issues of the nogil, PyPy and HPy projects. > > > Macros already modified > ======================= > > The following C API macros have already been modified to disallow using > them as l-value: > > * ``PyCell_SET()`` > * ``PyList_SET_ITEM()`` > * ``PyTuple_SET_ITEM()`` > * ``Py_REFCNT()`` (Python 3.10): ``Py_SET_REFCNT()`` must be used > * ``_PyGCHead_SET_FINALIZED()`` > * ``_PyGCHead_SET_NEXT()`` > * ``asdl_seq_GET()`` > * ``asdl_seq_GET_UNTYPED()`` > * ``asdl_seq_LEN()`` > * ``asdl_seq_SET()`` > * ``asdl_seq_SET_UNTYPED()`` > > For example, ``PyList_SET_ITEM(list, 0, item) < 0`` now fails with a > compiler error as expected. > > > References > ========== > > * `Python C API: Add functions to access PyObject > <https://vstinner.github.io/c-api-abstract-pyobject.html>`_ (October > 2021) article by Victor Stinner > * `[C API] Disallow using PyFloat_AS_DOUBLE() as l-value > <https://bugs.python.org/issue45476>`_ > (October 2021) > * `[capi-sig] Py_TYPE() and Py_SIZE() become static inline functions > < > https://mail.python.org/archives/list/capi-sig@python.org/thread/WGRLTHTHC32DQTACPPX36TPR2GLJAFRB/ > >`_ > (September 2021) > * `[C API] Avoid accessing PyObject and PyVarObject members directly: > add Py_SET_TYPE() and Py_IS_TYPE(), disallow Py_TYPE(obj)=type > <https://bugs.python.org/issue39573>`__ (February 2020) > * `bpo-30459: PyList_SET_ITEM could be safer > <https://bugs.python.org/issue30459>`_ (May 2017) > > > Copyright > ========= > > This document is placed in the public domain or under the > CC0-1.0-Universal license, whichever is more permissive. > _______________________________________________ > Python-Dev mailing list -- python-dev@python.org > To unsubscribe send an email to python-dev-leave@python.org > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at > https://mail.python.org/archives/list/python-dev@python.org/message/KPIJPPJ6XVNOLGZQD2PFGMT7LBJMTTCO/ > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
On Tue, Nov 30, 2021 at 7:34 PM Guido van Rossum <guido@python.org> wrote:
How about *not* asking for an exception and just following the PEP 387 process? Is that really too burdensome?
The Backward Compatibility section gives an explanation: "This change does not follow the PEP 387 deprecation process. There is no known way to emit a deprecation warning when a macro is used as a l-value, but not when it's used differently (ex: r-value)." Apart of compiler warnings, one way to implement the PEP 387 "deprecation process" would be to announce the change in two "What's New in Python 3.X?" documents. But I expect that it will not be efficient. Extract of the Rejected Idea section: "(...) only few developers read the documentation, and only a minority is tracking changes of the Python C API documentation." In my experience, even if a DeprecationWarning is emitted at runtime, developers miss or ignore it. See the recent "[Python-Dev] Do we need to remove everything that's deprecated?" discussion and complains about recent removal of deprecated features, like: * collections.MutableMapping was deprecated for 7 Python versions (deprecated in 3.3) -- removed in 3.9 alpha, reverted in 3.9 beta, removed again in 3.11 * the "U" open() flag was deprecated for 10 Python versions (deprecated in 3.0) -- removed in 3.9 alpha, reverted in 3.9 beta, removed again in 3.11 For this specific PEP changes, I consider that the number of impacted projects is low enough to skip a deprecation process: only 4 projects are known to be impacted. One year ago (Python 3.10), 16 were impacted, and 12 have already been updated in the meanwhile. I'm talking especially about Py_TYPE() and Py_SIZE() changes which, again, has been approved by the Steering Council. Victor
On Tue, Nov 30, 2021 at 7:52 PM Victor Stinner <vstinner@python.org> wrote:
For this specific PEP changes, I consider that the number of impacted projects is low enough to skip a deprecation process: only 4 projects are known to be impacted. One year ago (Python 3.10), 16 were impacted, and 12 have already been updated in the meanwhile.
I ran a code search in PyPI top 5000 projects: my tool downloaded 4760 tarball and ZIP archives. On 4760 projects, only 16 of them (0.3%) are affected by the PEP 674: 99.7% (4744) are not affected ;-) IMO it's manageable to help these 16 projects to be updated for Python 3.11 (if the PEP is accepted) before the final version (scheduled for 2022-10-03). I expected that projects would only be affected by Py_TYPE() and Py_SIZE() changes. But I also found exactly 2 projects affected by the PyDescr_TYPE() and PyDescr_NAME() changes: code generated by SWIG in M2Crypto and mecab-python3 projects. I will investigate how to update SWIG for that. More details in the issue: https://bugs.python.org/issue45476#msg407410 Victor -- Night gathers, and now my watch begins. It shall not end until my death.
On 30. 11. 21 19:52, Victor Stinner wrote:
On Tue, Nov 30, 2021 at 7:34 PM Guido van Rossum <guido@python.org> wrote:
How about *not* asking for an exception and just following the PEP 387 process? Is that really too burdensome?
The Backward Compatibility section gives an explanation:
"This change does not follow the PEP 387 deprecation process. There is no known way to emit a deprecation warning when a macro is used as a l-value, but not when it's used differently (ex: r-value)."
Apart of compiler warnings, one way to implement the PEP 387 "deprecation process" would be to announce the change in two "What's New in Python 3.X?" documents. But I expect that it will not be efficient. Extract of the Rejected Idea section:
"(...) only few developers read the documentation, and only a minority is tracking changes of the Python C API documentation."
In my experience, even if a DeprecationWarning is emitted at runtime, developers miss or ignore it. See the recent "[Python-Dev] Do we need to remove everything that's deprecated?" discussion and complains about recent removal of deprecated features, like:
* collections.MutableMapping was deprecated for 7 Python versions (deprecated in 3.3) -- removed in 3.9 alpha, reverted in 3.9 beta, removed again in 3.11 * the "U" open() flag was deprecated for 10 Python versions (deprecated in 3.0) -- removed in 3.9 alpha, reverted in 3.9 beta, removed again in 3.11
For this specific PEP changes, I consider that the number of impacted projects is low enough to skip a deprecation process: only 4 projects are known to be impacted. One year ago (Python 3.10), 16 were impacted, and 12 have already been updated in the meanwhile. I'm talking especially about Py_TYPE() and Py_SIZE() changes which, again, has been approved by the Steering Council.
The current version of the PEP looks nice, but I don't think the rationale is strong enough. I believe we should: - Mark the l-value usage as deprecated in the docs, - And then do nothing until we find an actual case where this issue blocks development (or is actively dangerous for users). Specifically, for each of the Rationale parts: ## Using a macro as a l-value The practice was often discouraged (e.g. by GET in many of the names), or even relatively widely and successfully used (e.g. Py_TYPE before Python 3.9). If we would deprecate using Py_REFCNT as l-value in the docs, but wait with the conversion until it was *actually* needed, we would not lose anything: - Users would *still* have no period of visible compiler warnings or DeprecationWarning. - There would be more time for users to react to the documentation warning. Or even come up with a linter, or try compiling their favorite extensions with HPy/nogil and fixing the issues *on their own schedule*. ## CPython nogil fork In CPython, we cannot change structs that are part of the stable ABI -- such as PyObject.ob_refcnt. IMO, getting rid of the macros that access ob_refcnt is a relatively small part of this issue. AFAICS, the technical change is trivial compared to nogil, and can be easily made in the nogil fork -- if it is actually necessary in the end. ## HPy project There is no reason for "Searching and replacing Py_SET_SIZE()". If this change was not made, and Py_SIZE was semi-mechanically replaced by HPy_Length(), then misuses of Py_SIZE (using is as l-value) can be detected very easily: just compile against HPy. And if the autoreplacer is smart enough to see when it should use HPyTupleBuilder/HPyListBuilder, then it can surely see when Py_SIZE is (mis)used as l-value! There will always be some changes necessary when porting extensions to HPy. CPython should definitely indicate what the best practice is, so that HPY adopters have an easy time convincing projects to take their pull requests -- but it should not break code for people who don't care about HPy yet. ## GraalVM Python This PEP is not enough to get rid of wrappers in GraalVM, yet it forces users of CPython to adapt. Is it a part of a *plan* to remove wrappers, or just a minor step in what looks like the general direction? I do agree this PEP looks like a good step towards a long-term goal. But even so, it should be made when it *actually benefits* existing users, or allows some concrete good thing -- an optimization, a more maintainable implementation, something that outweighs the need for churn in code that worked up to now. Overall, the Rationale seems hollow to me. It's breaking existing code -- however bad that code is -- in the name of ideals rather than concrete improvements. Until disallowing macros as l-values allows concrete improvements in CPython, it should be the job of linters. FWIW, I do encourage alternative implementations to just not support l-value macros. There are only few projects doing this, and the fix is often (but not always!) easy. This should be a very small part of porting something to a different Python implementation (but I could definitely be wrong here, please correct me).
On Tue, 7 Dec 2021 15:39:25 +0100 Petr Viktorin <encukou@gmail.com> wrote:
On 30. 11. 21 19:52, Victor Stinner wrote:
On Tue, Nov 30, 2021 at 7:34 PM Guido van Rossum <guido@python.org> wrote:
How about *not* asking for an exception and just following the PEP 387 process? Is that really too burdensome?
The Backward Compatibility section gives an explanation:
"This change does not follow the PEP 387 deprecation process. There is no known way to emit a deprecation warning when a macro is used as a l-value, but not when it's used differently (ex: r-value)."
Apart of compiler warnings, one way to implement the PEP 387 "deprecation process" would be to announce the change in two "What's New in Python 3.X?" documents. But I expect that it will not be efficient. Extract of the Rejected Idea section:
"(...) only few developers read the documentation, and only a minority is tracking changes of the Python C API documentation."
In my experience, even if a DeprecationWarning is emitted at runtime, developers miss or ignore it. See the recent "[Python-Dev] Do we need to remove everything that's deprecated?" discussion and complains about recent removal of deprecated features, like:
* collections.MutableMapping was deprecated for 7 Python versions (deprecated in 3.3) -- removed in 3.9 alpha, reverted in 3.9 beta, removed again in 3.11 * the "U" open() flag was deprecated for 10 Python versions (deprecated in 3.0) -- removed in 3.9 alpha, reverted in 3.9 beta, removed again in 3.11
For this specific PEP changes, I consider that the number of impacted projects is low enough to skip a deprecation process: only 4 projects are known to be impacted. One year ago (Python 3.10), 16 were impacted, and 12 have already been updated in the meanwhile. I'm talking especially about Py_TYPE() and Py_SIZE() changes which, again, has been approved by the Steering Council.
The current version of the PEP looks nice, but I don't think the rationale is strong enough. I believe we should: - Mark the l-value usage as deprecated in the docs, - And then do nothing until we find an actual case where this issue blocks development (or is actively dangerous for users).
Is there a way to emit a compilation warning when those macros are used as l-values? Even if only enabled on some compilers. Regards Antoine.
Sorry for stepping in - but I am seeing too many arguments in favour of the rules because "they are the rules", and just Victor arguing with what is met in the "real world". But if this update can be done by a simple search/replace on the C source of projects, I can only perceive two scenarios this will affect: well maintained projects, for which it is fixable in minutes, and stale packages, no longer released that "happen to work" when someone downloads and builds for new Python versions. In these cases, the build will fail. If the person trying the build can't fix it, but can take the error to a proper, or high visibility, forum, someone will be able to come to the fix, leading to renewed visibility for the otherwise stale package. On Tue, 7 Dec 2021 at 12:40, Antoine Pitrou <antoine@python.org> wrote:
On Tue, 7 Dec 2021 15:39:25 +0100 Petr Viktorin <encukou@gmail.com> wrote:
On 30. 11. 21 19:52, Victor Stinner wrote:
On Tue, Nov 30, 2021 at 7:34 PM Guido van Rossum <guido@python.org> wrote:
How about *not* asking for an exception and just following the PEP 387 process? Is that really too burdensome?
The Backward Compatibility section gives an explanation:
"This change does not follow the PEP 387 deprecation process. There is no known way to emit a deprecation warning when a macro is used as a l-value, but not when it's used differently (ex: r-value)."
Apart of compiler warnings, one way to implement the PEP 387 "deprecation process" would be to announce the change in two "What's New in Python 3.X?" documents. But I expect that it will not be efficient. Extract of the Rejected Idea section:
"(...) only few developers read the documentation, and only a minority is tracking changes of the Python C API documentation."
In my experience, even if a DeprecationWarning is emitted at runtime, developers miss or ignore it. See the recent "[Python-Dev] Do we need to remove everything that's deprecated?" discussion and complains about recent removal of deprecated features, like:
* collections.MutableMapping was deprecated for 7 Python versions (deprecated in 3.3) -- removed in 3.9 alpha, reverted in 3.9 beta, removed again in 3.11 * the "U" open() flag was deprecated for 10 Python versions (deprecated in 3.0) -- removed in 3.9 alpha, reverted in 3.9 beta, removed again in 3.11
For this specific PEP changes, I consider that the number of impacted projects is low enough to skip a deprecation process: only 4 projects are known to be impacted. One year ago (Python 3.10), 16 were impacted, and 12 have already been updated in the meanwhile. I'm talking especially about Py_TYPE() and Py_SIZE() changes which, again, has been approved by the Steering Council.
The current version of the PEP looks nice, but I don't think the rationale is strong enough. I believe we should: - Mark the l-value usage as deprecated in the docs, - And then do nothing until we find an actual case where this issue blocks development (or is actively dangerous for users).
Is there a way to emit a compilation warning when those macros are used as l-values? Even if only enabled on some compilers.
Regards
Antoine.
_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/YMMRRVVY... Code of Conduct: http://python.org/psf/codeofconduct/
On Tue, 2021-12-07 at 13:54 -0300, Joao S. O. Bueno wrote:
Sorry for stepping in - but I am seeing too many arguments in favour of the rules because "they are the rules", and just Victor arguing with what is met in the "real world".
But if this update can be done by a simple search/replace on the C source of projects, I can only perceive two scenarios this will affect: well maintained projects, for which it is fixable in minutes, and stale packages, no longer released that "happen to work" when someone downloads and builds for new Python versions. In these cases, the build will fail. If the person trying the build can't fix it, but can take the error to a proper, or high visibility, forum, someone will be able to come to the fix, leading to renewed visibility for the otherwise stale package.
The problem are really less-maintained projects that may miss it or take very long to react. And that may be very frustrating for their users (who will not have a work-around beyond patching the project!). So the question is whether these are so few users, that it is OK to break them (even many bug fixes will break someone, after all). The other consideration may be that documenting the change for a 1-2 years may achieve almost nothing except frustrating Victor ;). One thing we once did in NumPy (for a runtime problem), was to intentionally break everyone at pre-release/dev time to point out what code needed fixing. Then flip the switch back at release time as to not break production. After a long enough time we enabled it for release mode. Not saying that it was nice, but it was the only alternative would have been to never fix it. A similar switch could be worthwhile if it helps Victor with experimenting on the dev-branch or reach a useful amount of projects. Of course, I am not sure it would do either... Cheers, Sebastian
On Tue, 7 Dec 2021 at 12:40, Antoine Pitrou <antoine@python.org> wrote:
On Tue, 7 Dec 2021 15:39:25 +0100 Petr Viktorin <encukou@gmail.com> wrote:
On 30. 11. 21 19:52, Victor Stinner wrote:
On Tue, Nov 30, 2021 at 7:34 PM Guido van Rossum <guido@python.org> wrote:
How about *not* asking for an exception and just following the PEP 387 process? Is that really too burdensome?
The Backward Compatibility section gives an explanation:
"This change does not follow the PEP 387 deprecation process. There is no known way to emit a deprecation warning when a macro is used as a l-value, but not when it's used differently (ex: r-value)."
Apart of compiler warnings, one way to implement the PEP 387 "deprecation process" would be to announce the change in two "What's New in Python 3.X?" documents. But I expect that it will not be efficient. Extract of the Rejected Idea section:
"(...) only few developers read the documentation, and only a minority is tracking changes of the Python C API documentation."
In my experience, even if a DeprecationWarning is emitted at runtime, developers miss or ignore it. See the recent "[Python-Dev] Do we need to remove everything that's deprecated?" discussion and complains about recent removal of deprecated features, like:
* collections.MutableMapping was deprecated for 7 Python versions (deprecated in 3.3) -- removed in 3.9 alpha, reverted in 3.9 beta, removed again in 3.11 * the "U" open() flag was deprecated for 10 Python versions (deprecated in 3.0) -- removed in 3.9 alpha, reverted in 3.9 beta, removed again in 3.11
For this specific PEP changes, I consider that the number of impacted projects is low enough to skip a deprecation process: only 4 projects are known to be impacted. One year ago (Python 3.10), 16 were impacted, and 12 have already been updated in the meanwhile. I'm talking especially about Py_TYPE() and Py_SIZE() changes which, again, has been approved by the Steering Council.
The current version of the PEP looks nice, but I don't think the rationale is strong enough. I believe we should: - Mark the l-value usage as deprecated in the docs, - And then do nothing until we find an actual case where this issue blocks development (or is actively dangerous for users).
Is there a way to emit a compilation warning when those macros are used as l-values? Even if only enabled on some compilers.
Regards
Antoine.
_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/YMMRRVVY... Code of Conduct: http://python.org/psf/codeofconduct/
_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/WNTZZYLT... Code of Conduct: http://python.org/psf/codeofconduct/
On 12/7/2021 6:52 PM, Sebastian Berg wrote:
One thing we once did in NumPy (for a runtime problem), was to intentionally break everyone at pre-release/dev time to point out what code needed fixing. Then flip the switch back at release time as to not break production. After a long enough time we enabled it for release mode.
Not saying that it was nice, but it was the only alternative would have been to never fix it.
I like this idea. We'd have to turn it back for RC, and ensure that it's possible to have working code both before/after the change. We may be getting enough usage during beta for it to be worthwhile, though we still have the problem of knock-on effects (where e.g. until NumPy works, nothing that depends on it can even begin testing). Cheers, Steve
On Tue, Dec 7, 2021 at 1:09 PM Steve Dower <steve.dower@python.org> wrote:
On 12/7/2021 6:52 PM, Sebastian Berg wrote:
One thing we once did in NumPy (for a runtime problem), was to intentionally break everyone at pre-release/dev time to point out what code needed fixing. Then flip the switch back at release time as to not break production. After a long enough time we enabled it for release mode.
Not saying that it was nice, but it was the only alternative would have been to never fix it.
I like this idea. We'd have to turn it back for RC, and ensure that it's possible to have working code both before/after the change. We may be getting enough usage during beta for it to be worthwhile, though we still have the problem of knock-on effects (where e.g. until NumPy works, nothing that depends on it can even begin testing).
Yeah, this sounds like a good approach *for things where the alternative is never to fix it*. -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
On Tue, Dec 7, 2021 at 7:55 PM Sebastian Berg <sebastian@sipsolutions.net> wrote:
One thing we once did in NumPy (for a runtime problem), was to intentionally break everyone at pre-release/dev time to point out what code needed fixing. Then flip the switch back at release time as to not break production. After a long enough time we enabled it for release mode.
Not saying that it was nice, but it was the only alternative would have been to never fix it.
A similar switch could be worthwhile if it helps Victor with experimenting on the dev-branch or reach a useful amount of projects. Of course, I am not sure it would do either...
Py_TYPE() and Py_SIZE() macros have been converted to static inline functions in May 2020 (Python 3.10 dev cycle): https://github.com/python/cpython/commit/ad3252bad905d41635bcbb4b76db30d570c... Honestly, when the change was merged, I didn't expect that it would break anything. Sadly, it broke multiple projects, including important ones like Cython and numpy. So it was reverted in Python 3.10. Once enough projects got prepared for these changes, I merged again the change in Python 3.11. Then more affected projects have been discovered. I also worked on tooling to discover affected projects before users start complaining ;-) And I wrote the PEP 674 which documents and justify these changes ;-) Victor
On Tue, Dec 7, 2021 at 9:00 AM Joao S. O. Bueno <jsbueno@python.org.br> wrote:
Sorry for stepping in - but I am seeing too many arguments in favour of the rules because "they are the rules", and just Victor arguing with what is met in the "real world".
But if this update can be done by a simple search/replace on the C source of projects, I can only perceive two scenarios this will affect: well maintained projects, for which it is fixable in minutes, and stale packages, no longer released that "happen to work" when someone downloads and builds for new Python versions. In these cases, the build will fail. If the person trying the build can't fix it, but can take the error to a proper, or high visibility, forum, someone will be able to come to the fix, leading to renewed visibility for the otherwise stale package.
It sounds like you (like probably many others) don't quite understand why "the rules" exist. Please trust me that what you perceive as simple updates, is not so simple when you take the whole ecosystem of dependencies into account. You may recall the long transition from Python 2 to 3. This was so painful in part because the core dev team (led by myself) was similarly optimistic about "oh, people can easily fix the few small things that will crop up". In practice, things like this take a very long time to fix everywhere, as people are waiting for their dependencies to solve the issue first before they can even start testing, and so on. (For example, many libraries still don't have wheels for Python 3.10, even though it was released over two months ago and 3.10.1 is already out.) -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
An idea for unmaintained incompatible C extensions is to update the C code when a C extension is installed, as part of the build process. For example, replace "Py_TYPE(obj) = new_type" with "Py_SET_TYPE(obj, new_type)". Something similar to the old "2to3" option of setuptools. The upgrade_pythoncapi.py of the pythoncapi_compat project is already a partial implementation of idea: it can update the C code, but it's not integrated in tools like setuptools. It's just an idea, a full implementation should be designed and written. Victor On Tue, Dec 7, 2021 at 5:56 PM Joao S. O. Bueno <jsbueno@python.org.br> wrote:
Sorry for stepping in - but I am seeing too many arguments in favour of the rules because "they are the rules", and just Victor arguing with what is met in the "real world".
But if this update can be done by a simple search/replace on the C source of projects, I can only perceive two scenarios this will affect: well maintained projects, for which it is fixable in minutes, and stale packages, no longer released that "happen to work" when someone downloads and builds for new Python versions. In these cases, the build will fail. If the person trying the build can't fix it, but can take the error to a proper, or high visibility, forum, someone will be able to come to the fix, leading to renewed visibility for the otherwise stale package.
On Tue, 7 Dec 2021 at 12:40, Antoine Pitrou <antoine@python.org> wrote:
On Tue, 7 Dec 2021 15:39:25 +0100 Petr Viktorin <encukou@gmail.com> wrote:
On 30. 11. 21 19:52, Victor Stinner wrote:
On Tue, Nov 30, 2021 at 7:34 PM Guido van Rossum <guido@python.org> wrote:
How about *not* asking for an exception and just following the PEP 387 process? Is that really too burdensome?
The Backward Compatibility section gives an explanation:
"This change does not follow the PEP 387 deprecation process. There is no known way to emit a deprecation warning when a macro is used as a l-value, but not when it's used differently (ex: r-value)."
Apart of compiler warnings, one way to implement the PEP 387 "deprecation process" would be to announce the change in two "What's New in Python 3.X?" documents. But I expect that it will not be efficient. Extract of the Rejected Idea section:
"(...) only few developers read the documentation, and only a minority is tracking changes of the Python C API documentation."
In my experience, even if a DeprecationWarning is emitted at runtime, developers miss or ignore it. See the recent "[Python-Dev] Do we need to remove everything that's deprecated?" discussion and complains about recent removal of deprecated features, like:
* collections.MutableMapping was deprecated for 7 Python versions (deprecated in 3.3) -- removed in 3.9 alpha, reverted in 3.9 beta, removed again in 3.11 * the "U" open() flag was deprecated for 10 Python versions (deprecated in 3.0) -- removed in 3.9 alpha, reverted in 3.9 beta, removed again in 3.11
For this specific PEP changes, I consider that the number of impacted projects is low enough to skip a deprecation process: only 4 projects are known to be impacted. One year ago (Python 3.10), 16 were impacted, and 12 have already been updated in the meanwhile. I'm talking especially about Py_TYPE() and Py_SIZE() changes which, again, has been approved by the Steering Council.
The current version of the PEP looks nice, but I don't think the rationale is strong enough. I believe we should: - Mark the l-value usage as deprecated in the docs, - And then do nothing until we find an actual case where this issue blocks development (or is actively dangerous for users).
Is there a way to emit a compilation warning when those macros are used as l-values? Even if only enabled on some compilers.
Regards
Antoine.
_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/YMMRRVVY... Code of Conduct: http://python.org/psf/codeofconduct/
_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/WNTZZYLT... Code of Conduct: http://python.org/psf/codeofconduct/
-- Night gathers, and now my watch begins. It shall not end until my death.
On 07. 12. 21 17:54, Joao S. O. Bueno wrote:
Sorry for stepping in - but I am seeing too many arguments in favour of the rules because "they are the rules", and just Victor arguing with what is met in the "real world".
OTOH, coming up with rules and then blatantly ignoring them is silly at best. If the rules are bad, they should definitely be changed. And if a case is exceptional enough, we should make an exception -- but to make an exception we need a very good understanding of why the rules are the way they are (and in this case. I don't think any single person has the proper understanding). One of the roles the backwards compatibility policy serves is a promise to our users. They can expect to not run into problems if they only upgrade to every second Python version and fix deprecation warnings (except for "extreme situations such as dangerously broken or insecure features or features no one could reasonably be depending on"). That is, IMO, a pretty good reason to consider sticking to the rules.
Extract of the Backward Compatibility section of the PEP: "This change does not follow the PEP 387 deprecation process. There is no known way to emit a deprecation warning only when a macro is used as a l-value, but not when it's used differently (ex: as a r-value)." Victor On Tue, Dec 7, 2021 at 4:41 PM Antoine Pitrou <antoine@python.org> wrote:
On Tue, 7 Dec 2021 15:39:25 +0100 Petr Viktorin <encukou@gmail.com> wrote:
On 30. 11. 21 19:52, Victor Stinner wrote:
On Tue, Nov 30, 2021 at 7:34 PM Guido van Rossum <guido@python.org> wrote:
How about *not* asking for an exception and just following the PEP 387 process? Is that really too burdensome?
The Backward Compatibility section gives an explanation:
"This change does not follow the PEP 387 deprecation process. There is no known way to emit a deprecation warning when a macro is used as a l-value, but not when it's used differently (ex: r-value)."
Apart of compiler warnings, one way to implement the PEP 387 "deprecation process" would be to announce the change in two "What's New in Python 3.X?" documents. But I expect that it will not be efficient. Extract of the Rejected Idea section:
"(...) only few developers read the documentation, and only a minority is tracking changes of the Python C API documentation."
In my experience, even if a DeprecationWarning is emitted at runtime, developers miss or ignore it. See the recent "[Python-Dev] Do we need to remove everything that's deprecated?" discussion and complains about recent removal of deprecated features, like:
* collections.MutableMapping was deprecated for 7 Python versions (deprecated in 3.3) -- removed in 3.9 alpha, reverted in 3.9 beta, removed again in 3.11 * the "U" open() flag was deprecated for 10 Python versions (deprecated in 3.0) -- removed in 3.9 alpha, reverted in 3.9 beta, removed again in 3.11
For this specific PEP changes, I consider that the number of impacted projects is low enough to skip a deprecation process: only 4 projects are known to be impacted. One year ago (Python 3.10), 16 were impacted, and 12 have already been updated in the meanwhile. I'm talking especially about Py_TYPE() and Py_SIZE() changes which, again, has been approved by the Steering Council.
The current version of the PEP looks nice, but I don't think the rationale is strong enough. I believe we should: - Mark the l-value usage as deprecated in the docs, - And then do nothing until we find an actual case where this issue blocks development (or is actively dangerous for users).
Is there a way to emit a compilation warning when those macros are used as l-values? Even if only enabled on some compilers.
Regards
Antoine.
_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/YMMRRVVY... Code of Conduct: http://python.org/psf/codeofconduct/
-- Night gathers, and now my watch begins. It shall not end until my death.
On 8/12/21 4:36 am, Antoine Pitrou wrote:
Is there a way to emit a compilation warning when those macros are used as l-values? Even if only enabled on some compilers.
Maybe the macro could be written so that it expands to something with a cast around it? I think gcc has an option for warning about casts used as l-values. -- Greg
Are you talking about gcc -Wignored-qualifiers? It seems like such warning is only emitted where the function is *defined*, not where the function is *called*. Example: --- const int f(void) { return 1; } int main() { return f(); } --- Output: --- $ gcc -Wextra y.c -o y y.c:1:1: warning: type qualifiers ignored on function return type [-Wignored-qualifiers] 1 | const int f(void) { return 1; } | ^~~~~ --- I'm only able to get a compile *error* when a macro is used as a l-value. Example: --- struct Point { int x; int y; }; #define POINT_X(p) ((int)p.x) int main() { struct Point p = {1, 2}; int x = POINT_X(p); // r-value ok POINT_X(p) = 1; // l-value ERROR return 0; } --- With "#define POINT_X(p) (p.x)", the macro can be used as l-value and r-value. Victor On Wed, Dec 8, 2021 at 1:43 AM Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
On 8/12/21 4:36 am, Antoine Pitrou wrote:
Is there a way to emit a compilation warning when those macros are used as l-values? Even if only enabled on some compilers.
Maybe the macro could be written so that it expands to something with a cast around it? I think gcc has an option for warning about casts used as l-values.
-- Greg _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/DG6GBX6O... Code of Conduct: http://python.org/psf/codeofconduct/
-- Night gathers, and now my watch begins. It shall not end until my death.
On Tue, Dec 7, 2021 at 3:43 PM Petr Viktorin <encukou@gmail.com> wrote:
If we would deprecate using Py_REFCNT as l-value in the docs, but wait with the conversion until it was *actually* needed, we would not lose anything: (...) ## CPython nogil fork
In CPython, we cannot change structs that are part of the stable ABI -- such as PyObject.ob_refcnt. IMO, getting rid of the macros that access ob_refcnt is a relatively small part of this issue.
The question is if the nogil optimization is relevant for CPython? It's a trade-off between backward compatibility and performance. It seems like some core devs are enthusiast by the idea of getting this work into CPython: count it in ;-) Getting rid of the GIL is an old dream which became reality: the abiltiy of using all CPU cores at the same time! But not all technical issues have to be solved at once. First, the *API* can be made compatible with nogil: the Python 3.10 Py_RECNT() change made this function compatible with nogil. Later, the question of the stable *ABI* can be studied.
## GraalVM Python
This PEP is not enough to get rid of wrappers in GraalVM, yet it forces users of CPython to adapt.
Currently, it's considered as second class citizen because of the C API issues. The idea is to put GraalVM Python at the same level as CPython. I would like to put all Python implementation in a fair competition. The competition is great for innovating and should benefit to everybody! The PEP 674 is part of this goal. You're right that this PEP alone is not enough to remove all wrappers: it doesn't solve all problems. On the other side, a large PEP like PEP 620 cannot be accepted... because it requires too many changes at once, it's not technically possible to implement all changes in a single Python. It must be done incrementally over multiple Python versions. That's why I'm trying to split the PEP 620 into smaller PEPs (now: PEP 670, PEP 674), write a better rationale for each incremental change, and better study the backward compatibility issues of each incompatible change.
Until disallowing macros as l-values allows concrete improvements in CPython, it should be the job of linters.
Do you know existing linters for C extensions? I don't know any :-(
FWIW, I do encourage alternative implementations to just not support l-value macros. There are only few projects doing this, and the fix is often (but not always!) easy. This should be a very small part of porting something to a different Python implementation (but I could definitely be wrong here, please correct me).
First, PyPy tried to only implement a subset of the C API and promote cffi for incompatible C extensions. This approach failed. So PyPy decided to be as compatible as possible with CPython for the C API. Now PyPy basically supports the whole C API and doesn't want to drop support for the C extensions "abusing" the C API, like using macros as l-value. The problem is that supporting all C API "abuses" requires to basically copy everything from CPython and it's inefficient. I didn't mention PyPy in the PEP 674 since this PEP alone doesn't help PyPy to avoid converting efficient PyPy objects to inefficient CPython objects when they are accessed by the C API (PyPy cpyext module). Victor -- Night gathers, and now my watch begins. It shall not end until my death.
Hi Victor, I wonder if there's a role for HPy in this context? What if instead of evolving the stable ABI and the limited API, instead we were to focus on first-class support for HPy? Surely 5 release cycles would be enough to completely remove the stable ABI and perhaps even the limited API in favor of HPy? Or am I misunderstanding the place of HPy in the ecosystem? --Guido PS. Eric wrote an analyzer for C code and checked it in under Tools/c-analyzer. This currently focuses on finding globals, but perhaps it forms a good starting point for a linter for C extensions? On Tue, Dec 7, 2021 at 3:53 PM Victor Stinner <vstinner@python.org> wrote:
On Tue, Dec 7, 2021 at 3:43 PM Petr Viktorin <encukou@gmail.com> wrote:
If we would deprecate using Py_REFCNT as l-value in the docs, but wait with the conversion until it was *actually* needed, we would not lose anything: (...) ## CPython nogil fork
In CPython, we cannot change structs that are part of the stable ABI -- such as PyObject.ob_refcnt. IMO, getting rid of the macros that access ob_refcnt is a relatively small part of this issue.
The question is if the nogil optimization is relevant for CPython? It's a trade-off between backward compatibility and performance. It seems like some core devs are enthusiast by the idea of getting this work into CPython: count it in ;-) Getting rid of the GIL is an old dream which became reality: the abiltiy of using all CPU cores at the same time!
But not all technical issues have to be solved at once. First, the *API* can be made compatible with nogil: the Python 3.10 Py_RECNT() change made this function compatible with nogil. Later, the question of the stable *ABI* can be studied.
## GraalVM Python
This PEP is not enough to get rid of wrappers in GraalVM, yet it forces users of CPython to adapt.
Currently, it's considered as second class citizen because of the C API issues. The idea is to put GraalVM Python at the same level as CPython.
I would like to put all Python implementation in a fair competition. The competition is great for innovating and should benefit to everybody!
The PEP 674 is part of this goal. You're right that this PEP alone is not enough to remove all wrappers: it doesn't solve all problems.
On the other side, a large PEP like PEP 620 cannot be accepted... because it requires too many changes at once, it's not technically possible to implement all changes in a single Python. It must be done incrementally over multiple Python versions. That's why I'm trying to split the PEP 620 into smaller PEPs (now: PEP 670, PEP 674), write a better rationale for each incremental change, and better study the backward compatibility issues of each incompatible change.
Until disallowing macros as l-values allows concrete improvements in CPython, it should be the job of linters.
Do you know existing linters for C extensions? I don't know any :-(
FWIW, I do encourage alternative implementations to just not support l-value macros. There are only few projects doing this, and the fix is often (but not always!) easy. This should be a very small part of porting something to a different Python implementation (but I could definitely be wrong here, please correct me).
First, PyPy tried to only implement a subset of the C API and promote cffi for incompatible C extensions. This approach failed.
So PyPy decided to be as compatible as possible with CPython for the C API. Now PyPy basically supports the whole C API and doesn't want to drop support for the C extensions "abusing" the C API, like using macros as l-value.
The problem is that supporting all C API "abuses" requires to basically copy everything from CPython and it's inefficient. I didn't mention PyPy in the PEP 674 since this PEP alone doesn't help PyPy to avoid converting efficient PyPy objects to inefficient CPython objects when they are accessed by the C API (PyPy cpyext module).
Victor -- Night gathers, and now my watch begins. It shall not end until my death. _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/UFPN4VUV... Code of Conduct: http://python.org/psf/codeofconduct/
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
For me, HPy is the only valid stable API and stable ABI in the long term which is efficient on any Python implementation. Its design is very different than the C API: HPy avoids all C API design mistakes, it doesn't leak any implementation detail. HPy can already be used today on CPython, even if it's not directly provided by CPython. Providing HPy as a first-class citizen in CPython, as already done in PyPy, would be great to promote HPy! However, HPy evolves quickly and so needs to be released more frequently than CPython. At least, we could promote it more in the C API documentation, as we already promote Cython. Promoting the HPy usage doesn't solve any issue listed in PEP 620, 670 and 674 since CPython still has to continue supporting the C API. We will only be fully free to make any change in Python internals without having to care about breaking the C API once the *LAST* C extensions using the C API will disappear... Look at Python 2.7 which is still used in 2021. I bet that C extensions using the C API are not doing to disappear soon. For me, the question is: => Is it ok to no longer be able to make any change in Python internals because of the public C API? The sub-question is: => Is it ok to have a slow deprecation process and wait 5 to 10 years until it will be possible again to evolve the Python internals? Obviously, my answer is that we must change the C API as soon as possible to allow again changing Python internals and help other Python implementations to support the C API. -- One option for CPython would be to have a native HPy support, and emulate the legacy C API with something similar to what PyPy does with its cpyext module. That may make C extensions using the C API slower and may increase their memory usage. Victor On Wed, Dec 8, 2021 at 1:08 AM Guido van Rossum <guido@python.org> wrote:
Hi Victor,
I wonder if there's a role for HPy in this context? What if instead of evolving the stable ABI and the limited API, instead we were to focus on first-class support for HPy? Surely 5 release cycles would be enough to completely remove the stable ABI and perhaps even the limited API in favor of HPy? Or am I misunderstanding the place of HPy in the ecosystem?
--Guido
PS. Eric wrote an analyzer for C code and checked it in under Tools/c-analyzer. This currently focuses on finding globals, but perhaps it forms a good starting point for a linter for C extensions?
-- Night gathers, and now my watch begins. It shall not end until my death.
On 08. 12. 21 1:47, Victor Stinner wrote:
For me, HPy is the only valid stable API and stable ABI in the long term which is efficient on any Python implementation. Its design is very different than the C API: HPy avoids all C API design mistakes, it doesn't leak any implementation detail.
HPy can already be used today on CPython, even if it's not directly provided by CPython.
Providing HPy as a first-class citizen in CPython, as already done in PyPy, would be great to promote HPy! However, HPy evolves quickly and so needs to be released more frequently than CPython. At least, we could promote it more in the C API documentation, as we already promote Cython.
Promoting the HPy usage doesn't solve any issue listed in PEP 620, 670 and 674 since CPython still has to continue supporting the C API. We will only be fully free to make any change in Python internals without having to care about breaking the C API once the *LAST* C extensions using the C API will disappear... Look at Python 2.7 which is still used in 2021. I bet that C extensions using the C API are not doing to disappear soon.
For me, the question is:
=> Is it ok to no longer be able to make any change in Python internals because of the public C API?
The sub-question is:
=> Is it ok to have a slow deprecation process and wait 5 to 10 years until it will be possible again to evolve the Python internals?
No, they should be evolved when they *need* to be evolved. This PEP calls for breaking things because they *might* need evolving in the future, but doesn't presemt any immediate benefit. In that case, I think it's better document that we don't like some usage, but only do the removals when they're helpful. Especially in cases where there can't be a proper deprecation period.
Hi, On Tue, 7 Dec 2021 16:08:27 -0800 Guido van Rossum <guido@python.org> wrote:
I wonder if there's a role for HPy in this context? What if instead of evolving the stable ABI and the limited API, instead we were to focus on first-class support for HPy?
The hope with the HPy project is indeed to provide a C API that is close to the original API to make porting easy and have it perform as close to the existing API as possible. At the same time, HPy is sufficently removed to be a good "C extension API" (as opposed to a stable subset of the CPython implementation API) that does not leak implementation details. To ensure this latter property is why we try to develop everything in parallel for CPython, PyPy, and GraalVM Python. On Wed, 8 Dec 2021 01:47:38 +0100 Victor Stinner <vstinner@python.org> wrote:
Providing HPy as a first-class citizen in CPython, as already done in PyPy, would be great to promote HPy! However, HPy evolves quickly and so needs to be released more frequently than CPython. At least, we could promote it more in the C API documentation, as we already promote Cython.
Indeed, at this point HPy is still evolving very fast. We are still solving issues while migrating NumPy, have begun adding support for HPy to Cython, and will start working on pybind11 soon. I believe by the time we have these users of the existing C API working, HPy should be in a state where it is generally useful and can be deemed stable enough that further development can follow a more stable process.
For me, HPy is the only valid stable API and stable ABI in the long term which is efficient on any Python implementation. Its design is very different than the C API: HPy avoids all C API design mistakes, it doesn't leak any implementation detail.
In the long run the HPy project would like to become a promoted API to write Python C extensions. Tim Felgentreff (on behalf of the HPy team)
I am sorry for the multi-post, it seems the first message I sent and which I thought got lost somewhere along the way was simply routed via pidgeon. I promise to wait for the message to arrive more patiently next time. Tim ________________________________ From: Tim Felgentreff <tim.felgentreff@oracle.com> Sent: Wednesday, 8 December 2021, 18:21 To: python-dev@python.org Subject: [External] : [Python-Dev] Re: PEP 674: Disallow using macros as l-value Hi, On Tue, 7 Dec 2021 16:08:27 -0800 Guido van Rossum <guido@python.org> wrote:
I wonder if there's a role for HPy in this context? What if instead of evolving the stable ABI and the limited API, instead we were to focus on first-class support for HPy?
The hope with the HPy project is indeed to provide a C API that is close to the original API to make porting easy and have it perform as close to the existing API as possible. At the same time, HPy is sufficently removed to be a good "C extension API" (as opposed to a stable subset of the CPython implementation API) that does not leak implementation details. To ensure this latter property is why we try to develop everything in parallel for CPython, PyPy, and GraalVM Python. On Wed, 8 Dec 2021 01:47:38 +0100 Victor Stinner <vstinner@python.org> wrote:
Providing HPy as a first-class citizen in CPython, as already done in PyPy, would be great to promote HPy! However, HPy evolves quickly and so needs to be released more frequently than CPython. At least, we could promote it more in the C API documentation, as we already promote Cython.
Indeed, at this point HPy is still evolving very fast. We are still solving issues while migrating NumPy, have begun adding support for HPy to Cython, and will start working on pybind11 soon. I believe by the time we have these users of the existing C API working, HPy should be in a state where it is generally useful and can be deemed stable enough that further development can follow a more stable process.
For me, HPy is the only valid stable API and stable ABI in the long term which is efficient on any Python implementation. Its design is very different than the C API: HPy avoids all C API design mistakes, it doesn't leak any implementation detail.
In the long run the HPy project would like to become a promoted API to write Python C extensions. Tim Felgentreff (on behalf of the HPy team) _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://urldefense.com/v3/__https://mail.python.org/mailman3/lists/python-de... Message archived at https://urldefense.com/v3/__https://mail.python.org/archives/list/python-dev... Code of Conduct: https://urldefense.com/v3/__http://python.org/psf/codeofconduct/__;!!ACWV5N9...
Hi, The HPy project is great and I'm supporting this cool project! As I wrote, IMO it's the best solution for the long term. My first concern is that right now, HPy is not mature nor widely used, and CPython still has to continue supporting a large amount of C extensions which are not likely to be ported to HPy soon. Well, we can do nothing, continue supporting these C extensions unmodified and we're good. My second concern is the inability to evolve CPython internals to implement new optimizations, and the inefficient implementation of the current C API in PyPy, GraalPython, etc. Sadly, HPy will only solve these problems when the last C extensions will be ported to HPy: when it will be reasonable to drop CPython public C API. Since Python 2.7 is still used in 2021, I expect that porting most C extensions to HPy will take a few years. Porting a C extension to HPy can be done incrementally on CPython. But the port is not cheap, a lot of code has to be adapted. I'm proposing to make the C API "less bad" by fixing the parts which are clearly identified as "bad". Only a minority of C extensions require changes, and on these C extensions, usually only a few lines need to be changed. For example, numpy is made of 307,300 lines of C code, and adapting numpy to the PEP 674 only modified 11 lines (now using Py_SET_TYPE or Py_SET_SIZE) and to add 4 lines (to define Py_SET_TYPE/Py_SET_SIZE for Python 3.8 and older). I expect that porting numpy to HPy will require to modify more lines. Right now, it's hard to bet which approach is the best: fix the current C API, or focus on HPy. IMO it would be too risky to only bet on HPy. At least, the main advantage of HPy is that it's currently fully developed outside CPython and it doesn't require any change in the C API! Victor On Wed, Dec 8, 2021 at 6:21 PM Tim Felgentreff <tim.felgentreff@oracle.com> wrote:
Hi,
On Tue, 7 Dec 2021 16:08:27 -0800 Guido van Rossum <guido@python.org> wrote:
I wonder if there's a role for HPy in this context? What if instead of evolving the stable ABI and the limited API, instead we were to focus on first-class support for HPy?
The hope with the HPy project is indeed to provide a C API that is close to the original API to make porting easy and have it perform as close to the existing API as possible. At the same time, HPy is sufficently removed to be a good "C extension API" (as opposed to a stable subset of the CPython implementation API) that does not leak implementation details. To ensure this latter property is why we try to develop everything in parallel for CPython, PyPy, and GraalVM Python.
On Wed, 8 Dec 2021 01:47:38 +0100 Victor Stinner <vstinner@python.org> wrote:
Providing HPy as a first-class citizen in CPython, as already done in PyPy, would be great to promote HPy! However, HPy evolves quickly and so needs to be released more frequently than CPython. At least, we could promote it more in the C API documentation, as we already promote Cython.
Indeed, at this point HPy is still evolving very fast. We are still solving issues while migrating NumPy, have begun adding support for HPy to Cython, and will start working on pybind11 soon. I believe by the time we have these users of the existing C API working, HPy should be in a state where it is generally useful and can be deemed stable enough that further development can follow a more stable process.
For me, HPy is the only valid stable API and stable ABI in the long term which is efficient on any Python implementation. Its design is very different than the C API: HPy avoids all C API design mistakes, it doesn't leak any implementation detail.
In the long run the HPy project would like to become a promoted API to write Python C extensions.
Tim Felgentreff (on behalf of the HPy team) _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/2NFJC7UA... Code of Conduct: http://python.org/psf/codeofconduct/
-- Night gathers, and now my watch begins. It shall not end until my death.
On Tue, 7 Dec 2021 16:08:27 -0800 Guido van Rossum guido@python.org wrote: I wonder if there’s a role for HPy in this context? What if instead of evolving the stable ABI and the limited API, instead we were to focus on first-class support for HPy? The hope with the HPy project is indeed to provide a C API that is close to the original API to make porting easy and have it perform as close to the existing API as possible. At the same time, HPy is sufficently removed to be a good “C extension API” (as opposed to a stable subset of the CPython implementation API) that does not leak implementation details. To ensure this latter property is why we try to develop everything in parallel for CPython, PyPy, and GraalVM Python. On Wed, 8 Dec 2021 01:47:38 +0100 Victor Stinner vstinner@python.org wrote: Providing HPy as a first-class citizen in CPython, as already done in PyPy, would be great to promote HPy! However, HPy evolves quickly and so needs to be released more frequently than CPython. At least, we could promote it more in the C API documentation, as we already promote Cython. Indeed, at this point HPy is still evolving very fast. We are still solving issues while migrating NumPy, have begun adding support for HPy to Cython, and will start working on pybind11 soon. I believe by the time we have these users of the existing C API working, HPy should be in a state where it is generally useful and can be deemed stable enough that further development can follow a more stable process. For me, HPy is the only valid stable API and stable ABI in the long term which is efficient on any Python implementation. Its design is very different than the C API: HPy avoids all C API design mistakes, it doesn’t leak any implementation detail. In the long run the HPy project would like to become a promoted API to write Python C extensions. Tim Felgentreff, posted by Matti Picus (on behalf of the HPy team)
Victor Stinner wrote:
First, PyPy tried to only implement a subset of the C API and promote cffi for incompatible C extensions. This approach failed.
Just to be clear: I don't think any of the PyPy devs assumed the huge Python c-extension ecosystem would suddenly run to adopt CFFI, so calling our effort to emulate the C-API a consequence of any failure is a bit of an exaggeration.
So PyPy decided to be as compatible as possible with CPython for the C API. Now PyPy basically supports the whole C API and doesn't want to drop support for the C extensions "abusing" the C API, like using macros as l-value.
I think it would be more accurate to say that PyPy, as a small project in the huge Python ecosystem, is trying to support any C-API used widely by the community. If there is a common PyPy voice (after all, we are also a project with many opinions), we don't "not want to drop support" nor "want to drop support" for any commonly used C-API interfaces, rather we want to stay out of this argument and promote alternatives such as CFFI, cppyy, and HPy instead.
Victor
Matti
participants (10)
-
Antoine Pitrou -
Greg Ewing -
Guido van Rossum -
Joao S. O. Bueno -
matti.picus＠gmail.com -
Petr Viktorin -
Sebastian Berg -
Steve Dower -
Tim Felgentreff -
Victor Stinner