SC reply to PEP 674 -- Disallow using macros as l-values
Hello, Thank you for submitting PEP 674 -- Disallow using macros as l-values! For Py_TYPE, we respect the previous steering council's decision: https://github.com/python/steering-council/issues/79 The Py_TYPE macro can be changed as proposed in 3.11. However, the old SC explicitly noted that the exception only applies to Py_SET_TYPE, and that in general, there should be a deprecation notice in some way or form. After deliberating the proposed changes, the SC agrees that they will make CPython better. However, we are not convinced that they should be made *right now*. We see no reason to rush. We would like to ask you to rework the PEP to reflect these suggestions. It is not feasible to provide visible deprecation warnings for the proposed changes. We feel that this is not a good reason to skip the deprecation process entirely. Quite the opposite: if we need to skip a step, we should move even more carefully than usual. We suggest the following process: First, add notices to any documentation that using the macro as a l-value is a deprecated, CPython-specific detail. It should be clear that the only reason this usage is allowed is for backwards compatibility, and that alternate implementations of the C API are free to not allow this. Then, wait until all supported versions of CPython include any relevant replacement macros such as Py_SET_SIZE. (This doesn't mean adding "set" variants to all the changed macros -- just to those few where setting can be useful and safe.) With the current schedule, this means waiting until Python 3.14, which is planned for October 2024, when Python 3.8 will reach end-of-life. The SC can give an exception to shorten the wait for specific macros, if there is a real benefit to it. (For example: the nogil branch should include any necessary changes, and they should be approved and merged with nogil itself). Finally, ask the new Steering Council to make the change and finally disallow using the macros as l-values. If there's no negative feedback or unexpected downsides, we hope that the future SC will summarily approve. Hopefully, this strategy will also work for PyDescr_NAME and other macros that were left out of the PEP for now. Additionally, consider making the change in the 3.11 version of the limited API. This version is opt-in, so users are free to do the switch when it is convenient for them. Also, it would allow early testing and review of the changes. Further discussion about this should happen on python-dev, like usual. - Petr, on behalf of the Steering Council
Hi, Thanks for looking into my PEP 674! I don't understand well why Py_SIZE() cannot be changed until Py_SET_SIZE() is available on all supported Python versions (so affected projects don't have to add a compatibility code), whereas it's ok to require a compatibility code to keep support for Python 3.8 and older for the approved Py_TYPE() change (Py_SET_TYPE was added to Python 3.9). Many projects affected by the Py_SIZE() change are already affected by the Py_TYPE() change, and so need a compatibility code anyway: 33% (5/15) of affected projects are affected by Py_SIZE() and Py_TYPE() changes (see below). Not changing Py_SIZE() now avoids adding a compatibility code to the 33% (5/15) of affected projects only affected by the Py_SIZE() change. Either changing Py_TYPE() *and* Py_SIZE(), or none, would make more sense to me. Well, I would prefer to change both, since the work is already done. The change is already part of Python 3.11 and all projects known to be afffected are already fixed. And the Py_TYPE() change was already approved. -- The Py_TYPE() change requires a compatibility code to get Py_SET_TYPE() on Python 3.8 and older, use pythoncapi_compat.h or add: #if PY_VERSION_HEX < 0x030900A4 # define Py_SET_TYPE(obj, type) ((Py_TYPE(obj) = (type)), (void)0) #endif The Py_SIZE() change requires a similar compatibility code. For example, boost defines Py_SET_TYPE() *and* Py_SET_SIZE(): #if PY_VERSION_HEX < 0x030900A4 # define Py_SET_TYPE(obj, type) ((Py_TYPE(obj) = (type)), (void)0) # define Py_SET_SIZE(obj, size) ((Py_SIZE(obj) = (size)), (void)0) #endif -- Affected projects from PEP 674. Projects affected by Py_SIZE() and Py_TYPE() changes (5): * guppy3: Py_SET_TYPE(), Py_SET_SIZE(), Py_SET_REFCNT(), use pythoncapi_compat.h * bitarray: Py_SET_TYPE(), Py_SET_SIZE(), use pythoncapi_compat.h * mypy: Py_SET_TYPE(), Py_SET_SIZE(), use pythoncapi_compat.h * numpy: Py_SET_TYPE(), Py_SET_SIZE(), custom compatibility code * boost: Py_SET_TYPE(), Py_SET_SIZE(), custom compatibility code Projects only affected by the Py_SIZE() change (5): * python-snappy: Py_SET_SIZE(), use pythoncapi_compat.h * recordclass: use custom py_refcnt() and py_set_size() macros * Cython: Py_SET_SIZE(), Py_SET_REFCNT(), custom compatibility code * immutables: Py_SET_SIZE(), use pythoncapi_compat.h * zstd: Py_SET_SIZE(), use pythoncapi_compat.h Projects only affected by Py_TYPE() change (5): * datatable: Py_SET_TYPE(), Py_SET_REFCNT(), use pythoncapi_compat.h * mercurial: Py_SET_TYPE(), use pythoncapi_compat.h * pycurl: Py_SET_TYPE(), custom compatibility code * duplicity: Py_SET_TYPE(), test PY_MAJOR_VERSION and PY_MINOR_VERSION, or use Py_TYPE() as l-value * gobject-introspection: Py_SET_TYPE(), custom compatibility code These examples don't count the larger number of affected projects using Cython which only need to re-run Cython to use Py_SET_REFCNT(), Py_SET_TYPE() and Py_SET_SIZE(). I would like to add that 100% of the top 5000 PyPI projects are already fixed for PEP 674, but 26 projects need a release including a fix (which will likely happend before Python 3.11 final release). Victor -- Night gathers, and now my watch begins. It shall not end until my death.
On 22. 02. 22 15:10, Victor Stinner wrote:
Hi,
Thanks for looking into my PEP 674!
I don't understand well why Py_SIZE() cannot be changed until Py_SET_SIZE() is available on all supported Python versions (so affected projects don't have to add a compatibility code), whereas it's ok to require a compatibility code to keep support for Python 3.8 and older for the approved Py_TYPE() change (Py_SET_TYPE was added to Python 3.9).
See the exception for Py_Type, https://github.com/python/steering-council/issues/79#issuecomment-982702725 :
This decision only applies to Py_SET_TYPE
We mean that in general (not for this change as we have made an exception per our previous argument), there should be a deprecation notice in some way or form (in these cases must be in the docs/what's new as there is no way to make the preprocessor warn only when we need it). We just wanted to reiterate that in general, we still want two releases with some deprecation notice for any deprecation, unless exempt.
The SC didn't want to revert the earlier SC's decision. But this PEP lists the dozens of new macros, and no clear reason why the general policy shouldn't apply.
Many projects affected by the Py_SIZE() change are already affected by the Py_TYPE() change, and so need a compatibility code anyway: 33% (5/15) of affected projects are affected by Py_SIZE() and Py_TYPE() changes (see below).
Not changing Py_SIZE() now avoids adding a compatibility code to the 33% (5/15) of affected projects only affected by the Py_SIZE() change.
Either changing Py_TYPE() *and* Py_SIZE(), or none, would make more sense to me. Well, I would prefer to change both, since the work is already done. The change is already part of Python 3.11 and all projects known to be afffected are already fixed. And the Py_TYPE() change was already approved.
IMO it's still possible to change none of them *now* and go through the a deprecation-in-docs period for everything, if you want that.
--
The Py_TYPE() change requires a compatibility code to get Py_SET_TYPE() on Python 3.8 and older, use pythoncapi_compat.h or add:
#if PY_VERSION_HEX < 0x030900A4 # define Py_SET_TYPE(obj, type) ((Py_TYPE(obj) = (type)), (void)0) #endif
The Py_SIZE() change requires a similar compatibility code. For example, boost defines Py_SET_TYPE() *and* Py_SET_SIZE():
#if PY_VERSION_HEX < 0x030900A4 # define Py_SET_TYPE(obj, type) ((Py_TYPE(obj) = (type)), (void)0) # define Py_SET_SIZE(obj, size) ((Py_SIZE(obj) = (size)), (void)0) #endif
Right. But on 3.9+ you don't need any of that, and as since there is no need to do the change *right* now, it can wait until 3.8 becomes history.
--
Affected projects from PEP 674.
Projects affected by Py_SIZE() and Py_TYPE() changes (5):
* guppy3: Py_SET_TYPE(), Py_SET_SIZE(), Py_SET_REFCNT(), use pythoncapi_compat.h * bitarray: Py_SET_TYPE(), Py_SET_SIZE(), use pythoncapi_compat.h * mypy: Py_SET_TYPE(), Py_SET_SIZE(), use pythoncapi_compat.h * numpy: Py_SET_TYPE(), Py_SET_SIZE(), custom compatibility code * boost: Py_SET_TYPE(), Py_SET_SIZE(), custom compatibility code
Projects only affected by the Py_SIZE() change (5):
* python-snappy: Py_SET_SIZE(), use pythoncapi_compat.h * recordclass: use custom py_refcnt() and py_set_size() macros * Cython: Py_SET_SIZE(), Py_SET_REFCNT(), custom compatibility code * immutables: Py_SET_SIZE(), use pythoncapi_compat.h * zstd: Py_SET_SIZE(), use pythoncapi_compat.h
Projects only affected by Py_TYPE() change (5):
* datatable: Py_SET_TYPE(), Py_SET_REFCNT(), use pythoncapi_compat.h * mercurial: Py_SET_TYPE(), use pythoncapi_compat.h * pycurl: Py_SET_TYPE(), custom compatibility code * duplicity: Py_SET_TYPE(), test PY_MAJOR_VERSION and PY_MINOR_VERSION, or use Py_TYPE() as l-value * gobject-introspection: Py_SET_TYPE(), custom compatibility code
These examples don't count the larger number of affected projects using Cython which only need to re-run Cython to use Py_SET_REFCNT(), Py_SET_TYPE() and Py_SET_SIZE().
I would like to add that 100% of the top 5000 PyPI projects are already fixed for PEP 674, but 26 projects need a release including a fix (which will likely happend before Python 3.11 final release).
Thank you for surveying and fixing popular public projects. As you know, there are many more that are not public, popular, nor fixed.
On Tue, Feb 22, 2022 at 1:54 PM Petr Viktorin <encukou@gmail.com> wrote:
First, add notices to any documentation that using the macro as a l-value is a deprecated, CPython-specific detail. It should be clear that the only reason this usage is allowed is for backwards compatibility, and that alternate implementations of the C API are free to not allow this.
In practice, alternate implementations must implement exactly the same C API than CPython, otherwise they become incompatible with projects affected by PEP 674 (41 projects including major projects like Cython and numpy). Let me quote Matti about the PyPy project: "I think it would be more accurate to say that PyPy, as a small project in the huge Python ecosystem, is trying to support any C-API used widely by the community. If there is a common PyPy voice (after all, we are also a project with many opinions), we don't "not want to drop support" nor "want to drop support" for any commonly used C-API interfaces, rather we want to stay out of this argument and promote alternatives such as CFFI, cppyy, and HPy instead." https://mail.python.org/archives/list/python-dev@python.org/message/3HGX42QF... The whole purpose of PEP 674 is to remove this constraint in alternate implementations *and* in CPython. Victor -- Night gathers, and now my watch begins. It shall not end until my death.
participants (2)
-
Petr Viktorin -
Victor Stinner