[Python-Dev] Re: PEP 674 – Disallow using macros as l-value (version 2)

On 31. 01. 22 15:30, Victor Stinner wrote:

On Mon, Jan 31, 2022 at 2:31 PM Petr Viktorin wrote:

...

...

PEP: 674 Title: Disallow using macros as l-value

The current PEP is named "Disallow using Py_TYPE() and Py_SIZE() macros as l-values", which is misleading: it proposes to change 65 macros.

Right, I made changes since I posted the "version 2" to python-dev 13 days ago. Since nobody replied to my thread (before you), I directly submitted the latest (updated) PEP to the Steering Council: https://github.com/python/steering-council/issues/100

I decided to put "Py_TYPE() and Py_SIZE()" in the title, since in practice, only these two macros are causing troubles. The PEP changes 65 macros in total. In the latest PEP version, exactly 2 macros are breaking projects: Py_TYPE and Py_SIZE.

This is a reasonable assumption if you think that backwards-incompatible changes are OK to make if they don't affect "projects". (Presumably, projects that you know about.) If I don't agree wit that, I can't rely on the title/abstract to decide how much time I'll need to spend on the PEP :(

...

...

* Allow evolving CPython internals;

In the current PEP, this is now "Allow evolving CPython internals (change the PyObject structure);"

But that's not really true: the PyObject won't become changeable if the PEP is accepted, since `ob_type` is part of the stable ABI, and direct access to this field is compiled into existing extensions that use the stable ABI.

When I created https://bugs.python.org/issue39573, I chose the title "Make PyObject an opaque structure in the limited C API". Two years later, I changed the title to "[C API] Avoid accessing PyObject and PyVarObject members directly: add Py_SET_TYPE() and Py_IS_TYPE(), disallow Py_TYPE(obj)=type" since I understood that this project requires multiple incremental steps (and maybe changes should be done in multiple Python versions).

You're right that the PEP 674 alone is not enough to fully make the structure opaque. The next interesting problem is that PyType_FromSpec() and PyType_Spec API (which are part of the stable ABI!) use indirectly sizeof(PyObject) in the PyType_Spec.basicsize member.

One option to make the PyObject structure opaque would be to modify the PyObject structure to make it empty, and move its members into a new private _PyObject structure. This _PyObject structure would be allocated before the PyObject* pointer, same idea as the current PyGC_Head header which is also allocated before the PyObject* pointer.

The main blocker issue is that it breaks the stable ABI.

How should the PEP abstract be rephrased to be more accurate?

Remove the point from the Abstract?

...

...

On the PyPI top 5000 projects, only 14 projects (0.3%) are affected by 4 macro changes. Moreover, 24 projects just have to regenerate their Cython code to use ``Py_SET_TYPE()``.

This data was gathered in 2022. Since this change was made in 2020 and then reverted because it broke too many projects (which were presumably notified of the breakage and had 2 years to update), the 0.3% is misleading. It is a measure of the current porting effort, not an estimate of the chance of a given project being affected.

I tried to provide all data that I gathered. The following section list 14 projects that had to be fixed: https://www.python.org/dev/peps/pep-0674/#projects-released-with-a-fix

I didn't check if they are part of the current top 5000 PyPI project or not.

How should the abstract be rephrased to mention these projects? Does someone know how to check if these projects are part of the top 5000 PyPI projects?

I think you should make it clear that the percentage was taken after fixing many of the projects. That change will probably make the numbers look rather meaningless. So either explain why they're meaningful, or remove the paragraph. (Somewhat unrelated: you could also put the motivation in a Motivation section, rather than the Abstract. If that leaves the abstract with just one sentence, it's good -- provided the sentence is still a good summary of the proposed changes.)