Wait, where is the HPy project in that plan? :-) The HPy project (brand new C API) is a good solution for the long term!

My concerns about HPy right now is that, in short, CPython has to continue supporting the C API for a few more years, and we cannot evolve CPython before it will become reasonable to consider removing the "legacy" C API.

I explained that in details in the PEP 674 (Disallow using Py_TYPE() and Py_SIZE() macros as l-values): https://www.python.org/dev/peps/pep-0674/#relationship-with-the-hpy-project

In parallel, we should continue promoting the usage of Cython, cffi, pybind11 and HPy, rather than using directly the C API.

Victor

On Jan 28, 2022, at 09:00, Steve Dower steve.dower@python.org wrote:

Does HPy have any clear guidance or assistance for their users to keep it up to date?

I'm concerned that if we simply substitute "support the C API for everyone" with "support the C API for every version of HPy" we're no better off.

Will it ever make sense to pull HPy into the CPython repo so that they evolve together? I can see advantages and disadvantages. If there’s a point in the future where we can just start promoting HPy as an official alternative C API, then it will likely get more traction over time. The disadvantage is that HPy would evolve at the same annual pace as CPython.

-Barry

On 1/28/2022 5:15 PM, Barry Warsaw wrote:

On Jan 28, 2022, at 09:00, Steve Dower steve.dower@python.org wrote:

...

Does HPy have any clear guidance or assistance for their users to keep it up to date?

I'm concerned that if we simply substitute "support the C API for everyone" with "support the C API for every version of HPy" we're no better off.

Will it ever make sense to pull HPy into the CPython repo so that they evolve together? I can see advantages and disadvantages. If there’s a point in the future where we can just start promoting HPy as an official alternative C API, then it will likely get more traction over time. The disadvantage is that HPy would evolve at the same annual pace as CPython.

Possibly, but we'd have to be really careful to not actually *evolve* HPy. It would essentially be a new stable API, but ideally one that uses all the preprocessor tricks we can (and perhaps runtime tricks) to compile against any CPython version rather than just the one that it comes with.

PSF "ownership" is probably enough to make it official (for those people who need everything to be "official"). I don't think that's necessary, but it does smooth the path for some people to be willing to use it.

Cheers, Steve

If a project does R releases per year for P platforms that need to support V versions of Python, they would normally have to build R * P * V wheels. With a stable ABI, they could reduce that to R * P. That's the key point, right?

Can HPy do that?

actually, it can do even better than that. When you compile an HPy extension you can choose which ABI to target: - CPython ABI: in this modality, all HPy_* calls are statically translated (using static inline functions) into the corresponding Py_* call, and it generates modules like foo.cpython-38-x86_64-linux-gnu.so, undistinguishable from a "normal" module

- HPy universal ABI: in this modality, it generates something like foo.hpy-x86_64-linux-gnu.so: all API calls are done through the HPyContext (which is basically a giant vtable): this module can be loaded by any implementation which supports the HPy universal ABI, including CPython, PyPy and GraalPython.

The main drawback of the universal ABI is that it's slightly slower because it goes through the vtable indirection for every call, in particular HPy_Dup/HPy_Close which are mapped to Py_INCREF/Py_DECREF. Some early benchmark indicate a 5-10% slowdown. We haven't benchmarked it against the stable ABI though.

Of course, in order to be fully usable, the HPy universal ABI will need special support by PyPI/pip/etc, because at the moment it is impossible to package it inside a wheel, AFAIK.

ciao, Antonio

On Fri, Jan 28, 2022 at 6:28 PM Guido van Rossum guido@python.org wrote:

I think we will get *one* chance in the next decade to get it right. Whether that's HPy or evolution of the C API I'm not sure.

Would you mind to elaborate? Which risk do you expect from switching to HPy and from fixing the C API (introducing incompatible C API changes)?

For me, promoting HPy and evolution of the C API are complementary, can and must done in parallel for me. As I explained in PEP 674, while HPy does help C extensions writers, it doesn't solve any problem for CPython right now. CPython is still blocked by implementation details leaked throught the C API that we must still maintain for a few more years.

Victor, am I right that the (some) stable ABI will remain important because projects don't have resources to build wheels for every Python release? If a project does R releases per year for P platforms that need to support V versions of Python, they would normally have to build R * P * V wheels. With a stable ABI, they could reduce that to R * P. That's the key point, right?

There are different use cases.

1) First, my main worry is that we put a high pressure on maintainers of most important Python dependencies before the next of a new Python version, because we want them to handle the flow of incompatible C API changes before the final Python 3.x versions is released, to get them available when Python 3.x final is released.

It annoys core developers who cannot change things in Python without getting an increasing number of complains about a large number of broken packages, sometimes with a request to revert.

It annoys C extensions maintainers who have to care about Python alpha and beta releases which are not convenient to use (ex: not available in Linux distributions). Moreover, it became common to ask multiple changes and multiple releases before a Python final release, since more incompatible changes are introduced in Python (before the beta1).

2) Second, as you said, the stable ABI reduces the number of binary packages which have to be built. Small projects with a little team (ex: a single person) don't have resources to set up a CI and maintain it to build all these packages. It's doable, but it isn't free.

--

The irony of the situation is that we must break the C API (hiding structures is technically an incompatible change)... to make the C API stable. Breaking it now to make it stable later.

We already broke the C API many times in the past. The difference here is that changes are done in the purpose of bending it towards the limited C API and the stable ABI.

My expectation is that replacing frame->f_code with PyFrame_GetCode() only has to be done exactly once: this API is not going this change. Sadly, the changes are not limited to frame->f_code, more changes are needed. For example, for PyFrameObject, accesses to every structure member must have to go through a function call (getter or setter function). Hopefully, only a small number of member are used by C extensions.

The tricky part is to think about the high level API ("use cases") rather than just adding functions doing "return struct->member" and "struct->member = new_value". The PyThreadState_EnterTracing() and PyThreadState_LeaveTracing() functions added to Python 3.11 are a good example: the API is "generic" and the implementation changes 2 structure members, not a single one.

In practice, what I did since Python 3.8 is to introduce a small number of C API changes per Python versions. We tried the "fix all the things at once" approach (!!!) with Python 3, and it... didn't go well. All C extensions had to suddenly write their own compatibility layer for a large number of C API functions (ex: replace PyInt_xxx with PyLong_xxx, without losing Python 2 support!). The changes that I'm introducing in the C API usually impact less than 100 extensions in total (usually, I would say between 10 and 25 per Python version, but it's hard to measure exactly).

Can HPy do that?

I wish more projects are incrementally rewritten with Cython, cffi, pybind11 and HPy, and so slowly move away using directly the C API.

Yes, HPy support an "universal build" mode which allows to only build a C extension once, and use it on multiple *CPython* versions *and* (that's the big news!) multiple *PyPy* versions! I even heard that it also brings GraalPython support for free ;-)

Victor

On Mon, Jan 31, 2022 at 1:48 PM Petr Viktorin encukou@gmail.com wrote:

(* we could also break the stable ABI, and we could even do it reasonably safely over a long period of time, but that's a whole different discussion.)

IMO the stable ABI must be change in the long term, it still leaks too many implementation details. But right now, I didn't gather enough data about the problematic APIs and what must be changed exactly. I would prefer to only do once the work will be really blocked and there would be no other choice.

Right now, I'm focused on fixing the *API*. It doesn't require to break the stable ABI.

If we change the stable ABI, I would prefer to fix multiple issues at once. Examples:

* No longer return borrowed references (ex: PyDict_GetItem is part of the stable ABI) and no longer steal references (ex: PyModule_AddObject)

* Disallow getting direct access into an object data without a function to "release" the data. For example, PyBytes_AsString() gives a direct access into the string, but Python doesn't know when the C extension is done with it, and when it's safe to delete the object. Such API prevents to move Python objects in memory (implement a moving garbage collector in Python).

* Disallow dereferencing a PyObject* pointer: most structures must be opaque. It indirectly means that accessing directly structure members must also be disallowed. PEP 670 and PEP 674 are partially fixing the issues.

Victor

On Mon, Jan 31, 2022 at 4:03 PM Petr Viktorin encukou@gmail.com wrote:

...

If we change the stable ABI, I would prefer to fix multiple issues at once. Examples:

• No longer return borrowed references (ex: PyDict_GetItem is part of

the stable ABI) and no longer steal references (ex: PyModule_AddObject)

• Disallow getting direct access into an object data without a

function to "release" the data. For example, PyBytes_AsString() gives a direct access into the string, but Python doesn't know when the C extension is done with it, and when it's safe to delete the object. Such API prevents to move Python objects in memory (implement a moving garbage collector in Python).

• Disallow dereferencing a PyObject* pointer: most structures must be

opaque. It indirectly means that accessing directly structure members must also be disallowed. PEP 670 and PEP 674 are partially fixing the issues.

(...) fixing these in the API first is probably the way to go.

That's what I already did in the past and what I plan to do in the future.

Victor

Hi Guido,

My "north star", as you say, is the HPy "design" (not the actual HPy API). I would like to convert PyObject* to opaque handles: dereferencing a PyObject* pointer would simply fail with a compiler error.

I'm working bottom-to-top: prepare PyObject and PyVarObject to become opaque, *and* top-to-bottom: prepare subclasses (structures "inheriting" from PyObject and PyVarObject) to become opaque like PyFrameObject.

IMO if PyObject* becomes a handle, the migration to the HPy API should be much easier.

So far, a large part of the work has been identifying which APIs are preventing to move the current C API to opaque handles. The second part has been fixing these APIs one by one, starting with changes which don't break the C API.

In the last 5 years, I fixed many issues of the C API. Very few projects were impacted since I avoided incompatible changes on purpose.

Now I reached the end of my (current) queue with the most controversial changes: incompatible C API changes. I wrote PEP 670 and 674 to share my overall rationale and communicate on the reasons why I consider that these changes will make our life easier next years.

I'm well aware that in the short term, the benefits are very limited. But we cannot jump immediately to opaque PyObject* handles. This work must be done incrementally.

On Thu, Feb 3, 2022 at 1:40 AM Guido van Rossum guido@python.org wrote:

...

1. First, my main worry is that we put a high pressure on maintainers

of most important Python dependencies before the next of a new Python version, because we want them to handle the flow of incompatible C API changes before the final Python 3.x versions is released, to get them available when Python 3.x final is released.

Hm, maybe we should reduce the flow. And e.g. reject PEP 674...

We need to find a the right limit when introducing C API to not break too many C extensions "per Python release". Yes, PEP 674 is an example of incompatible C API change, but Python 3.11 already got many others which don't are unrelated to that PEP.

For example, the optimization work done by your Microsoft team broke a bunch of projects using PyThreadState and PyFrameObject APIs. See the related What's New in Python 3.11 entries: https://docs.python.org/dev/whatsnew/3.11.html#id2 (I didn't keep track of which projects are affected by these changes.)

I would like to ensure that it remains possible to optimize Python, but for that, we need to reduce the friction related to C API changes. What's the best approach for that remains an open question. I'm proposing some solutions, we are discussing advantages and drawbacks ;-)

...

It annoys core developers who cannot change things in Python without getting an increasing number of complains about a large number of broken packages, sometimes with a request to revert.

You are mostly talking about yourself here, right? Since the revert requests were mostly aimed at you. :-)

Latest requests for revert are not about C API changes that I made.

Cython and the C API exception change: https://mail.python.org/archives/list/python-dev@python.org/thread/RS2C53LDZ...

There are other requests for revert in Python 3.11 related to Python changes, not to the C API.

So far, I only had to revert 2 changes about my C API work:

* PyType_HasFeature(): the change caused a performance regression on macOS, sadly Python cannot be built with LTO. With LTO (all platforms but macOS), my change doesn't affect performances.

* Py_TYPE() / Py_SIZE() change: I reverted my change to have more time to prepare affected projects. Two years later, I consider that this preparation work is now done (all affected projects are ready) and so I submitted the PEP 674. Affected projects (merged and pending fixes): https://www.python.org/dev/peps/pep-0674/#backwards-compatibility

...

Moreover, it became common to ask multiple changes and multiple releases before a Python final release, since more incompatible changes are introduced in Python (before the beta1).

Sorry, your grammar confuses me. Who is asking whom to do what here?

Cython is the best example. During the development cycle of a new Python version, my Fedora team adapts Cython to the next Python and then request a release to get an official release supporting an alpha Python version.

A release is not strictly required by us (we can apply downsteam patches), but it's more convenient for us and for people who cannot use our work-in-progress "COPR" (repository of RPM packages specific to Fedora). When we ask a project (using Cython) to merge our pull request, maintainers want to test it on the current Python alpha version, and it's not convenient when Cython is unusable.

Between Python 3.x alpha1 and Python 3.x final, there might be multiple Cython releases to handle each time a bunch of C API incompatible changes. On Python and Cython sides, so far, there was no coordination to group incompatible changes, they land between alpha1 and beta1 in an irregular fashion.

...

2. Second, as you said, the stable ABI reduces the number of binary

packages which have to be built. Small projects with a little team (ex: a single person) don't have resources to set up a CI and maintain it to build all these packages. It's doable, but it isn't free.

Maybe we need to help there. For example IIRC conda-forge will build conda packages -- maybe we should offer a service like that for wheels?

Tooling to automate the creation of wheel binary packages targeting the stable ABI would help. C extensions likely need changing a few function calls which are not supported by the limited C API. Someone has to do this work.

My hope is that the quantify of changes is small (ex: modify 2 or 3 function calls) for small C extensions. I didn't try in practice yet.

...

The irony of the situation is that we must break the C API (hiding structures is technically an incompatible change)... to make the C API stable. Breaking it now to make it stable later.

The question is whether that will ever be enough. Unless we manage to get rid of the INCREF/DECREF macros completely (from the public C API anyway) we still can't change object layout.

I'm open to ideas if someone has a better plan ;-)

Keeping reference counting for consumers of the C API (C extensions) doesn't prevent to avoid reference counting inside Python (switch to a completely different GC implementation). PyPy cpyext is a concrete example of that.

The nogil fork keeps Py_INCREF()/Py_DECREF() functions, but changes their implementation. If we have to (ex: if we merge nogil), we can convert Py_INCREF() / Py_DECREF() static inline functions to regular functions tomorrow (and then change their implementation) without breaking the API.

Adding abstractions (getter and setter functions) on PyObject give more freedom to consider different options to evolve Python tomorrow.

Is this worth it? Maybe we should just declare those structs and APIs *unstable* and tell people who use them that they can expect to be broken by each alpha release. As you say, hopefully this doesn't affect most people. Likely it'll affect Cython dramatically but Cython is such a special case that trying to evolve the C API will never satisfy them. We'll have to deal with it separately. (Debuggers are a more serious concern. We may need to provide higher-level APIs for debuggers to do the things they need to do. Mark's PEP 669 should help here.)

IMO we only need to add 5 to 10 functions to cover most use cases involving PyThreadState and PyFrameObject. The remaining least common usages can continue to require changes at each Python releases. The short term goal is only to reduce the number of required changes per Python releases.

Yes, I'm talking about Cython, debuggers and profilers.

Another example is that Cython currently calls PyCode_New() to create a fake frame object with a filename and line number. IMO it's the wrong abstraction level: Python should provide a function to create a frame with a filename and line number, so the caller doesn't have to bother about the complex PyCode_New() API and frequent PyCodeObject changes. (Correct me if this problem has already been solved in Python.)

...

In practice, what I did since Python 3.8 is to introduce a small number of C API changes per Python versions. (...)

How *do* you count this? Try to compile the top 5000 PyPI packages?

I'm using code search in the source code of top 5000 PyPI packages and I'm looking at broken packages in Fedora when we update Python. Also, sometimes people add a comment on an issue to mention that their project is broken by a change.

That might severely undercount a long tail of proprietary extensions.

Right.

Victor -- Night gathers, and now my watch begins. It shall not end until my death.

On Wed, Feb 2, 2022 at 4:46 PM Guido van Rossum guido@python.org wrote:

A few notes in this:

Maybe we need to help there. For example IIRC conda-forge will build conda packages -- maybe we should offer a service like that for wheels?

Yes, conda-forge used a complex CI system to build binaries conda packages for a variety of Python versions. And it does have some support for development versions. ONce a "feedstock" is developed, it's remarkably painless to get all the binaries up and available.

I imagine someone could borrow a bunch of that code to make a system to build wheels.

In fact, ther is the MAcPYthon org:

https://github.com/MacPython

Which began as a place to share building scripts for Mac binaries, but has expanded to build wheels for multiple platforms for the scipy stack. I don't know how it works these days -- I am no longer involved since I discovered conda, but they seem to have some nice stuff there -- perhaps it could be leveraged for more projects.

However: one of the challenges for building C extensions is that they often depend on external C libs -- and that is exactly the problem that conda was built to address. So in a sense, a conda-forge-like auto-build system is inherently easier for conda packages than binary wheels.

Which doesn't mean it couldn't be done -- just that the challenge of third party libs would need to be addressed.

In any case, someone would have to do the work, as usual.

-CHB

On Fri, Feb 4, 2022 at 12:55 AM Eric V. Smith eric@trueblade.com wrote:

It seems to me that moving PyObject* to be a handle leaves you in a place very similar to HPy. So why not just focus on making HPy suitable for developing C extensions, leave the existing C API alone, and eventually abandon the existing C API?

I agree (but I'm biased :)), but I think there is also an important point which is easy to miss/overlook: it is not enough to declare that now you have handles instead of refcounting, you also need a way to enforce/check that the handles are used correctly.

CPython might declare that object references are now handles and that each handle must be closed individually: this would work formally, but as long as handles are internally implemented on top of refcounting, things like closing the same handle twice would just continue to work if by chance the total refcount is still correct. This means that we will have extensions which will be formally incorrect but will work well on CPython, and horribly break as soon as you try to load them on e.g. PyPy.

That's the biggest selling point of the HPy debug mode: in debug mode, HPy actively check that handles are closed properly, and it warns you if you close a handle twice or forget to close a handle, even on CPython.

ciao, Antonio

On Fri, Feb 4, 2022 at 12:52 AM Eric V. Smith eric@trueblade.com wrote:

On 2/3/2022 12:15 PM, Victor Stinner wrote:

...

IMO if PyObject* becomes a handle, the migration to the HPy API should be much easier.

It seems to me that moving PyObject* to be a handle leaves you in a place very similar to HPy. So why not just focus on making HPy suitable for developing C extensions, leave the existing C API alone, and eventually abandon the existing C API?

I tried to explain the reasons why HPy doesn't solve all problems in the PEP 674: https://www.python.org/dev/peps/pep-0674/#the-c-api-is-here-is-stay-for-a-fe...

One problem is to provide a better C API to users: HPy is great for that!

Another problem is the inability to evolve Python because the C API leaks implementation details: HPy doesn't solve this problem because Python must continue supporting the C API for a few more years.

My approach is to (slowly) bend the C API towards HPy design/API to ease the migration to HPy *and* (slowly) allow changing more Python internals (without affecting the public C API).

Victor

On 31. 01. 22 16:14, Victor Stinner wrote:

On Mon, Jan 31, 2022 at 4:03 PM Petr Viktorin encukou@gmail.com wrote:

...

...

If we change the stable ABI, I would prefer to fix multiple issues at once. Examples:

• No longer return borrowed references (ex: PyDict_GetItem is part of

the stable ABI) and no longer steal references (ex: PyModule_AddObject)

• Disallow getting direct access into an object data without a

function to "release" the data. For example, PyBytes_AsString() gives a direct access into the string, but Python doesn't know when the C extension is done with it, and when it's safe to delete the object. Such API prevents to move Python objects in memory (implement a moving garbage collector in Python).

• Disallow dereferencing a PyObject* pointer: most structures must be

opaque. It indirectly means that accessing directly structure members must also be disallowed. PEP 670 and PEP 674 are partially fixing the issues.

(...) fixing these in the API first is probably the way to go.

That's what I already did in the past and what I plan to do in the future.

I see a problem in the subject of this thread: "Slowly bend the C API towards the limited API to get a stable ABI for everyone" is not a good summary of the proposed changes -- that is, bend *all* API (both the general public API and the limited API) to make most structs opaque, etc.

If the summary doesn't match what's actually proposed, it's hard to discuss the proposal. Especially if the concrete plan changes often.

Anyway, I propose a different plan than what I think you are proposing:

- Add new "good" API where the current API is currently lacking. For examrle, PyModule_AddObjectRef is the "good" alternative to PyModule_AddObject -- it returns a strong reference. You've done a lot of great work here, and the API is much better for it.

- "Soft"-deprecate the "bad" API: document that it's only there for existing working code. Why not remove it? The issue is that it *is* possible to use the existing API correctly, and many extension authors have spent a lot of time and effort to do just that. If we force them to use a new API that makes writing correct code easier, it won't actually make their job easier if they've already found the caveats and fixed them.

- Remove the "bad" API from newer versions of the limited API. Extension authors can "opt in" to this new version, gaining new features of the limited API but losing the deprecated parts. (Here is the part where we should make sure the removals are well-documented, provide tools to "modernize" code, etc.)

- Proactively work with popular/important projects (top PyPI packages, distro packages) to move to the latest API. The benefit for a CPython devs here is that we can see the pain points and unforeseen use cases, test any modernization tools, having a "reality check" on how invasive the changes actually are, and helping HPy & other implementations succeed even if they don't implement deprecated API.

- Agree with HPy and other implementations of the limited API that it's not necessary for them to support the deprecated parts.

- When (and only when) a deprecated API is actually harmful -- i.e. it blocks new improvements that benefit actual users in the short term -- it should be deprecated and removed. (Even better, if instead of removing it could be e.g. replaced by a function that's 3x slower, or leaks memory on exit, then it should.)

Basically, instead of "We'll remove this API now because it prevents moving to a hypothetical moving garbage collector", it should be "Here is a moving garbage collector that speeds Python up by 30%, but to add it we need to remove these 30 deprecated APIs". The deprecation can be proactive, but not the removal.

On 07. 02. 22 14:26, Victor Stinner wrote:

On Mon, Feb 7, 2022 at 2:08 PM Petr Viktorin encukou@gmail.com wrote:

...

Basically, instead of "We'll remove this API now because it prevents moving to a hypothetical moving garbage collector", it should be "Here is a moving garbage collector that speeds Python up by 30%, but to add it we need to remove these 30 deprecated APIs". The deprecation can be proactive, but not the removal.

PEP 674 gives 3 concrete examples of issues already affecting the CPython nogil fork, HPy and GraalPython. They are not hypothetical.

HPy and GraalPython are different projects, and their issues don't affect current users of CPython. It would be great to make an easy-to-use API that would be compatible with HPy and GraalPython. But not at the cost of breaking existing code. Are the advantages of moving to HPy or Graalpython worth porting the code? IMO that's a question each extension author should be able to ask. They shouldn't be forced to do the port by CPython.

As for nogil, that's a promising experiment. If it turns out to be successful, let's remove the parts that are blocking it. Do we already know which parts those are?

CPython is also affected by these issues, but the benefits of PEP 674 (alone) are too indirect, so I chose to avoid mentioning CPython issues directly, to avoid confusion.

IMO, CPython issues are the ones most relevant to CPython. If yo're bringing them up, could you be more specific about them? If we don't discuss them, how do we know if there better ways to solve them than what you're proposing?

It's possible to workaround them: more or less copy/paste CPython inefficient code, as PyPy did years ago. The problem is that the workaround is inefficient and so PyPy cpyext remains slow. Well, HPy address the cpyext performance problem for PyPy and GraalPython ;-)

Hooray! So, the issue is addressed, and we don't need to break code that doesn't care about performance on PyPy! Sounds perfect! More seriously: I expect that there are a lot of extension authors that value API stability over performance on PyPy. They're doing nothing wrong and they don't deserve to be punished for using API they had available (and docs they had available) some years ago.

I don't think that the question is if there is a real problem or not. The question is what's the best migration plan to move existing C extensions towards a better API which don't suffer from these problems.

Once 95% of C extensions will use the limited C API, we would still not be able to change Python internals, because of the 5% remaining C extensions which are stuck at the legacy C API.

And don't forget extensions that use the "bad" parts of limited API, like using Py_TYPE as l-value in Python 3.10-. IMO, the required ABI changes will be more drastic than the API changes.

When we want to change CPython internals, IMO we should know the exact set of APIs that'll need to be removed. Without an actual proposed change, I don't think we can know that. HPy is a good indicator of what API should be deprecated (i.e. where we should find a better way of doing things), but I don't think it gives good reasons for breaking changes.

On 07. 02. 22 15:29, Victor Stinner wrote:

On Mon, Feb 7, 2022 at 2:26 PM Victor Stinner vstinner@python.org wrote:

...

CPython is also affected by these issues, but the benefits of PEP 674 (alone) are too indirect, so I chose to avoid mentioning CPython issues directly, to avoid confusion.

A concrete example of problem caused by exposing structures in the C API (unrelated to PEP 674). It's a tricky problem...

typedef struct { PyObject_VAR_HEAD Py_hash_t ob_shash; char ob_sval[1]; } PyBytesObject;

The "char ob_sval[1];" syntax used to declare an array is an undefined behavior if the array is longer in memory. On a bytes object of 4 bytes, accessing ob_sval[3] works, but is an undefined behavior.

=> see https://bugs.python.org/issue40120 for details

The problem can be solved by using "char ob_sval[];" syntax, but we cannot use this syntax in the public C header, since it causes compiler errors if the header is built with a C++ compiler (not to build Python itself, but build a C++ extension using the Python C API). Removing the structure from the public C API would solve the C++ issue.

That sounds like a good candidate for deprecation, and adding proper getters if we didn't have them. But I don't think we actually need to remove the deprecated struct from the public C API.

On 03. 02. 22 1:40, Guido van Rossum wrote: [...]

I understand the CPython is stuck supporting the de-facto standard C API for a long time. But unless we pick a "north star" (as people call it nowadays) of what we want to support in say 5-10 years, the situation will never improve.

My point about "getting one chance to get it right in the next decade" is that we have to pick that north star, so we can tell users which horse to bet on. If the north star we pick is HPy, things will be clear. If it is evolving the C API things will also be clear. But I think we have to pick one, and stick to it so users (i.e., package maintainers/developers) have clarity.

A few months later, here's a draft of a “north star” document. Is this close to what you had in mind?

https://docs.google.com/document/d/1lrvx-ujHOCuiuqH71L1-nBQFHreI8jsXC966AFu9...

Please comment (here or there) as appropriate :)

IMO it would be better to keep the HPy design as the long term goal:

* Refer to Python objects with opaque handles * All structures are opaque (with a few exceptions, like PyType_Spec)

It will likely take multiple iterations (Python releases) to reach this goal, and incompatible C API changes may need a PEP (like PEP 674), but IMO it's good to keep this goal in mind.

Otherwise, it's not easy to understand the rationale for changes like https://peps.python.org/pep-0674/ "PEP 674 – Disallow using macros as l-values".

Victor

On 06. 04. 22 9:23, Tim Felgentreff wrote:

Hi,

(I briefly commented also on the doc regarding this)

I’m probably misinterpreting the exact goals. I read “stable ABI for everyone” and I’m thinking “what needs to happen to stay binary compatible and working for a couple of decades at least”.

Well, the doc is more down to the ground than this thread can be :)

If that were the goal, I think the ideas around HPy’s handles and it’s usage of a context to get access to function pointers are most important, since that will ensure that new APIs can be added and old ones removed without breaking ABI. (This is very similar to how e.g. libraries like SDL deal with the problem that distributors want to update the SDL library and have it still work with 10 year old proprietary games).

In particular, even things like slots in types need to be opaque. An optimizing runtime may want to use varying layouts for both types and objects - allowing direct access into any runtime structures prevents that. Exposed structures should not be used for any runtime objects, only as specs for construction of those runtime objects.

With Python's current general API backwards compatibility policy (PEP 387), it wouldn't make much sense to make ABI guarantees that strong. In reality you're likely to get DeprecationWarnings and runtime exceptions, and will need to recompile extensions well before decades+-scale ABI incompatibilities hit you.

IIUC, adopting HPy would break API: adding a context argument to all functions would would be so massive a change, it'd be easier to just call that HPy rather than Python C-API. And the doc I shared is for the C-API. (Perhaps CPython can even move to HPy and implement C-API as a veener on top -- but I don't think the original C-API can go away in our lifetimes.)

On Tue, 5 Apr 2022 22:54:00 +0200 Victor Stinner vstinner@python.org wrote:

IMO it would be better to keep the HPy design as the long term goal:

• Refer to Python objects with opaque handles
• All structures are opaque (with a few exceptions, like PyType_Spec)

If the HPy design is the long term goal, why not just recommend that people use HPy? And keep the C API for expert users with specific needs that are not accomodated by HPy.

To me, it seems that trying to change the C API to be "like HPy" is creating a lot of work, churn and pain for little gain.

(and, yes, perhaps HPy needs to be funded or supported by the PSF if it doesn't advance fast enough)

Regards

Antoine.

It will likely take multiple iterations (Python releases) to reach this goal, and incompatible C API changes may need a PEP (like PEP 674), but IMO it's good to keep this goal in mind.

Otherwise, it's not easy to understand the rationale for changes like https://peps.python.org/pep-0674/ "PEP 674 – Disallow using macros as l-values".

Victor

On Wed, 20 Apr 2022 12:52:53 +0200 Victor Stinner vstinner@python.org wrote:

On Wed, Apr 20, 2022 at 10:03 AM Antoine Pitrou antoine@python.org wrote:

...

If the HPy design is the long term goal, why not just recommend that people use HPy? And keep the C API for expert users with specific needs that are not accomodated by HPy.

To me, it seems that trying to change the C API to be "like HPy" is creating a lot of work, churn and pain for little gain.

If you put HPy aside, "Fixing" the C API has multiple advantages for CPython and its (C API) users.

With the caveat that the "fixing" probably requires users to fix their packages as well.

For consumers of the C API (C extensions, Cython, pybind11, etc.), once most implementation details will be hidden, the C API will become way more stable.

The *API* is quite stable already if you don't use the private/internal functions. Perhaps you're thinking about the ABI?

...

(and, yes, perhaps HPy needs to be funded or supported by the PSF if it doesn't advance fast enough)

What can be done in practice for that? If I understood correctly, Oracle is sponsoring the project since they want to use HPy for GraalPython (of their GraalVM).

Also, Anaconda recently hired Antonio Cuni, hopefully giving him sufficient time to work on HPy. So perhaps nothing needs to be done in practice.

Regards

Antoine.