On Fri, Jan 28, 2022 at 6:28 PM Guido van Rossum <guido@python.org> wrote:
I think we will get *one* chance in the next decade to get it right. Whether that's HPy or evolution of the C API I'm not sure.
Would you mind to elaborate? Which risk do you expect from switching to HPy and from fixing the C API (introducing incompatible C API changes)? For me, promoting HPy and evolution of the C API are complementary, can and must done in parallel for me. As I explained in PEP 674, while HPy does help C extensions writers, it doesn't solve any problem for CPython right now. CPython is still blocked by implementation details leaked throught the C API that we must still maintain for a few more years.
Victor, am I right that the (some) stable ABI will remain important because projects don't have resources to build wheels for every Python release? If a project does R releases per year for P platforms that need to support V versions of Python, they would normally have to build R * P * V wheels. With a stable ABI, they could reduce that to R * P. That's the key point, right?
There are different use cases. 1) First, my main worry is that we put a high pressure on maintainers of most important Python dependencies before the next of a new Python version, because we want them to handle the flow of incompatible C API changes before the final Python 3.x versions is released, to get them available when Python 3.x final is released. It annoys core developers who cannot change things in Python without getting an increasing number of complains about a large number of broken packages, sometimes with a request to revert. It annoys C extensions maintainers who have to care about Python alpha and beta releases which are not convenient to use (ex: not available in Linux distributions). Moreover, it became common to ask multiple changes and multiple releases before a Python final release, since more incompatible changes are introduced in Python (before the beta1). 2) Second, as you said, the stable ABI reduces the number of binary packages which have to be built. Small projects with a little team (ex: a single person) don't have resources to set up a CI and maintain it to build all these packages. It's doable, but it isn't free. -- The irony of the situation is that we must break the C API (hiding structures is technically an incompatible change)... to make the C API stable. Breaking it now to make it stable later. We already broke the C API many times in the past. The difference here is that changes are done in the purpose of bending it towards the limited C API and the stable ABI. My expectation is that replacing frame->f_code with PyFrame_GetCode() only has to be done exactly once: this API is not going this change. Sadly, the changes are not limited to frame->f_code, more changes are needed. For example, for PyFrameObject, accesses to every structure member must have to go through a function call (getter or setter function). Hopefully, only a small number of member are used by C extensions. The tricky part is to think about the high level API ("use cases") rather than just adding functions doing "return struct->member" and "struct->member = new_value". The PyThreadState_EnterTracing() and PyThreadState_LeaveTracing() functions added to Python 3.11 are a good example: the API is "generic" and the implementation changes 2 structure members, not a single one. In practice, what I did since Python 3.8 is to introduce a small number of C API changes per Python versions. We tried the "fix all the things at once" approach (!!!) with Python 3, and it... didn't go well. All C extensions had to suddenly write their own compatibility layer for a large number of C API functions (ex: replace PyInt_xxx with PyLong_xxx, without losing Python 2 support!). The changes that I'm introducing in the C API usually impact less than 100 extensions in total (usually, I would say between 10 and 25 per Python version, but it's hard to measure exactly).
Can HPy do that?
I wish more projects are incrementally rewritten with Cython, cffi, pybind11 and HPy, and so slowly move away using directly the C API. Yes, HPy support an "universal build" mode which allows to only build a C extension once, and use it on multiple *CPython* versions *and* (that's the big news!) multiple *PyPy* versions! I even heard that it also brings GraalPython support for free ;-) Victor -- Night gathers, and now my watch begins. It shall not end until my death.