On Wed, Apr 20, 2022 at 10:03 AM Antoine Pitrou <antoine@python.org> wrote:
If the HPy design is the long term goal, why not just recommend that people use HPy? And keep the C API for expert users with specific needs that are not accomodated by HPy.
To me, it seems that trying to change the C API to be "like HPy" is creating a lot of work, churn and pain for little gain.
If you put HPy aside, "Fixing" the C API has multiple advantages for CPython and its (C API) users. For consumers of the C API (C extensions, Cython, pybind11, etc.), once most implementation details will be hidden, the C API will become way more stable. The "C API > Porting to Python x.y" section of What's New in Python x.y should become way shorter. Or at least, the number of impacted C extensions should be way smaller. Sadly, fixing the C API to hide implementation details requires to adapt (modify) C extensions. Even if usually only a few lines should be changed, and the pythoncapi-compat project now automates most of these changes. For CPython, no longer leaking implementation details allow to change "any implementation detail" without getting a heavy and annoying backfire from grumpy users should be very comfortable. In Python 3.9, 3.10 and 3.11 development cycles, we got backfire multiple times, and each time, it was really unpleasant both for CPython core devs and for C extensions maintainers (both have legit concerns and use cases). While these changes should ease the migration to HPy, it's not my goal. HPy requires to add a "ctx" parameter, it's a different API (there are multiple subtle differences).
(and, yes, perhaps HPy needs to be funded or supported by the PSF if it doesn't advance fast enough)
What can be done in practice for that? If I understood correctly, Oracle is sponsoring the project since they want to use HPy for GraalPython (of their GraalVM). Victor -- Night gathers, and now my watch begins. It shall not end until my death.