Victor Stinner schrieb am 29.07.2018 um 21:46:
Last year, I gave a talk at the Language Summit (during Pycon) to explain that CPython should become 2x faster to remain competitive. IMHO all attempts to optimize Python (CPython forks) have failed because they have been blocked by the C API which implies strict constraints.
I started to write a proposal to change the C API to hide implementation details, to prepare CPython for future changes. It allows to experimental optimization ideas without loosing support for C extensions.
From Cython's POV, exposing internals is a good thing that helps making extension modules faster. *Hiding* internals would already break code, so I don't see the advantage over just *changing* the internals instead, but continuing to expose those new internals. The problem we have is the heap of C extensions that are no longer (actively) maintained, not those that are maintained but use internals.
Basically, PyPy shows that, given enough developer time, the C-API can be emulated even based on a very different runtime design, potentially with a handicap in terms of performance. If some parts of the C-API are to be replaced, that might be a way to go.
I also don't buy the argument that binary modules built for, say, Py3.6 must continue to import on Py3.9, for example. Supporting the last couple of supported releases with binary wheels has proven good enough IMHO, and rebuilding for a new CPython release seems acceptable, given that this also enables the use of new features. (Would be something to ask distributors, though.)
So, from my POV, I'd vote for
- allowing C-API changes in each X.Y release
- focussing on the API and not the ABI
- requiring a new binary wheel (or rebuild) for each X.Y release
- providing a compatibility layer for "removed" C-API functionality
- exposing any internals that may help extension modules
- maybe add a warning to the docs of exposed internals that these are more likely to change than other parts of the C-API
I'd also suggest to make Cython, pybind11 and cffi (maybe a few more) the preferred and official ways to extend and integrate with CPython, to keep those three up to date with all C-API changes, and to make it as easy as possible for users to build their code with them against new CPython releases.
If you want a more radical proposal, I'd deprecate the C-API documentation, push people into not caring about the C-API themselves, and then concentrate on keeping the major code integration tools out there compatible and fast with whatever CPython can provide as "exposed internals".
Stefan