On 1/29/21 7:43 PM, Victor Stinner wrote:
Hi,
Yeah, the stable ABI is not tested, maintained manually and errors are very common :-( Something should be done to simply its maintenance and reduce the risk of manual errors. This PEP looks like a great step forward!
If we want to optimize CPython (which remains an open question for 5 years at least!), the limited C API and the stable ABI must evolve. I tried to write down technical reasons to explain why: https://pythoncapi.readthedocs.io/optimize_python.html
For example, the ability to dereference a PyObject* to access PyObject members prevents implementing a moving garbage collector.
"Evolving" here means "introducing incompatible changes on purpose" which goes against the principle of a "stable" API :-( If we break the API once, maybe we should try take in account all currently known issues and try to address all of them at once. The problem is that the current C API is based on "PyObject*". Moving away from PyObject* would be a radical change (more or less rewrite the whole C API). So it doesn't seem possible to fix the API "at once". (I'm not talking about the stable ABI here, but the API.)
I think "introducing incompatible changes on purpose" makes Python worse without much benefit. Has any of the recent incompatible changes actually made Python faster?
Besides the non-opaque PyObject*, the worst thing that prevents "evolution" now is IMO that built-in static types like PyList_Type are exposed in the API and stable ABI as global symbols. This prevents making the GIL per-interpreter. Here's how stuff like that can be changed:
Formalize, test and document the stable ABI
Introduce new API (e.g. PyList_GetListType())
Deprecate the old API
Create
abi4, a new, incompatible version of the stable ABI, which doesn't include deprecated stuff. Create a flag to mark extension modules that use it.Make it possible to create subinterpreters with a separate GIL. These can only load
abi4extension modules.Work with various third-party projects to switch to abi4 so we can start getting its benefits, and improve abi4 along the way.
?. In Python 4.0, drop abi3.
The HPy project tries to design a new C API from scratch which doesn't inherit these design issues. Extensions written with HPy are faster on PyPy than the same existing written with the Python C API, and HPy is as fast as the Python C API on CPython. HPy C API is incompatible with the existing Python C API but the migration should not be "too hard" (I didn't try, so I cannot testify). HPy design should also provide a stable ABI (but the project is not mature yet). One advantage is that HPy doesn't require to touch CPython, it's developed externally.
HPy is still young since it's uncertain to only bet on it. That's why I'm modifying the C API in CPython (hide implementation details: PEP 620) in the meanwhile to make it "less broken" (prepare it for future optimizations).
PEP 620 has many things, some of which conflict with the accepted PEP 387. I don't think all of it is a good plan (and I guess that's why it wasn't accepted).
It would be great if optimizing Python and making the stable ABI would not be two "incompatible" goals. But the practical question is more "do we want to fix a few issues with minor incompatible changes in the short term?" or "do we want to fix all issues with major incompatible changes at once for the long term?". Since HPy is developed externally, it doesn't seem incompatible to fix the limited C API and the stable ABI in CPython and develop HPy in parallel.
About the PEP, maybe it would be interesting to explain that it doesn't try to solve all issues at once for practical reasons, and that projects like HPy can be solutions for that.
I don't think it's worth it to mention that the PEP doesn't try to solve all issues at once. Of course it doesn't :)