Hi,
I pushed many changes in Python 3.9 to hide implementation details:
- Add getter and setter functions for PyFrameObject, PyThreadState and PyInterpreterState structures. Examples: PyFrame_GetCode() (frame->f_code) and PyThreadState_GetInterpreter() (tstate->interp); PyFrame_GetCode() returns a strong reference (not a borrowed reference).
- Move private functions to the internal C API.
- etc.
See https://docs.python.org/dev/whatsnew/3.9.html#c-api-changes for the full list.
My overall project is described in this draft PEP: https://github.com/vstinner/misc/blob/master/cpython/pep-opaque-c-api.rst
I created multiple issues to prepare the C API to make more structures opaque:
- PyGC_Head: https://bugs.python.org/issue40241
- PyObject: https://bugs.python.org/issue39573
- PyTypeObject: https://bugs.python.org/issue40170
- PyThreadState: https://bugs.python.org/issue39573
- PyInterpreterState: https://bugs.python.org/issue35886 (done in Python 3.8)
Related to that, Dong-hee Na and me pushed changes to prevent using Py_TYPE(), Py_REFCNT() and Py_SIZE() as l-value. "Py_TYPE(obj) = new_type" must now be written "Py_SET_TYPE(obj, new_type);". I updated Cython for that: https://github.com/cython/cython/commit/63ab284df20cfe99638f3f352080648c8a3b...
Many changes are somehow backward incompatible. Some are more incompatible than others. It's not easy for me to estimate how many C extensions will be broken by these incompatible changes.
So far, it was quite easy to update C extensions to Python 3.9.0 beta1. One project call directly PyImport_Cleanup() which was described as an "internal" function. This function was moved to the internal C API under _PyImport_Cleanup(). You should not call it directly!
In case of big troubles, there is always the ability to revert some incompatible changes, to fix broken projects, before re-applying these changes.
I know that these incompatible changes are annoying, but I consider that we have to go through this painful road to enhance (optimize) CPython, but also other Python implementations like PyPy, IronPython or GraalPython.
The HPy project proposes a whole new C API written correctly from the beginning. It is a different approach, but it doesn't solve the problem of the big number of existing C extensions which will take several months or years to be ported to HPy. So far, HPy doesn't offer any automated conversion from the current C API to the new HPy C API. But my goal is that CPython C API and HPy C API will converge somehow in the long term. Each small change that I'm making is moving the current C API a little bit closer to HPy!
By the way, use Cython for *new* C extension, don't use the C API directly! It's already said in the CPython C API documentation! These C API changes are targeting existing C extensions written with the C API ;-)
Victor
Night gathers, and now my watch begins. It shall not end until my death.