Hi,

I pushed many changes in Python 3.9 to hide implementation details:

• Add getter and setter functions for PyFrameObject, PyThreadState and PyInterpreterState structures. Examples: PyFrame_GetCode() (frame->f_code) and PyThreadState_GetInterpreter() (tstate->interp); PyFrame_GetCode() returns a strong reference (not a borrowed reference).
• Move private functions to the internal C API.
• etc.

See https://docs.python.org/dev/whatsnew/3.9.html#c-api-changes for the full list.

My overall project is described in this draft PEP: https://github.com/vstinner/misc/blob/master/cpython/pep-opaque-c-api.rst

I created multiple issues to prepare the C API to make more structures opaque:

• PyGC_Head: https://bugs.python.org/issue40241
• PyObject: https://bugs.python.org/issue39573
• PyTypeObject: https://bugs.python.org/issue40170
• PyThreadState: https://bugs.python.org/issue39573
• PyInterpreterState: https://bugs.python.org/issue35886 (done in Python 3.8)

Related to that, Dong-hee Na and me pushed changes to prevent using Py_TYPE(), Py_REFCNT() and Py_SIZE() as l-value. "Py_TYPE(obj) = new_type" must now be written "Py_SET_TYPE(obj, new_type);". I updated Cython for that: https://github.com/cython/cython/commit/63ab284df20cfe99638f3f352080648c8a3b...

Many changes are somehow backward incompatible. Some are more incompatible than others. It's not easy for me to estimate how many C extensions will be broken by these incompatible changes.

So far, it was quite easy to update C extensions to Python 3.9.0 beta1. One project call directly PyImport_Cleanup() which was described as an "internal" function. This function was moved to the internal C API under _PyImport_Cleanup(). You should not call it directly!

In case of big troubles, there is always the ability to revert some incompatible changes, to fix broken projects, before re-applying these changes.

I know that these incompatible changes are annoying, but I consider that we have to go through this painful road to enhance (optimize) CPython, but also other Python implementations like PyPy, IronPython or GraalPython.

The HPy project proposes a whole new C API written correctly from the beginning. It is a different approach, but it doesn't solve the problem of the big number of existing C extensions which will take several months or years to be ported to HPy. So far, HPy doesn't offer any automated conversion from the current C API to the new HPy C API. But my goal is that CPython C API and HPy C API will converge somehow in the long term. Each small change that I'm making is moving the current C API a little bit closer to HPy!

By the way, use Cython for *new* C extension, don't use the C API directly! It's already said in the CPython C API documentation! These C API changes are targeting existing C extensions written with the C API ;-)

Victor

Night gathers, and now my watch begins. It shall not end until my death.