Hi,
I pushed many changes in Python 3.9 to hide implementation details:
* Add getter and setter functions for PyFrameObject, PyThreadState and
PyInterpreterState structures. Examples: PyFrame_GetCode()
(frame->f_code) and PyThreadState_GetInterpreter() (tstate->interp);
PyFrame_GetCode() returns a strong reference (not a borrowed
reference).
* Move private functions to the internal C API.
* etc.
See https://docs.python.org/dev/whatsnew/3.9.html#c-api-changes for
the full list.
My overall project is described in this draft PEP:
https://github.com/vstinner/misc/blob/master/cpython/pep-opaque-c-api.rst
I created multiple issues to prepare the C API to make more structures opaque:
* PyGC_Head: https://bugs.python.org/issue40241
* PyObject: https://bugs.python.org/issue39573
* PyTypeObject: https://bugs.python.org/issue40170
* PyThreadState: https://bugs.python.org/issue39573
* PyInterpreterState: https://bugs.python.org/issue35886 (done in Python 3.8)
Related to that, Dong-hee Na and me pushed changes to prevent using
Py_TYPE(), Py_REFCNT() and Py_SIZE() as l-value. "Py_TYPE(obj) =
new_type" must now be written "Py_SET_TYPE(obj, new_type);". I updated
Cython for that:
https://github.com/cython/cython/commit/63ab284df20cfe99638f3f352080648c8a3…
Many changes are somehow backward incompatible. Some are more
incompatible than others. It's not easy for me to estimate how many C
extensions will be broken by these incompatible changes.
So far, it was quite easy to update C extensions to Python 3.9.0
beta1. One project call directly PyImport_Cleanup() which was
described as an "internal" function. This function was moved to the
internal C API under _PyImport_Cleanup(). You should not call it
directly!
In case of big troubles, there is always the ability to revert some
incompatible changes, to fix broken projects, before re-applying these
changes.
I know that these incompatible changes are annoying, but I consider
that we have to go through this painful road to enhance (optimize)
CPython, but also other Python implementations like PyPy, IronPython
or GraalPython.
The HPy project proposes a whole new C API written correctly from the
beginning. It is a different approach, but it doesn't solve the
problem of the big number of existing C extensions which will take
several months or years to be ported to HPy. So far, HPy doesn't offer
any automated conversion from the current C API to the new HPy C API.
But my goal is that CPython C API and HPy C API will converge somehow
in the long term. Each small change that I'm making is moving the
current C API a little bit closer to HPy!
By the way, use Cython for *new* C extension, don't use the C API
directly! It's already said in the CPython C API documentation! These
C API changes are targeting existing C extensions written with the C
API ;-)
Victor
--
Night gathers, and now my watch begins. It shall not end until my death.
