Hi,
I modified Include/ header files to have a way more explicit separation between the different levels of Python C API:
Include/*.h is the "portable Python API" supposed to be efficient on CPython and PyPy, and ideally support a maximum number of Python versions
Include/cpython/*.h is the "portable CPython API" which should only be available on CPython (but PyPy had to emulate it for best compatibility with CPython) and so should be avoided if possible.
Include/internal/*.h is the "CPython internal API" which should not be used outside CPython code base. In practice, Cython wants to use it to emit the most efficient code. Debuggers and profiles also want to use it to able to *inspect* a Python process without executing code (don't modify the process). You need to access all structures, especially internal ones, for that.
"Portable" here means "should work on multiple Python versions" like support Python 3.6, 3.7 and 3.8.
... There is also an unclear "non-portable CPython API" which is somewhere between "portable CPython API" and "CPython internal API". I would prefer that this weird API simply goes away if possible: move it into the internal API?
A big change of Python 3.8 is that the "CPython internal API" is now installed by "make install", but accessing it requires to define Py_BUILD_CORE (you have to opt-in for this API). This change allows to move APIs as internal, since they remain available, whereas in Python 3.7 this API was not installed at all. It's a small step towards making more APIs internal.
For the 3 levels of API: I'm writing what we should get **in the long term**. Right now... it's a mess. Previously, we never seriously really looked how a function should be exposed.
For example, some API are exposed as "portable Python API" whereas they really must belong to the "CPython internal API". I recently moved "PyXXX_Fini()" APIs to the "CPython internal API". Outside CPython, it doesn't make any sense to call directly these functions: you really must call Py_Finalize() or Py_FinalizeEx(). In Python 3.7, some of these functions are surrounded by "#ifdef Py_BUILD_CORE", some others are surrounded by "#ifndef Py_LIMITED_API" and their name are not prefixed by "_Py" as if they are public functions... Extract of Python 3.7 Include/pylifecycle.h:
#ifdef Py_BUILD_CORE PyAPI_FUNC(void) _PyExc_Fini(void); PyAPI_FUNC(void) _PyGC_Fini(void); ... #endif /* Py_BUILD_CORE */
#ifndef Py_LIMITED_API PyAPI_FUNC(void) PyMethod_Fini(void); PyAPI_FUNC(void) PyFrame_Fini(void); ... #endif /* !Py_LIMITED_API */
The overall project is still heavily in Work-in-Progress state. It's moving slowly and steadily at least!
For example, Eric Snow is working on making more structures opaque in the "portable CPython API": [Python-Dev] Making PyInterpreterState an opaque type https://mail.python.org/pipermail/python-dev/2019-February/156344.html
Victor
Night gathers, and now my watch begins. It shall not end until my death.