[Python-ideas] PEP: Hide implementation details in the C API

[Stefan Behnel]

Victor Stinner schrieb am 11.07.2017 um 12:19:
> Split the ``Include/`` directory of CPython:
> 
> * ``python`` API: ``Include/Python.h`` remains the default C API
> * ``core`` API: ``Include/core/Python.h`` is a new C API designed for
>   building Python
> * ``stable`` API: ``Include/stable/Python.h`` is the stable ABI
> [...]
> Step 3: first pass of implementation detail removal
> ---------------------------------------------------
> 
> Modify the ``python`` API:
> 
> * Add a new ``API`` subdirectory in the Python source code which will
>   "implement" the Python C API
> * Replace macros with functions. The implementation of new functions
>   will be written in the ``API/`` directory. For example, Py_INCREF()
>   becomes the function ``void Py_INCREF(PyObject *op)`` and its
>   implementation will be written in the ``API`` directory.
> * Slowly remove more and more implementation details from this API.

>From a Cython perspective, it's (not great but) ok if these "implementation
details" were moved somewhere else, but it would be a problem if they
became entirely unavailable for external modules. Cython uses some of the
internals for performance reasons, and we adapt it to changes of these
internals whenever necessary.

The question then arises if this proposal fulfills its intended purpose if
Cython based tools like NumPy or lxml continued to use internal
implementation details in their Cython generated C code. Specifically
because that code is generated, I find it acceptable that it actively
exploits non-portable details, because it already takes care of adapting to
different Python platforms anyway. Cython has incorporated support for
CPython, PyPy and Pyston that way, adding others is probably not difficult,
and optimising for a specific one (usually CPython) is also easy.

The general rule of thumb in Cython core development is that it's ok to
exploit internals as long as there is a generic fallback through some C-API
operations which can be used in other Python implementations. I'd be happy
if that continued to be supported by CPython in the future. Exposing
CPython internals is a good thing! :)

Stefan