[Python-ideas] PEP: Hide implementation details in the C API

[Nick Coghlan]

On 13 July 2017 at 08:23, Stefan Behnel <stefan_ml at behnel.de> wrote:
> The general rule of thumb in Cython core development is that it's ok to
> exploit internals as long as there is a generic fallback through some C-API
> operations which can be used in other Python implementations. I'd be happy
> if that continued to be supported by CPython in the future. Exposing
> CPython internals is a good thing! :)

+1

This is my major motivation for suggesting "Include/cpython/" as the
directory for the header files that define a supported API that is
specific to CPython - it helps make it clear to other implementations
that it's OK to go beyond the portable Python C API, but such API
extensions should be clearly flagged as implementation specific so
that consumers can make an informed decision as to which level they
want to target.

I do want to revise my naming suggestions slightly though: I think it
would make sense for the internal APIs (the ones already guarded by
Py_BUILD_CORE) to be under "Include/_core/", where the leading
underscore helps to emphasise "if you are not working on CPython
itself, you should not be going anywhere near these header files".

I think the other key point to clarify will be API versioning, since
that will flow through to things like the C ABI compatibility tags in
the binary wheel format. Currently [1], that looks like:

    cp35m # Specifically built for CPython 3.5 with PyMalloc
    cp35dm # Debugging enabled
    cp3_10m # Disambiguation uses underscores
    pp18 # It's the implementation version, not the Python version

There's currently only one tag for the stable ABI:

    abi3 # Built for the stable ABI as of Python 3.2

So I think the existing Py_LIMITED_API/stable ABI is the right place
for the strict "No public structs!" policy that completely decouples
extension modules from CPython internals. We'll just need to refine
the definition of the compatibility tags so that folks can properly
indicate the minimum required version of that API:

    abi3 # Py_LIMITED_API=0x03020000
    abi32 # Py_LIMITED_API=0x03020000
    abi33 # Py_LIMITED_API=0x03030000
    abi34 # Py_LIMITED_API=0x03040000
    abi35 # Py_LIMITED_API=0x03050000
    etc...

Where Py_PORTABLE_API would come in is that it could be less strict on
the "no public structs" rule (allowing some structs to be exposed as
needed to enable building key projects like NumPy and lxml), and
instead represent an API that offered source code and extension module
portability across Python implementations, rather than strict ABI
stability across versions.

    api37 # Py_PORTABLE_API=0x03070000
    api38 # Py_PORTABLE_API=0x03080000
    api39 # Py_PORTABLE_API=0x03090000
    api3_10 # Py_PORTABLE_API=0x030A0000
    etc...

Cheers,
Nick.

[1] https://www.python.org/dev/peps/pep-0425/#details

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia