On 13 July 2017 at 08:23, Stefan Behnel email@example.com wrote:
The general rule of thumb in Cython core development is that it's ok to exploit internals as long as there is a generic fallback through some C-API operations which can be used in other Python implementations. I'd be happy if that continued to be supported by CPython in the future. Exposing CPython internals is a good thing! :)
This is my major motivation for suggesting "Include/cpython/" as the directory for the header files that define a supported API that is specific to CPython - it helps make it clear to other implementations that it's OK to go beyond the portable Python C API, but such API extensions should be clearly flagged as implementation specific so that consumers can make an informed decision as to which level they want to target.
I do want to revise my naming suggestions slightly though: I think it would make sense for the internal APIs (the ones already guarded by Py_BUILD_CORE) to be under "Include/_core/", where the leading underscore helps to emphasise "if you are not working on CPython itself, you should not be going anywhere near these header files".
I think the other key point to clarify will be API versioning, since that will flow through to things like the C ABI compatibility tags in the binary wheel format. Currently , that looks like:
cp35m # Specifically built for CPython 3.5 with PyMalloc cp35dm # Debugging enabled cp3_10m # Disambiguation uses underscores pp18 # It's the implementation version, not the Python version
There's currently only one tag for the stable ABI:
abi3 # Built for the stable ABI as of Python 3.2
So I think the existing Py_LIMITED_API/stable ABI is the right place for the strict "No public structs!" policy that completely decouples extension modules from CPython internals. We'll just need to refine the definition of the compatibility tags so that folks can properly indicate the minimum required version of that API:
abi3 # Py_LIMITED_API=0x03020000 abi32 # Py_LIMITED_API=0x03020000 abi33 # Py_LIMITED_API=0x03030000 abi34 # Py_LIMITED_API=0x03040000 abi35 # Py_LIMITED_API=0x03050000 etc...
Where Py_PORTABLE_API would come in is that it could be less strict on the "no public structs" rule (allowing some structs to be exposed as needed to enable building key projects like NumPy and lxml), and instead represent an API that offered source code and extension module portability across Python implementations, rather than strict ABI stability across versions.
api37 # Py_PORTABLE_API=0x03070000 api38 # Py_PORTABLE_API=0x03080000 api39 # Py_PORTABLE_API=0x03090000 api3_10 # Py_PORTABLE_API=0x030A0000 etc...