[Python-Dev] Make the stable API-ABI usable

Victor Stinner victor.stinner at gmail.com
Fri Nov 17 20:05:06 EST 2017


Hi,

tl; dr I propose to extend the existing "stable API" to make it almost
as complete as the current API. For example, add back
PyTuple_GET_ITEM() to be stable API, but it becomes a function call
rather than a macro. The final question is if it's not too late to
iterate on an implementation of this idea for Python 3.7? Knowing that
the stable API doesn't affect the "current API" at all, since the "new
C API" (extended stable API) would only be accessible using an
*opt-in* flag.


Since I failed to find time to write a proper PEP (sorry about that),
and the incoming deadline for Python 3.7 features is getting closer, I
decided to dump my dump into this email. Sorry about the raw
formatting.

It's a third turn of feedback. The first one was done in python-ideas
last July, the second was at the CPython sprint last September at
Instagram. Both were positive.

   https://mail.python.org/pipermail/python-ideas/2017-July/046399.html

The C API of CPython is amazing. It made CPython as powerful as we
know it today. Without the C API, Python wouldn't have numpy. Without
numpy, Python wouldn't be as popular as it is nowadays. Ok, enough for
the introduction.

The C API is awful. I hate it so much! The C API is old, error prone,
too big, and expose every single CPython implementation detail... The
C API is likely the first reason why faster implementation of Python
(PyPy?) are not as popular as they should be.

The fact that the C API is so widely used prevents many evolutions of
CPython. CPython is stuck by its own (public) C API.

PyPy developers spent a lot of time to make cffi great and advertize
it all around the world. While it's a great project, the fact is that
the C API *is* still used and the motivation to rewrite old code
written with the C API to cffi, Cython or whateer else is too low.
Maybe because the carrot is not big enough. Let's make a bigger
carrot! (bigger than PyPy amazing performances? I'm not sure that it's
doable :-( we will see)

Using Cython, it's simpler to write C extensions and Cython code is
more "portable" on different Python versions, since the C API evolved
the last 10 years. For example, Python 3 renamed PyString with PyBytes
and dropped the PyInt type. Writing a C extension working on Python 2
and Python 3 requires to "pollute" the code with many #ifdef. Cython
helps to reduce them (or even avoid them? sorry, I don't know well
Cython).

The C *API* is tidely linked to the *ABI*. I tried to explain it with
an example in my article:

   https://vstinner.github.io/new-python-c-api.html

A known ABI issue is that it's not possible to load C extensions
compiled in "release mode" on a Python compiled in "debug mode". The
debug mode changes the API in a subtle way which changes the ABI and
so makes C extensions incompatible. This issue prevents many people to
use a Python debug build, whereas debug builds are very helpful to
detect bugs earlier.

Another issue is that C extensions must be recompiled for each Python
release (3.5, 3.6, 3.7, etc.). Linux vendors like Red Hat cannot
provide a single binary for multiple Python versions which prevents to
upgade the "system" Python in the lifecycle of a distribution major
version (or at least, it makes things much more complicated in term of
packaging, it multiply packages for each Python version...).

.
.
.

Don't worry (be happy!), I'm not writing this email to complain, but
to propose a solution. Aha!

I propose to modify the API step by step to add more functions to the
"stable ABI" (PEP 384) to be able to compile more and more C
extensions using the "stable API" (please try to follow, first it was
B, now it's P... seriously, the difference between the ABI and the API
is subtle, to simplify let's say that it's the same thing, ok? :-)).

I wrote a draft PEP, but I never found time to update it after the two
rounds of feedbacks (python-ideas and the sprint) to write a proper
PEP. So I will only give a link to my draft, sorry!

  https://github.com/vstinner/misc/blob/master/python/pep_c_api.rst

In short, the plan is to add back the PyTuple_GET_ITEM() *API* to the
"stable API" but change its implementation to a *function call* rather
than the existing macro, so the compiled C extension will use a
function call and so don't rely on the ABI anymore.


My plan is to have two main milestones:

(1) Python 3.7: Extend the *existing* opt-in "stable API" which
requires to compile C extensions in a special mode. Add maybe an
option in distutils to ease the compilation of a C extension with the
"stable API"?

(2) In Python 3.8, --if the project is successful and the performance
overhead is acceptable compared the advantages of having C extensions
working on multiple Python verisons--, make the "stable API (without
implementation details)" the default, but add a new opt-in option to
give access to the "full API (with implementation details)" for
debuggers and other people who understand what they do (like Cython?).

Note: currently, the "stable API" is accessible using Py_LIMITED_API
define, and the "full API" is accessible using Py_BUILD_CORE define.
No define gives the current C API.


My problem is more on the concrete implementation:

* Need to provide two different API using the same filenames (like:
#include "Python.h")

* Need to extend distutils to have a flag to compile a C extension
with one specific API (define Py_LIMITED_API or Py_BUILD_CORE?)

* Need to test many C extensions and check how many extensions are broken


My plan for Python 3.7 is to not touch the current API at all. There
is no risk of backward incompatibility. You should only get issues if
you opt-in for the new API without implementation details.


Final note: Nothing new under the sun: PyPy already implemented my
"idea"! Where the idea is a C API without macros; PyTuple_GET_ITEM()
is already a function call in PyPy ;-)


Final question: Is it acceptable to iterate on many small changes on
the C API implement this idea in Python 3.7? Maybe only write partial
implementation, and finish it in Python 3.8?

Victor


More information about the Python-Dev mailing list