[Python-Dev] Make the stable API-ABI usable

Sat Nov 18 09:13:28 EST 2017

Your email didn't compile. The compiler says that it's a naming conflict,
but actually I think you forgot a semicolon! (Don't worry, it happens to
all of us, whether we be happy or not :-)

--Koos

On Sat, Nov 18, 2017 at 3:05 AM, Victor Stinner <victor.stinner at gmail.com>
wrote:

> Hi,
>
> tl; dr I propose to extend the existing "stable API" to make it almost
> as complete as the current API. For example, add back
> PyTuple_GET_ITEM() to be stable API, but it becomes a function call
> rather than a macro. The final question is if it's not too late to
> iterate on an implementation of this idea for Python 3.7? Knowing that
> the stable API doesn't affect the "current API" at all, since the "new
> C API" (extended stable API) would only be accessible using an
> *opt-in* flag.
>
>
> Since I failed to find time to write a proper PEP (sorry about that),
> and the incoming deadline for Python 3.7 features is getting closer, I
> decided to dump my dump into this email. Sorry about the raw
> formatting.
>
> It's a third turn of feedback. The first one was done in python-ideas
> last July, the second was at the CPython sprint last September at
> Instagram. Both were positive.
>
>    https://mail.python.org/pipermail/python-ideas/2017-July/046399.html
>
> The C API of CPython is amazing. It made CPython as powerful as we
> know it today. Without the C API, Python wouldn't have numpy. Without
> numpy, Python wouldn't be as popular as it is nowadays. Ok, enough for
> the introduction.
>
> The C API is awful. I hate it so much! The C API is old, error prone,
> too big, and expose every single CPython implementation detail... The
> C API is likely the first reason why faster implementation of Python
> (PyPy?) are not as popular as they should be.
>
> The fact that the C API is so widely used prevents many evolutions of
> CPython. CPython is stuck by its own (public) C API.
>
> PyPy developers spent a lot of time to make cffi great and advertize
> it all around the world. While it's a great project, the fact is that
> the C API *is* still used and the motivation to rewrite old code
> written with the C API to cffi, Cython or whateer else is too low.
> Maybe because the carrot is not big enough. Let's make a bigger
> carrot! (bigger than PyPy amazing performances? I'm not sure that it's
> doable :-( we will see)
>
> Using Cython, it's simpler to write C extensions and Cython code is
> more "portable" on different Python versions, since the C API evolved
> the last 10 years. For example, Python 3 renamed PyString with PyBytes
> and dropped the PyInt type. Writing a C extension working on Python 2
> and Python 3 requires to "pollute" the code with many #ifdef. Cython
> helps to reduce them (or even avoid them? sorry, I don't know well
> Cython).
>
> The C *API* is tidely linked to the *ABI*. I tried to explain it with
> an example in my article:
>
>    https://vstinner.github.io/new-python-c-api.html
>
> A known ABI issue is that it's not possible to load C extensions
> compiled in "release mode" on a Python compiled in "debug mode". The
> debug mode changes the API in a subtle way which changes the ABI and
> so makes C extensions incompatible. This issue prevents many people to
> use a Python debug build, whereas debug builds are very helpful to
> detect bugs earlier.
>
> Another issue is that C extensions must be recompiled for each Python
> release (3.5, 3.6, 3.7, etc.). Linux vendors like Red Hat cannot
> provide a single binary for multiple Python versions which prevents to
> upgade the "system" Python in the lifecycle of a distribution major
> version (or at least, it makes things much more complicated in term of
> packaging, it multiply packages for each Python version...).
>
> .
> .
> .
>
> Don't worry (be happy!), I'm not writing this email to complain, but
> to propose a solution. Aha!
>
> I propose to modify the API step by step to add more functions to the
> "stable ABI" (PEP 384) to be able to compile more and more C
> extensions using the "stable API" (please try to follow, first it was
> B, now it's P... seriously, the difference between the ABI and the API
> is subtle, to simplify let's say that it's the same thing, ok? :-)).
>
> I wrote a draft PEP, but I never found time to update it after the two
> rounds of feedbacks (python-ideas and the sprint) to write a proper
> PEP. So I will only give a link to my draft, sorry!
>
>   https://github.com/vstinner/misc/blob/master/python/pep_c_api.rst
>
> In short, the plan is to add back the PyTuple_GET_ITEM() *API* to the
> "stable API" but change its implementation to a *function call* rather
> than the existing macro, so the compiled C extension will use a
> function call and so don't rely on the ABI anymore.
>
>
> My plan is to have two main milestones:
>
> (1) Python 3.7: Extend the *existing* opt-in "stable API" which
> requires to compile C extensions in a special mode. Add maybe an
> option in distutils to ease the compilation of a C extension with the
> "stable API"?
>
> (2) In Python 3.8, --if the project is successful and the performance
> overhead is acceptable compared the advantages of having C extensions
> working on multiple Python verisons--, make the "stable API (without
> implementation details)" the default, but add a new opt-in option to
> give access to the "full API (with implementation details)" for
> debuggers and other people who understand what they do (like Cython?).
>
> Note: currently, the "stable API" is accessible using Py_LIMITED_API
> define, and the "full API" is accessible using Py_BUILD_CORE define.
> No define gives the current C API.
>
>
> My problem is more on the concrete implementation:
>
> * Need to provide two different API using the same filenames (like:
> #include "Python.h")
>
> * Need to extend distutils to have a flag to compile a C extension
> with one specific API (define Py_LIMITED_API or Py_BUILD_CORE?)
>
> * Need to test many C extensions and check how many extensions are broken
>
>
> My plan for Python 3.7 is to not touch the current API at all. There
> is no risk of backward incompatibility. You should only get issues if
> you opt-in for the new API without implementation details.
>
>
> Final note: Nothing new under the sun: PyPy already implemented my
> "idea"! Where the idea is a C API without macros; PyTuple_GET_ITEM()
> is already a function call in PyPy ;-)
>
>
> Final question: Is it acceptable to iterate on many small changes on
> the C API implement this idea in Python 3.7? Maybe only write partial
> implementation, and finish it in Python 3.8?
>
> Victor
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> k7hoven%40gmail.com
>

-- 
+ Koos Zevenhoven + http://twitter.com/k7hoven +
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20171118/8e49366b/attachment.html>