[Python-Dev] Experiment an opt-in new C API for Python? (leave current API unchanged)

[Victor Stinner]

In short, you don't have to modify your C extensions and they will
continue to work as before on Python 3.8.

I only propose to add a new C API, don't touch the existing one in any
way. Introducing backward incompatible changes to the existing C API
is out of my plan.

/usr/bin/python3.8 will support C extensions compiled with the old C
API and C extensions compiled with the new C API.

My plan also includes to be able to write C extensions compatible with
the old and new C API in a single code base. As we have Python code
working nicely on Python 2 and Python 3 (thanks to six, mostly). In my
experience, having two branches or two repositories for two flavors of
the same code is a nice recipe towards inconsistent code and painful
workflow.

Le mer. 14 nov. 2018 à 15:53, Paul Moore <p.f.moore at gmail.com> a écrit :
> It occurs to me that we may be talking at cross purposes. I noticed
> https://pythoncapi.readthedocs.io/backward_compatibility.html#forward-compatibility-with-python-3-8-and-newer
> which seems to be saying that 3rd party code *will* need to change for
> 3.8.

Oh. It's badly explained in that case. This section is only about C
extensions which really want to become compatible with the new C API.

> You mention removed functions there, so I guess "stop using the
> removed functions and you'll work with 3.8+ and <=3.7" is the
> compatible approach - but it doesn't offer a way for projects that
> *need* the functionality that's been removed to move forward.

If you need a removed functions, don't use the new C API.

> So to try to be clear, your proposal is that in 3.8:
>
> 1. The existing C API will remain
> 2. A new C API will be *added* that 3rd party projects can use should
> they wish to.

Yes, that's it. Add a new API, don't touch existing API.

> And in 3.9 onwards, both C APIs will remain, maybe with gradual and
> incremental changes that move users of the existing C API closer and
> closer to the new one (via deprecations, replacement APIs etc as per
> our normal compatibility rules).

Honestly, it's too early to say if we should modify the current C API
in any way.

I only plan to put advices in the *documentation*. Something like
"this function is really broken, don't use it" :-) Or "you can use xxx
instead which makes your code compatible with the new C API". But I
don't plan it to modify the doc soon. It's too early at this point.

> Or is the intention that at *some*
> point there will be a compatibility break and the existing API will
> simply be removed in favour of the "new" API?

That's out of the scope of *my* plan.

Maybe someone else will show up in 10 years and say "ok, let's
deprecate the old C API". But in my experience, legacy stuff never
goes away :-) (Python 2, anyone?)

> The above is clear, but I don't see what incentive there is in that
> scenario for anyone to actually migrate to the new API...

https://pythoncapi.readthedocs.io/ tries to explain why you should
want to be compatible with the new C API.

The main advantage of the new C API is to compile your C extension
once and use it on multiple runtimes:

* use PyPy for better performances (better than with the old C API)
* use a Python Debug Runtime which contains additional runtime checks
to detect various kinds of bugs in your C extension
* distribute a single binary working on multiple Python versions
(compile on 3.8, use it on 3.9): "stable ABI" -- we are no there yet,
I didn't check what should be done in practice for that

I hope that "later" we will get a faster CPython using <put amazing
new optimizations there>, only compatible with C extensions compiled
with the new C API. My secret hope is that it should ease the
experimentation of a (yet another) JIT compiler for CPython :-)

Victor