[Python-Dev] Making PyInterpreterState an opaque type

[Stefan Behnel]

Nick Coghlan schrieb am 19.02.19 um 15:00:
> On Tue, 19 Feb 2019 at 20:41, Antoine Pitrou wrote:
>> On Mon, 18 Feb 2019 19:04:31 -0800 Steve Dower wrote:
>>> If you always rebuild your extension for every micro version (3.x.y) of
>>> CPython, then sure, go ahead and use this.
>>
>> Usually we would guarantee that API details don't change in bugfix
>> versions (i.e. the 3.x.y -> 3.x.(y + 1) transition).  Has that changed?
>> That may turn out a big problem for several third-party extensions...
> 
> This is the genuine technical difference between the three levels:
> 
> * Py_BUILD_CORE -> no ABI stability guarantees at all
> * standard -> stable within a maintenance branch
> * Py_LIMITED_API -> stable across feature releases

I'm happy with this split, and i think this is how it should be. There is
no reason (not withstanding critical bugs) to break the C-API within a
maintenance (3.x) release series. Apart from the 3.5.[12] slip, CPython has
proven very reliable in these guarantees.

We can (or at least could) easily take care in Cython to enable version
specific features and optimisations only from CPython alpha/beta releases
on, and not when they should become available in later point releases, so
that users can compile their code in, say, CPython 3.7.5 and it will work
correctly in 3.7.1.

We never cared about Py_BUILD_CORE (because that's obviously internal), and
it's also not very likely that we will have a Py_LIMITED_API backend
anywhere in the near future (although we would consider PRs for it that
implement the support as an optional C compile time feature).

What I would ask, though, and I think that's also Jeroen's request, is to
be careful what you lock up behind Py_BUILD_CORE. Any new functionality
should be available to extension modules by default, unless there is a good
reason why it should remain internal. Usually, there is a reason why this
functionality was added, and I doubt that there are many cases where these
reasons are entirely internal to CPython.

One thing that is not mentioned above are underscore private C-API
functions. I imagine that they are a bit annoying for CPython itself
because promoting them to public means renaming them, which is already a
breaking change. But they are a clear marker for potential future breakage,
which is good. Still, my experience so far suggests that they also fall
under the "keep stable in maintenance branch" rule, which is even better.

So, yeah, I'm happy with the status quo, and a bit worried about all the
moving around of declarations and that scent of a sword of Damocles hanging
over their potential confinement. IMHO, things should just be public and
potentially marked as "unstable" to advertise a risk of breakage in a
future CPython X.Y feature releases. Then it's up to the users to decide
how much work they want to invest into keeping up with C-API changes vs.
potentially sub-optimal but stable C-API usage.

Stefan