[Python-Dev] Making PyInterpreterState an opaque type

[Steve Dower]

On 18Feb.2019 1324, Jeroen Demeyer wrote:
> Still, do we really need so many levels of API:
> (1) stable API (with #define Py_LIMITED_API)
> (2) public documented API
> (3) private undocumented API (the default exposed API)
> (4) internal API (with #define Py_BUILD_CORE)
> 
> I would argue to fold (4) into (3). Applications using (3) already know
> that they are living dangerously by using private API.

I agree completely. It's unfortunate we ended up in a state where the
stable API was opt-in, but that's where we are now and we have to
transition carefully.

The ideal would be:
* default to cross-version supported APIs (i.e. stable for all 3.*)
* opt-in to current-version stable APIs (i.e. stable for all 3.7.*)
* extra opt-in to unstable APIs (i.e. you are guaranteed to break one
day without warning)

> I'm afraid of hiding actually useful private macros under Py_BUILD_CORE.
> For example, Modules/_functoolsmodule.c and Modules/_json.c use API
> functions from (4). But if an API function is useful for implementing
> functools or json, then it's probably also useful for external extension
> modules: what if I want to implement something similar to functools or
> json, why shouldn't I be allowed to use those same API functions?
> 
> For a very concrete example, was it really necessary to put
> _PyTuple_ITEMS in (4)? That's used in _functoolsmodule.c. Especially
> given that the very similar PySequence_Fast_ITEMS is in (2), that seems
> like a strange and arbitrary limiting choice.

The reason to do this is that we can "guarantee" that we've fixed all
users when we change the internal representation. Otherwise, the
internal memory layout becomes part of the public ABI, which is what we
want to fix. (PyTuple_GET_ITEM is just as problematic, FWIW.)

If you always rebuild your extension for every micro version (3.x.y) of
CPython, then sure, go ahead and use this. But you're by far into the
minority of users/developers, and so we really don't want to optimise
for this case when it's going to break the 90%+ of people who don't
recompile everything all the time.

Cheers,
Steve