[Python-Dev] Making PyInterpreterState an opaque type

[Jeroen Demeyer]

On 2019-02-18 21:17, Eric Snow wrote:
> Historically our approach to keeping API private was to use underscore
> prefixes and to leave them out of the documentation (along with
> guarding with "#ifndef Py_LIMITED_API").  However, this has lead to
> occasional confusion and breakage, and even to leaking things into the
> stable ABI that shouldn't have been.  Lately we've been working on
> making the distinction between internal and public API (and stable
> ABI) more clear and less prone to accidental exposure.  Victor has
> done a lot of work in this area.
>
> So I'd like to understand your objection.

First of all, if everybody can actually #define Py_BUILD_CORE and get 
access to the complete API, I don't mind so much. But then it's 
important that this actually keeps working (i.e. that those headers will 
always be installed).

Still, do we really need so many levels of API:
(1) stable API (with #define Py_LIMITED_API)
(2) public documented API
(3) private undocumented API (the default exposed API)
(4) internal API (with #define Py_BUILD_CORE)

I would argue to fold (4) into (3). Applications using (3) already know 
that they are living dangerously by using private API.

I'm afraid of hiding actually useful private macros under Py_BUILD_CORE. 
For example, Modules/_functoolsmodule.c and Modules/_json.c use API 
functions from (4). But if an API function is useful for implementing 
functools or json, then it's probably also useful for external extension 
modules: what if I want to implement something similar to functools or 
json, why shouldn't I be allowed to use those same API functions?

For a very concrete example, was it really necessary to put 
_PyTuple_ITEMS in (4)? That's used in _functoolsmodule.c. Especially 
given that the very similar PySequence_Fast_ITEMS is in (2), that seems 
like a strange and arbitrary limiting choice.


Jeroen.