[Python-Dev] Making PyInterpreterState an opaque type

[Steve Dower]

On 19Feb2019 0229, Jeroen Demeyer wrote:
> On 2019-02-19 04:04, Steve Dower wrote:
>> On 18Feb.2019 1324, Jeroen Demeyer wrote:
>>> For a very concrete example, was it really necessary to put
>>> _PyTuple_ITEMS in (4)? That's used in _functoolsmodule.c. Especially
>>> given that the very similar PySequence_Fast_ITEMS is in (2), that seems
>>> like a strange and arbitrary limiting choice.
>>
>> The reason to do this is that we can "guarantee" that we've fixed all
>> users when we change the internal representation.
> 
> I think that CPython should then at least "eat its own dog food" and 
> don't use any of the internal functions/macros when implementing the 
> stdlib. As I said before: if a function/macro is useful for implementing 
> stdlib functionality like "functools" or "json", it's probably useful 
> for external modules too.

I'm inclined to agree, but then I'm also one of the advocates for 
breaking out as much as possible of the stdlib into pip-installable 
modules, which would necessitate this :)

There are certainly parts of the stdlib that are there to _enable_ the 
use of these features without exposing them - asyncio has some good 
examples of this. But the rest probably don't. If they really do, then 
we would have to define stable ways to get the same functionality (one 
example - numpy currently relies on being able to check the refcount to 
see if it equals 1, but we could easily provide a 
"Py_HasNoOtherReferences" function that would do the same thing and also 
allow us to one day move or remove reference counts without breaking numpy).

That said, one of the criteria for "are you eligible to use the internal 
API" is "will users always have matched builds of this module" - for the 
standard library, the answer is yes, and so they can use that API.

For third-party extension modules, the answer _might_ be yes, which 
means they _might_ be able to use the API. But both of those "mights" 
are outside of the control of the core development team, so we can't 
take that responsibility for you.

The best we can do is make it easy to use the stable APIs, and make 
using the unstable APIs a deliberate choice so that those who do so are 
aware that they are now more responsible for their user's success than 
they thought they were.

Cheers,
Steve