[Python-Dev] Rename Include/internals/ to Include/pycore/

[Victor Stinner]

Hi,

Python C API has no strict separation between the 3 levels of API:

* core: Py_BUILD_CORE define
* stable: Py_LIMITED_API define (it has a value)
* regular: no define

IMHO the current design of header files is done backward: by default,
everything is public. To exclude an API from core or stable, "#ifdef
Py_BUILD_CORE" and "#ifndef Py_LIMITED_API" are used. This design
caused issues in the past: functions, variables or something else
exposed whereas they were supposed to be "private".

I propose a practical solution for that: Include/*.h files would only
be be public API. The "core" API would live in a new subdirectory:
Include/pycore/*.h. Moreover, files of this subdirectory would have
the prefix "pycore_". For example, Include/objimpl.h would have a
twin: Include/pycore/pycore_objimpl.h which extend the public API with
"core" APIs.

I also propose to automatically load the twin: Include/objimpl.h would
load Include/pycore/pycore_objimpl.h if Py_BUILD_CORE is defined:

#ifdef Py_BUILD_CORE
#  include "pycore/pycore_objimpl.h"
#endif

Only in some rare cases, you would have to explicitly use: #include
"pycore/pycore_pygetopt.h". This header is fully private, there is no
public header in Include/pygetopt.h. Or maybe we should modify
Include/Python.h to also include "pycore/pycore_pygetopt.h" if
Py_BUILD_CORE is defined? Well, that's just a detail.

First milestone:

* Create Include/pycore/
* Move Py_BUILD_CORE specific code into Include/pycore/pycore_*.h
* Automatically include pycore files from Include/*.h files (#ifdef
Py_BUILD_CORE)

Second milestone:

* Find a solution for Py_LIMITED_API :-)

Backward compatibility? The overall change is fully backward
compatible. The default API doesn't change. C code (using header fles)
doesn't have to be changed.

Only a specific kinds of applications like debugger may have to be
modified if they really have to access the "core" API, the "most
private" API. Honestly, today it's unclear to me if this API can
technically be used outside CPython.

Victor