On Mon, Jun 13, 2022 at 5:36 PM Marc-Andre Lemburg <mal@egenix.com> wrote:
In the past we always said: "_Py* is an internal API. Use at your own risk.", which I guess is somewhere between the warning and the strict "don't use" policy you are describing.
In the last year, I tried to go further: make sure that it's no longer technically possible to use internal functions (the ones that I modified).
I made two changes:
Move API from the public API to the internal API: if you really want to access to internal API, you know have to (1) define Py_BUILD_CORE (2) use a different header file (with "internal/" in its path).
Replace PyAPI_FUNC() with "extern", so it's technically no longer possible to use modified functions outside Python itself. This change cannot be done if an internal API is used by a stdlib module built as a shared library.
The problem with the "don't use" policy is that in some cases, there are no public APIs available to do certain things and so the extension writers have to resort to the private ones to implement their logic.
My goal is to reduce the size of the C API and clarify what's public and what's internal.
That's how I noticed that _PyFloat_Pack8() was used and I added a tested and documented public API for that instead in Python 3.11 and I added it to pythoncapi-compat for older Python versions (the implementation just uses the private _PyFloat functions).
For me, anything with _Py and _PY prefix falls into the "internal" bag.
Reducing the size of the C API is important to still be able to change Python internals. Right now, it's stressful for a core dev to modify Python, since it's still unclear what's public or not, and estimate if a change is going to break a project or not (hint: any change will always break one project somewhere ;-)).
E.g. to implement a free list for Python objects, you have to use _Py_NewReference() in order to create an object based on a memory area taken from the free list. If you want to create a bytes objects using overallocation, it's common to use _PyBytes_Resize() to resize the buffer to the final size.
I want to remove _Py_NewReference() but "sadly" it's used in 3rd party extensions, so for now, I kept it :-) https://github.com/python/cpython/issues/85161
I would be nice to have a "supported" (maybe private/internal?) API for that.
Victor
Night gathers, and now my watch begins. It shall not end until my death.