On Mon, Feb 1, 2021 at 11:06 AM Petr Viktorin <encukou@gmail.com> wrote:
Besides the non-opaque PyObject*, the worst thing that prevents "evolution" now is IMO that built-in static types like PyList_Type are exposed in the API and stable ABI as global symbols. This prevents making the GIL per-interpreter.
I created https://bugs.python.org/issue40601 to discuss the problem of static types "exposed" (indirectly) in the limited C API.
I wrote an experimental PR to see how we could do that: https://github.com/python/cpython/pull/24146
- Add Py_GetXXXType()
- Replace &XXX_Type with Py_GetXXXType()
These heap types must be created as early as possible, and destroyed as late as possible.
Py_GetXXXType() functions access the current interpreter. Currently, _PyInterpreterState_GET() returns NULL when the GIL is released. It means that accessing types is no longer possible when the GIL is released. I consider that it was never officially supported and so it's not an incompatible change, but some extensions do that and will crash with these changes (we can fix them and document the issue).
Here's how stuff like that can be changed:
- Formalize, test and document the stable ABI
- Introduce new API (e.g. PyList_GetListType())
- Deprecate the old API
I suggest to use something like https://github.com/pythoncapi/pythoncapi_compat to provide new "get" functions to old Python functions (they would access static types) and prepare extension modules for the future incompatible changes, without losing support with old Python versions.
For example: static inline PyTypeObject* Py_GetListType(void) { return &PyList_Type; }
Flag day migration as we did with Python 2 => Python 3 is usually a disaster and annoy everybody.
We should automate the migration as much as we can.
Victor
Night gathers, and now my watch begins. It shall not end until my death.