On Thu, 16 Dec 2021 13:25:53 +0100 Petr Viktorin <encukou@gmail.com> wrote:
On 16. 12. 21 12:33, Antoine Pitrou wrote:
On Tue, 14 Dec 2021 10:38:25 -0700 Eric Snow <ericsnowcurrently@gmail.com> wrote:
So we (the core devs) would effectively be requiring those extensions to support subinterpreters, regardless of letting them opt out. This situation has been weighing heavily on my mind since Nathaniel brought this up. Here are some ideas I've had or heard of about what we could do to help:
* add a page to the C-API documentation about how to support subinterpreters * identify the extensions most likely to be impacted and offer to help * add more helpers to the C-API to make adding subinterpreter support less painful * fall back to loading the extension in its own namespace (e.g. use ldm_open()) * fall back to copying the extension's file and loading from the copied file * ...
As a data point, in PyArrow, we have a bunch of C++ code that interacts with Python but doesn't belong in a particular Python module. That C++ code can of course have global state, including perhaps Python objects.
What might be nice would be a C API to allow creating interpreter-local opaque structs, for example:
void* Py_GetInterpreterLocal(const char* unique_name); void* Py_SetInterpreterLocal(const char* unique_name, void* ptr, void(*)() destructor);
Then in extension code you'd be able to write, e.g.:
What's the reason these can't be tied to the module?
Because the module is simply not known. This is C++ utility code called from several different Cython extension modules.
(As the author of PEP 630, which argues that module state is the best default place for this kind of mutable "globals", I'm interested in the cases where it isn't so.)
That works in a world where all third-party code using the CPython C API lives in a particular extension module. While it is certainly the most common case, I doubt it is universal.
How do you ensure these Python objects are destroyed by/before Py_Finalize()? (If you do that -- I realize it's not something people typically think about.)
When finalizing a given (sub)interpreter, it would visit all registered interpreter locals and call their "destructor" callback. Regards Antoine.