On Tue, 14 Dec 2021 10:38:25 -0700 Eric Snow <ericsnowcurrently@gmail.com> wrote:
So we (the core devs) would effectively be requiring those extensions to support subinterpreters, regardless of letting them opt out. This situation has been weighing heavily on my mind since Nathaniel brought this up. Here are some ideas I've had or heard of about what we could do to help:
* add a page to the C-API documentation about how to support subinterpreters * identify the extensions most likely to be impacted and offer to help * add more helpers to the C-API to make adding subinterpreter support less painful * fall back to loading the extension in its own namespace (e.g. use ldm_open()) * fall back to copying the extension's file and loading from the copied file * ...
As a data point, in PyArrow, we have a bunch of C++ code that interacts with Python but doesn't belong in a particular Python module. That C++ code can of course have global state, including perhaps Python objects. What might be nice would be a C API to allow creating interpreter-local opaque structs, for example: void* Py_GetInterpreterLocal(const char* unique_name); void* Py_SetInterpreterLocal(const char* unique_name, void* ptr, void(*)() destructor); Then in extension code you'd be able to write, e.g.: typedef struct { PyObject* cached_decimal; std::vector<std::string> some_other_data; } MyLocalState ; static void destroy_state(MyLocalState* state) { Py_XDECREF(state->cached_decimal); delete state; } static MyLocalState* get_state() { MyLocalState* state = NULL; state = Py_GetInterpreterLocal("pyarrow._lib.internal"); if (state == NULL) { state = new MyLocalState; Py_SetInterpreterLocal("pyarrow._lib.internal", state, destroy_state); } return state; } PyObject* get_decimal_module() { MyLocalState* state = get_state(); if (state->cached_decimal == NULL) { state->cached_decimal = PyImport_ImportModule("decimal"); } return state->cached_decimal; }
I'd appreciate your thoughts on what we can do to help. Thanks!
-eric