Hi Brett, IMO the PEP 630 is a good summary and a practical guide explaining how to port existing C extensions to newer C API which are compatible with subinterpreters, unloading a C extension and be able to load a C extension more than once (in the same interpreter): https://www.python.org/dev/peps/pep-0630/ I dislike justifying these changes only with subinterpreters. IMO it's better to justify that these changes are needed to be able to load and unload Python cleanly when an application embeds Python. A common use case is to support plugins in different programming languages, including Python. Some IRC clients and text editors have this use case. It's unpleasant when Python leaks lot of memory when Python is "unloaded", especially if the application is designed to load Python once per plugin. It's even more unpleasant when... they are bugs :-( Examples of changes needed by the PEP 630: * Add a state to the module: use a PyModuleDef.m_size value greater than 0 (usually a structure is used with ".m_size = sizeof(_abcmodule_state)") * Convert static types to heap types and store these types in the module state * Move global variable and function "static" variables into the module state * IMO the most complicated part is to retrieve the module state from functions which don't directly get the module as an argument, but only an instance of a type defined in the module. In some type "slot" function, you will need to use ... a *private* function... which was only added recently, in *Python 3.10*: _PyType_GetModuleByDef(). IMO the big problem of _PyType_GetModuleByDef() is that developers want supporting Python versions older than Python 3.10. For example, right now, numpy supports Python 3.7 and newer. Moreover, the fact that the function remains private in Python 3.11 is also an issue. Another challenge is how to check if a C extension is "fully" compatible with subinterpreters? In Python 3.11, Python still implements around 100 types as "static types" which are not compatible with subinterpreters, like &PyLong_Type and &PyUnicode_Type. I opened https://bugs.python.org/issue40601 about these static types, but it seems like changing it may break the C API *and* the stable ABI (maybe a clever hack will avoid that). One idea would be to add a macro excluding functions known to be unsafe with subinterpreters from the C API. For example, exclude "PyLong_Type" if the Py_SUBINTERPRETER_API macro is defined. These static types are accessed directly, but also indirectly. For example, PyLong_CheckExact() is implemented as a macro which access directly this type. Should we remove this function from the C API? Or implement it as a regular "opaque" function if Py_SUBINTERPRETER_API is defined? I would prefer an error at the build, rather than a crash at runtime :-( Victor On Wed, Dec 15, 2021 at 12:06 AM Brett Cannon <brett@python.org> wrote:
On Tue, Dec 14, 2021 at 9:41 AM Eric Snow <ericsnowcurrently@gmail.com> wrote:
One of the open questions relative to subinterpreters is: how to reduce the amount of work required for extension modules to support them? Thanks to Petr Viktorin for a lot of work he's done in this area (e.g. PEP 489)! Extensions also have the option to opt out of subinterpreter support.
However, that's only one part of the story. A while back Nathaniel expressed concerns with how making subinterpreters more accessible will have a negative side effect affecting projects that publish large extensions, e.g. numpy. Not all extensions support subinterpreters due to global state (incl. in library dependencies). The amount of work to get there may be large. As subinterpreters increase in usage in the community, so will demand increase for subinterpreter support in those extensions. Consequently, such projects be pressured to do the extra work (which is made even more stressful by the short-handed nature of most open source projects) .
So we (the core devs) would effectively be requiring those extensions to support subinterpreters, regardless of letting them opt out. This situation has been weighing heavily on my mind since Nathaniel brought this up. Here are some ideas I've had or heard of about what we could do to help:
* add a page to the C-API documentation about how to support subinterpreters * identify the extensions most likely to be impacted and offer to help * add more helpers to the C-API to make adding subinterpreter support less painful * fall back to loading the extension in its own namespace (e.g. use ldm_open()) * fall back to copying the extension's file and loading from the copied file * ...
I'd appreciate your thoughts on what we can do to help. Thanks!
What are the requirements put upon an extension in order to support subinterpreters? you hint at global state at the C level, but nothing else is mentioned. Is that it? _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/S4QV6SYR... Code of Conduct: http://python.org/psf/codeofconduct/
-- Night gathers, and now my watch begins. It shall not end until my death.