[capi-sig] Module state access from slot methods: _PyType_GetModuleByDef

Hello, Now that PEP 573 (Module State Access from C Extension Methods) is in, it's time to address the most important part missing from it: module state access from slot methods (like tp_init or nb_add).

This has long history; see e.g. a thread I started in 2015: https://mail.python.org/pipermail/import-sig/2015-July/thread.html#1037

Most recently, a possible solution, an MRO walker, was one of the last things removed from PEP 573 before acceptance, because the proposed solution was broken: see https://mail.python.org/archives/list/python-dev@python.org/message/PFBAQWQA...

Here's the removed text:

Slot methods

To allow access to per-module state_ from slot methods, an MRO walker will be implemented::

PyTypeObject *PyType_DefiningTypeFromSlotFunc(PyTypeObject *type,
                                              int slot, void *func)

The walker will go through bases of heap-allocated type and search for class that defines func at its slot.

The func does not need to be inherited by type (i.e. it may have been overridden in a subclass). The only requirement for the walker to find the defining class is that the defining class must be heap-allocated.

On failure, exception is set and NULL is returned.

and the last iteration of the implementation is here: https://github.com/Dormouse759/cpython/blob/41fe314c5124fa9434d56169868dfc03...

A MRO walker is not efficient. The other viable options I'm aware of are: themselves slots)

• putting a pointer in the class object (which would need reliable class-level storage, see https://mail.python.org/archives/list/capi-sig@python.org/thread/BWVQLET4L3W... )
• solutions that aren't fully general, like context variables or putting a pointer in each instance (thorny because constructors/initializers are

• "__typeslots__" mentioned in https://mail.python.org/pipermail/import-sig/2015-July/001035.html -- which are, IMO, quite invasive

Given that, I believe that some kind of MRO walker is still the best immediate solution, mainly because it's easiest to implement. If done well, it can only use current public API (limiting the maintenance burden you'd have with something wired into the internals, and allowing it to be copied to 3rd party code to support older Python versions), and should be easy to replace with a more performant solution once that comes out. And if it's added, more modules could switch to the stable ABI and eschew process-global state, so we get more exposure and experience with the remaining problems.

Now, the main problem with "PyType_DefiningTypeFromSlotFunc" is that it looked slot functions, which can be set from Python code, and doing so can easily introduce C-level failures. To solve this, my next thought is instead looking for something only available in C: the PyModuleDef. Here's an implementation of such an MRO walker, which I call "_PyType_GetModuleByDef": https://github.com/encukou/cpython/blob/98dd889575cf7d1688495983ba791e14894a...

My thinking is to start using this in 3.10 as internal API to get a feel for its strengths and weaknesses. It should be easy to rewrite the function to only use public API, and added to another project (like Cython). It should also be easy to make it public once we're sure it's the way forward.

What are your thoughts?