[capi-sig] Re: Module state access from slot methods: _PyType_GetModuleByDef

On 9 Jun 2020, at 16:56, Petr Viktorin encukou@gmail.com wrote:

Hello, Now that PEP 573 (Module State Access from C Extension Methods) is in, it's time to address the most important part missing from it: module state access from slot methods (like tp_init or nb_add).

This has long history; see e.g. a thread I started in 2015: https://mail.python.org/pipermail/import-sig/2015-July/thread.html#1037

Most recently, a possible solution, an MRO walker, was one of the last things removed from PEP 573 before acceptance, because the proposed solution was broken: see https://mail.python.org/archives/list/python-dev@python.org/message/PFBAQWQA...

Here's the removed text:

...

Slot methods

To allow access to per-module state_ from slot methods, an MRO walker will be implemented:: PyTypeObject *PyType_DefiningTypeFromSlotFunc(PyTypeObject *type, int slot, void *func) The walker will go through bases of heap-allocated type and search for class that defines func at its slot. The func does not need to be inherited by type (i.e. it may have been overridden in a subclass). The only requirement for the walker to find the defining class is that the defining class must be heap-allocated. On failure, exception is set and NULL is returned.

and the last iteration of the implementation is here: https://github.com/Dormouse759/cpython/blob/41fe314c5124fa9434d56169868dfc03...

A MRO walker is not efficient.

Not only that, the MRO walker is not necessarily correct, in particular when multiple types on the MRO uses the same C function to implement a slot.

The other viable options I'm aware of are:

• putting a pointer in the class object (which would need reliable class-level storage, see https://mail.python.org/archives/list/capi-sig@python.org/thread/BWVQLET4L3W... )
• solutions that aren't fully general, like context variables or putting a pointer in each instance (thorny because constructors/initializers are themselves slots)
• "__typeslots__" mentioned in https://mail.python.org/pipermail/import-sig/2015-July/001035.html -- which are, IMO, quite invasive

Wouldn’t it be possible to pass the class as an additional argument to the slot function? That requires changing the function signature for slots to use the information, but AFAIK the extra argument wouldn’t be an ABI break in practice if the extra argument is added to the end of the signature (if it would technically be undefined behaviour).

Ronald