Module state access from slot methods: _PyType_GetModuleByDef
Hello, Now that PEP 573 (Module State Access from C Extension Methods) is in, it's time to address the most important part missing from it: module state access from slot methods (like tp_init or nb_add).
This has long history; see e.g. a thread I started in 2015: https://mail.python.org/pipermail/import-sig/2015-July/thread.html#1037
Most recently, a possible solution, an MRO walker, was one of the last things removed from PEP 573 before acceptance, because the proposed solution was broken: see https://mail.python.org/archives/list/python-dev@python.org/message/PFBAQWQA...
Here's the removed text:
Slot methods
To allow access to
per-module state_ from slot methods, an MRO walker will be implemented::PyTypeObject *PyType_DefiningTypeFromSlotFunc(PyTypeObject *type, int slot, void *func)The walker will go through bases of heap-allocated
typeand search for class that definesfuncat itsslot.The
funcdoes not need to be inherited bytype(i.e. it may have been overridden in a subclass). The only requirement for the walker to find the defining class is that the defining class must be heap-allocated.On failure, exception is set and NULL is returned.
and the last iteration of the implementation is here: https://github.com/Dormouse759/cpython/blob/41fe314c5124fa9434d56169868dfc03...
A MRO walker is not efficient. The other viable options I'm aware of are: themselves slots)
- putting a pointer in the class object (which would need reliable class-level storage, see https://mail.python.org/archives/list/capi-sig@python.org/thread/BWVQLET4L3W... )
- solutions that aren't fully general, like context variables or putting a pointer in each instance (thorny because constructors/initializers are
- "__typeslots__" mentioned in https://mail.python.org/pipermail/import-sig/2015-July/001035.html -- which are, IMO, quite invasive
Given that, I believe that some kind of MRO walker is still the best immediate solution, mainly because it's easiest to implement. If done well, it can only use current public API (limiting the maintenance burden you'd have with something wired into the internals, and allowing it to be copied to 3rd party code to support older Python versions), and should be easy to replace with a more performant solution once that comes out. And if it's added, more modules could switch to the stable ABI and eschew process-global state, so we get more exposure and experience with the remaining problems.
Now, the main problem with "PyType_DefiningTypeFromSlotFunc" is that it looked slot functions, which can be set from Python code, and doing so can easily introduce C-level failures. To solve this, my next thought is instead looking for something only available in C: the PyModuleDef. Here's an implementation of such an MRO walker, which I call "_PyType_GetModuleByDef": https://github.com/encukou/cpython/blob/98dd889575cf7d1688495983ba791e14894a...
My thinking is to start using this in 3.10 as internal API to get a feel for its strengths and weaknesses. It should be easy to rewrite the function to only use public API, and added to another project (like Cython). It should also be easy to make it public once we're sure it's the way forward.
What are your thoughts?
On 9 Jun 2020, at 16:56, Petr Viktorin <encukou@gmail.com> wrote:
Hello, Now that PEP 573 (Module State Access from C Extension Methods) is in, it's time to address the most important part missing from it: module state access from slot methods (like tp_init or nb_add).
This has long history; see e.g. a thread I started in 2015: https://mail.python.org/pipermail/import-sig/2015-July/thread.html#1037
Most recently, a possible solution, an MRO walker, was one of the last things removed from PEP 573 before acceptance, because the proposed solution was broken: see https://mail.python.org/archives/list/python-dev@python.org/message/PFBAQWQA...
Here's the removed text:
Slot methods
To allow access to
per-module state_ from slot methods, an MRO walker will be implemented:: PyTypeObject *PyType_DefiningTypeFromSlotFunc(PyTypeObject *type, int slot, void *func) The walker will go through bases of heap-allocatedtypeand search for class that definesfuncat itsslot. Thefuncdoes not need to be inherited bytype(i.e. it may have been overridden in a subclass). The only requirement for the walker to find the defining class is that the defining class must be heap-allocated. On failure, exception is set and NULL is returned.and the last iteration of the implementation is here: https://github.com/Dormouse759/cpython/blob/41fe314c5124fa9434d56169868dfc03...
A MRO walker is not efficient.
Not only that, the MRO walker is not necessarily correct, in particular when multiple types on the MRO uses the same C function to implement a slot.
The other viable options I'm aware of are:
- putting a pointer in the class object (which would need reliable class-level storage, see https://mail.python.org/archives/list/capi-sig@python.org/thread/BWVQLET4L3W... )
- solutions that aren't fully general, like context variables or putting a pointer in each instance (thorny because constructors/initializers are themselves slots)
- "__typeslots__" mentioned in https://mail.python.org/pipermail/import-sig/2015-July/001035.html -- which are, IMO, quite invasive
Wouldn’t it be possible to pass the class as an additional argument to the slot function? That requires changing the function signature for slots to use the information, but AFAIK the extra argument wouldn’t be an ABI break in practice if the extra argument is added to the end of the signature (if it would technically be undefined behaviour).
Ronald
On 2020-06-09 18:23, Ronald Oussoren wrote:
On 9 Jun 2020, at 16:56, Petr Viktorin <encukou@gmail.com> wrote:
Hello, Now that PEP 573 (Module State Access from C Extension Methods) is in, it's time to address the most important part missing from it: module state access from slot methods (like tp_init or nb_add).
This has long history; see e.g. a thread I started in 2015: https://mail.python.org/pipermail/import-sig/2015-July/thread.html#1037
Most recently, a possible solution, an MRO walker, was one of the last things removed from PEP 573 before acceptance, because the proposed solution was broken: see https://mail.python.org/archives/list/python-dev@python.org/message/PFBAQWQA...
Here's the removed text:
Slot methods
To allow access to
per-module state_ from slot methods, an MRO walker will be implemented:: PyTypeObject *PyType_DefiningTypeFromSlotFunc(PyTypeObject *type, int slot, void *func) The walker will go through bases of heap-allocatedtypeand search for class that definesfuncat itsslot. Thefuncdoes not need to be inherited bytype(i.e. it may have been overridden in a subclass). The only requirement for the walker to find the defining class is that the defining class must be heap-allocated. On failure, exception is set and NULL is returned.and the last iteration of the implementation is here: https://github.com/Dormouse759/cpython/blob/41fe314c5124fa9434d56169868dfc03...
A MRO walker is not efficient.
Not only that, the MRO walker is not necessarily correct, in particular when multiple types on the MRO uses the same C function to implement a slot.
I'm not *that* worried about this. The module's author should be in control of how the C functions are used.
The other viable options I'm aware of are:
- putting a pointer in the class object (which would need reliable class-level storage, see https://mail.python.org/archives/list/capi-sig@python.org/thread/BWVQLET4L3W... )
- solutions that aren't fully general, like context variables or putting a pointer in each instance (thorny because constructors/initializers are themselves slots)
- "__typeslots__" mentioned in https://mail.python.org/pipermail/import-sig/2015-July/001035.html -- which are, IMO, quite invasive
Wouldn’t it be possible to pass the class as an additional argument to the slot function? That requires changing the function signature for slots to use the information, but AFAIK the extra argument wouldn’t be an ABI break in practice if the extra argument is added to the end of the signature (if it would technically be undefined behaviour).
Unfortunately, no. We're not in control of all code that calls the slot
methods. Anyone can call PyType_GetSlot(Py_TYPE(o), Py_tp_iter)(o).
On 9 Jun 2020, at 18:32, Petr Viktorin <encukou@gmail.com> wrote:
On 2020-06-09 18:23, Ronald Oussoren wrote:
On 9 Jun 2020, at 16:56, Petr Viktorin <encukou@gmail.com> wrote:
Hello, Now that PEP 573 (Module State Access from C Extension Methods) is in, it's time to address the most important part missing from it: module state access from slot methods (like tp_init or nb_add).
This has long history; see e.g. a thread I started in 2015: https://mail.python.org/pipermail/import-sig/2015-July/thread.html#1037
Most recently, a possible solution, an MRO walker, was one of the last things removed from PEP 573 before acceptance, because the proposed solution was broken: see https://mail.python.org/archives/list/python-dev@python.org/message/PFBAQWQA...
Here's the removed text:
Slot methods
To allow access to
per-module state_ from slot methods, an MRO walker will be implemented:: PyTypeObject *PyType_DefiningTypeFromSlotFunc(PyTypeObject *type, int slot, void *func) The walker will go through bases of heap-allocatedtypeand search for class that definesfuncat itsslot. Thefuncdoes not need to be inherited bytype(i.e. it may have been overridden in a subclass). The only requirement for the walker to find the defining class is that the defining class must be heap-allocated. On failure, exception is set and NULL is returned.and the last iteration of the implementation is here: https://github.com/Dormouse759/cpython/blob/41fe314c5124fa9434d56169868dfc03...
A MRO walker is not efficient. Not only that, the MRO walker is not necessarily correct, in particular when multiple types on the MRO uses the same C function to implement a slot.
I'm not *that* worried about this. The module's author should be in control of how the C functions are used.
It is highly unlikely that this will be a problem in real code, but we should try to avoid adding new gotchas to the C API.
The other viable options I'm aware of are:
- putting a pointer in the class object (which would need reliable class-level storage, see https://mail.python.org/archives/list/capi-sig@python.org/thread/BWVQLET4L3W... <https://mail.python.org/archives/list/capi-sig@python.org/thread/BWVQLET4L3W643QYK45BYFJBQT5NKKIK/> )
- solutions that aren't fully general, like context variables or putting a pointer in each instance (thorny because constructors/initializers are themselves slots)
- "__typeslots__" mentioned in https://mail.python.org/pipermail/import-sig/2015-July/001035.html <https://mail.python.org/pipermail/import-sig/2015-July/001035.html> -- which are, IMO, quite invasive Wouldn’t it be possible to pass the class as an additional argument to the slot function? That requires changing the function signature for slots to use the information, but AFAIK the extra argument wouldn’t be an ABI break in practice if the extra argument is added to the end of the signature (if it would technically be undefined behaviour).
Unfortunately, no. We're not in control of all code that calls the slot methods. Anyone can call
PyType_GetSlot(Py_TYPE(o), Py_tp_iter)(o).
That’s annoying. It is possible to work around that, but that gets ugly fast and would likely have to rely on libffi or a similar library.
Ronald
—
Twitter / micro.blog: @ronaldoussoren Blog: https://blog.ronaldoussoren.net/
[belatedly catching up on capi-sig]
On Wed, 10 Jun 2020 at 00:56, Petr Viktorin <encukou@gmail.com> wrote:
Given that, I believe that some kind of MRO walker is still the best immediate solution, mainly because it's easiest to implement.
The other reason I still like using the MRO walker as the foundation is that once there's a way for tp_new and/or tp_init to look up the relevant info, or for individual slots to look it up on first use, then making the access during runtime fast becomes a problem that's well suited to being accelerated via context local variables.
Now, the main problem with "PyType_DefiningTypeFromSlotFunc" is that it looked slot functions, which can be set from Python code, and doing so can easily introduce C-level failures. To solve this, my next thought is instead looking for something only available in C: the PyModuleDef. Here's an implementation of such an MRO walker, which I call "_PyType_GetModuleByDef": https://github.com/encukou/cpython/blob/98dd889575cf7d1688495983ba791e14894a...
This seems like a plausible candidate to me, but the "break on non-heap-type" bit seems odd to me at first glance, as I'd have expected a "continue" there rather than a "break".
If there's an object model implementation related reason why we can't ever have a heap type appear in the MRO after a non-heap-type, then we could have a break-with-explanatory-comment instead, but even then, a "continue" seems to be just as good as having "break" (we'll just potentially loop a few more times before throwing the error)
My thinking is to start using this in 3.10 as internal API to get a feel for its strengths and weaknesses. It should be easy to rewrite the function to only use public API, and added to another project (like Cython). It should also be easy to make it public once we're sure it's the way forward.
What are your thoughts?
That approach definitely worked to give me confidence in Victor's public initialisation API changes that I didn't have in the original out-of-tree changes.
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
If mro is untenable here due to runtime speed - is there any PEP or independent effort to improve it? It seems like the least disruptive option.
That’s annoying. It is possible to work around that, but that gets ugly fast and would likely have to rely on libffi or a similar library.
If we up the compiler standard to C11, there seems to be a standard way to overload on the number of arguments at compile time with _Generic.
Windows would be a b*** to deal with if we go for C11 as officially, MSVC does not support C11.
Looks like the newest version supports it: https://devblogs.microsoft.com/cppblog/c11-and-c17-standard-support-arriving...
Compile time compatibility isn't the problem for slot methods, the problem is with the dynamic dispatch table of function pointers stored on type objects - the pointers all have pre-declared signatures.
Function overloading should be helpful for evolving the C API in general, though (in particular, it may allow us to start accepting explicit interpreter and/or thread state parameters without needing new function names).
Cheers, Nick.
participants (5)
-
Mohamed Koubaa -
Nick Coghlan -
Petr Viktorin -
Ronald Oussoren -
William Pickard