Adding PyType_GetModuleByDef to the limited API
In bpo-42100, I added the private function _PyType_GetModuleByDef, which allows module state access from slot methods (like tp_init or nb_add), the main thing missing from PEP 573 (Module State Access from C Extension Methods). The method is now used in 10 stdlib modules, without known issues.
Since it was has optimized (thanks, Victor!), I haven't seen complaints about performance issues. Sadly, it has a loop so it's much slower than using C statics, but it's OK for many use cases.
I haven't found a better way to get per-module state for isolated modules (and the stable ABI).
So, I'd like to drop the leading underscore, and add PyType_GetModuleByDef directly to the limited API.
The function itself can be implemented using only limited API, though it's a bit tricky to do so correctly (and our implementation uses private speedups), so it's better if extension authors can use it as a pre-made building block.
The fact that it can be implemented with only limited API means that adding it should not add big burden for long-term maintenance in CPython, nor for alternate implementations of the C-API.
Is anyone against? Does anyone feel this should be discussed more widely than on capi-sig?
On 5/1/22 6:32 pm, Petr Viktorin wrote:
So, I'd like to drop the leading underscore, and add PyType_GetModuleByDef directly to the limited API.
This seems like a very helpful function, and if I recall correctly pretty much required for using the limited API with multi-stage module loading.
Sorry for the newbie question, that probably has been answered many times before but I don't know how to google for the answer.
What does this mean for ABI3 wheels?
An in too much detail: If this is part of 3.11, and I build a wheel using the newer limited API interface, will I still be able to call it ABI3?
If so does that mean that somehow I need to tell people "yes this is ABI3 but only for Python3.11+?
Is there tooling to help me do that?
Matti
On 05. 01. 22 17:45, Matti Picus wrote:
On 5/1/22 6:32 pm, Petr Viktorin wrote:
So, I'd like to drop the leading underscore, and add PyType_GetModuleByDef directly to the limited API.
This seems like a very helpful function, and if I recall correctly pretty much required for using the limited API with multi-stage module loading.
Sorry for the newbie question, that probably has been answered many times before but I don't know how to google for the answer.
What does this mean for ABI3 wheels?
An in too much detail: If this is part of 3.11, and I build a wheel using the newer limited API interface, will I still be able to call it ABI3?
If so does that mean that somehow I need to tell people "yes this is ABI3 but only for Python3.11+?
Yes. Compile with PY_LIMITED_API defined as 0x030B0000 (hex version for 3.11), which makes the extension compatible with 3.11+.
Is there tooling to help me do that?
Well, that's certainly an area that could be improved :) You can add Requires-Python to your package (wheel/sdist) to prevent installation on older Pythons, but if the extension file itself ends up installed on 3.10 or below, it'll likely fail in an unfriendly way (missing symbols).
On Wed, Jan 5, 2022 at 9:32 AM Petr Viktorin <encukou@gmail.com> wrote:
Since it was has optimized (thanks, Victor!), I haven't seen complaints about performance issues. Sadly, it has a loop so it's much slower than using C statics, but it's OK for many use cases.
Just to be clear, it loops over the type's MRO. So typically it won't involve more than one or two iterations.
So, I'd like to drop the leading underscore, and add PyType_GetModuleByDef directly to the limited API.
[snip]
Is anyone against? Does anyone feel this should be discussed more widely than on capi-sig?
Seems good to me. It seems useful right now and this API doesn't prevent us from finding a more efficient solution.
-eric
On Wed, Jan 5, 2022 at 8:32 AM Petr Viktorin <encukou@gmail.com> wrote:
In bpo-42100, I added the private function _PyType_GetModuleByDef, which allows module state access from slot methods (like tp_init or nb_add), the main thing missing from PEP 573 (Module State Access from C Extension Methods). The method is now used in 10 stdlib modules, without known issues.
Since it was has optimized (thanks, Victor!), I haven't seen complaints about performance issues. Sadly, it has a loop so it's much slower than using C statics, but it's OK for many use cases.
I haven't found a better way to get per-module state for isolated modules (and the stable ABI).
So, I'd like to drop the leading underscore, and add PyType_GetModuleByDef directly to the limited API.
The function itself can be implemented using only limited API, though it's a bit tricky to do so correctly (and our implementation uses private speedups), so it's better if extension authors can use it as a pre-made building block.
The fact that it can be implemented with only limited API means that adding it should not add big burden for long-term maintenance in CPython, nor for alternate implementations of the C-API.
Is anyone against? Does anyone feel this should be discussed more widely than on capi-sig?
I don't have an opinion, but the point of having SIGs to to minimize having to get more people involved in technical discussion that only apply to a subset of folks. If people feel this SIG is not known widely enough then we can post on python-dev about its existence, but otherwise this list should be good enough for a discussion.
Sounds good to me.
Erlend
On 5 Jan 2022, at 17:32, Petr Viktorin <encukou@gmail.com> wrote:
In bpo-42100, I added the private function _PyType_GetModuleByDef, which allows module state access from slot methods (like tp_init or nb_add), the main thing missing from PEP 573 (Module State Access from C Extension Methods). The method is now used in 10 stdlib modules, without known issues.
Since it was has optimized (thanks, Victor!), I haven't seen complaints about performance issues. Sadly, it has a loop so it's much slower than using C statics, but it's OK for many use cases.
I haven't found a better way to get per-module state for isolated modules (and the stable ABI).
So, I'd like to drop the leading underscore, and add PyType_GetModuleByDef directly to the limited API.
The function itself can be implemented using only limited API, though it's a bit tricky to do so correctly (and our implementation uses private speedups), so it's better if extension authors can use it as a pre-made building block.
The fact that it can be implemented with only limited API means that adding it should not add big burden for long-term maintenance in CPython, nor for alternate implementations of the C-API.
Is anyone against? Does anyone feel this should be discussed more widely than on capi-sig?
capi-sig mailing list -- capi-sig@python.org To unsubscribe send an email to capi-sig-leave@python.org https://mail.python.org/mailman3/lists/capi-sig.python.org/ Member address: erlend.aasland@innova.no
participants (5)
-
Brett Cannon -
Eric Snow -
Erlend Aasland -
Matti Picus -
Petr Viktorin