Static types and subinterpreters running in parallel
Hi, One option to solve the https://bugs.python.org/issue40601 "[C API] Hide static types from the limited C API" issue without breaking the backward compatibility is to leave the C API and the stable ABI as they are for the main interpreter (static types), but force subinterpreters running in parallel to use their own heap types. It means that subinterpreters would be able to use the main interpreter GIL or own their GIL. By default, the GIL would still be shared. C extensions using "&PyLongType" (static type) would continue to work in the main interpreter. C extensions which want to opt-in for running subinterpreters in parallel would not access to "&PyLong_Type" but be forced to call PyLong_GetType() (heap type). Internally, Python should be modified to replace "&PyLongType" with PyLong_GetType(). So the code would work with static types and heap types. It also means that a subinterpreter running in parallel would only be able to import C extensions built with explicit support for this feature. Otherwise, an ImportError would be raised. Victor -- Night gathers, and now my watch begins. It shall not end until my death.
Eric has been looking into this. It's probably the only solution if we can't get immutable objects. But I would prefer the latter, if we can get the performance penalty low enough. On Thu, Dec 16, 2021 at 2:31 AM Victor Stinner <vstinner@python.org> wrote:
Hi,
One option to solve the https://bugs.python.org/issue40601 "[C API] Hide static types from the limited C API" issue without breaking the backward compatibility is to leave the C API and the stable ABI as they are for the main interpreter (static types), but force subinterpreters running in parallel to use their own heap types. It means that subinterpreters would be able to use the main interpreter GIL or own their GIL. By default, the GIL would still be shared.
C extensions using "&PyLongType" (static type) would continue to work in the main interpreter.
C extensions which want to opt-in for running subinterpreters in parallel would not access to "&PyLong_Type" but be forced to call PyLong_GetType() (heap type).
Internally, Python should be modified to replace "&PyLongType" with PyLong_GetType(). So the code would work with static types and heap types.
It also means that a subinterpreter running in parallel would only be able to import C extensions built with explicit support for this feature. Otherwise, an ImportError would be raised.
Victor -- Night gathers, and now my watch begins. It shall not end until my death. _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/QTY25AHC... Code of Conduct: http://python.org/psf/codeofconduct/
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
On Thu, Dec 16, 2021 at 10:54 AM Guido van Rossum <guido@python.org> wrote:
Eric has been looking into this. It's probably the only solution if we can't get immutable objects.
Yep. I've investigated the following approach (for the objects exposed in the public and limited C-API): * add a pointer field to PyInterpreterState (or a sub-struct) for each of the objects * for the main interpreter, set those pointers to the existing statically declared objects * for subinterpreters make a copy (memcpy()?) and fix it up * add a lookup API and encourage extensions to use it * for 3.11+ change the symbols to macros: + in the internal C-API (Py_BUILD_CORE), the macro would resolve to the corresponding PyInterpreterState field + in the public C-API (and limited API extensions built with 3.11+), the macro would resolve to a call to a (non-inline) lookup function + for limited API extensions built against earlier Python versions we'd still export the existing symbols * limited API extensions built against pre-3.11 Python would only be allowed to run in the main interpreter on 3.11+ + they probably weren't built with subinterpreters in mind anyway There are still a number of details to sort out, but nothing that seems like a huge obstacle. Here are the ones that come to mind, along with other details, caveats, and open questions: * the static types exposed in the C-API are PyObject values rather than pointers + I solved this by dereferencing the result of the lookup function (Guido's idea), e.g. #define PyTuple_Type (*(_Py_GetObject_Tuple())) * there is definitely a penalty to using a per-interpreter lookup function + this would only apply to extension modules since internally we would access the PyInterpreterState fields directly + this is mostly a potential problem only when the object is directly referenced frequently (e.g. a tight loop), + the impact would probably center on use of the high-frequency singletons (None, True, False) and possibly with Py*_CheckExact() calls + would it be enough of a problem to be worth mitigating? how would we do so? * static types in extensions can't have tp_base set to a builtin type (since the macro won't resolve) + extensions that support subinterpreters (i.e. PEP 489) won't be using static types (a weak assumption) + extensions that do not support subinterpreters and still have static types would probably break + how to fix that? * limited API extensions built against 3.11+ but running under older Python versions would break? + how to fix that?
But I would prefer the latter, if we can get the performance penalty low enough.
Absolutely. Using immortal objects to solve this is a much simpler solution. -eric
On 16. 12. 21 20:24, Eric Snow wrote:
On Thu, Dec 16, 2021 at 10:54 AM Guido van Rossum <guido@python.org> wrote:
Eric has been looking into this. It's probably the only solution if we can't get immutable objects.
Yep. I've investigated the following approach (for the objects exposed in the public and limited C-API):
* add a pointer field to PyInterpreterState (or a sub-struct) for each of the objects * for the main interpreter, set those pointers to the existing statically declared objects * for subinterpreters make a copy (memcpy()?) and fix it up * add a lookup API and encourage extensions to use it * for 3.11+ change the symbols to macros: + in the internal C-API (Py_BUILD_CORE), the macro would resolve to the corresponding PyInterpreterState field + in the public C-API (and limited API extensions built with 3.11+), the macro would resolve to a call to a (non-inline) lookup function + for limited API extensions built against earlier Python versions we'd still export the existing symbols * limited API extensions built against pre-3.11 Python would only be allowed to run in the main interpreter on 3.11+ + they probably weren't built with subinterpreters in mind anyway
There are still a number of details to sort out, but nothing that seems like a huge obstacle. Here are the ones that come to mind, along with other details, caveats, and open questions:
* the static types exposed in the C-API are PyObject values rather than pointers + I solved this by dereferencing the result of the lookup function (Guido's idea), e.g. #define PyTuple_Type (*(_Py_GetObject_Tuple())) * there is definitely a penalty to using a per-interpreter lookup function + this would only apply to extension modules since internally we would access the PyInterpreterState fields directly + this is mostly a potential problem only when the object is directly referenced frequently (e.g. a tight loop), + the impact would probably center on use of the high-frequency singletons (None, True, False) and possibly with Py*_CheckExact() calls + would it be enough of a problem to be worth mitigating? how would we do so? * static types in extensions can't have tp_base set to a builtin type (since the macro won't resolve) + extensions that support subinterpreters (i.e. PEP 489) won't be using static types (a weak assumption) + extensions that do not support subinterpreters and still have static types would probably break + how to fix that? * limited API extensions built against 3.11+ but running under older Python versions would break? + how to fix that?
With Py_LIMITED_API lower than 3.11, the old API/ABI should continue working. But with none of the new features. Or I guess we could start requiring users to compile stable-ABI extensions with the lowest Python version they support (which is currently just best practice). That would be a major change, though -- and a harsh reminder that the API is "limited", not "stable" like the ABI. All in all, immortals do seem like much better solution, if they can be made to work.
participants (4)
-
Eric Snow -
Guido van Rossum -
Petr Viktorin -
Victor Stinner