[capi-sig]How to access the various levels of runtime state (e.g. PyInterpreterState, _PyRuntimeState)?
by Eric Snow
In https://bugs.python.org/issue36710 Victor wants to move away from
the _PyRuntime C global, instead passing the _PyRuntimeState around
explicitly. I'm in favor of the general idea, but not on a
case-by-case basis like this. I left a comment on that issue
(https://bugs.python.org/msg340945) that explains my position in more
detail. In retrospect, I should have just posted here. :) So I've
copied that comment below, as-is.
FYI, my intention is not to refuse Victor's objective in the issue.
Rather, I want to make sure we have consensus on a valid broader
objective on which to focus. This seemed like a perfect opportunity
to start a discussion about it.
-eric
--------------------------
Status Quo
============
For simplicity sake, let's say nearly all the code operates relative
to the 3 levels of runtime state:
* global - _PyRuntimeState
* interpreter - PyInterpreterState
* thread - PyThreadState
Furthermore, there are 3 groups of functions in the C-API:
* context-sensitive - operate relative to the current Python thread
* runtime-dependent - operate relative to some part of the runtime
state, regardless of thread
* runtime-independent - have nothing to do with CPython's runtime state
Most of the C-API is context-sensitive. A small portion is
runtime-dependent. A handful of functions are runtime-independent
(effectively otherwise stateless helper functions that only happen to
be part of the C-API).
Each context-sensitive function relies on there being a "runtime
context" it can use relative to the current OS thread. That context
consists of the current (i.e. active) PyThreadState, the corresponding
PyInterpreterState, and the global _PyRuntimeState. That context is
derived from data in TSS (see caveats below). This group includes
most of the C-API.
Each runtime-dependent function operates against one or more runtime
state target, regardless of the current thread context (or even if
there isn't one). The target state (e.g. PyInterpreterState) is
always passed explicitly. Again, this is only a small portion of the
C-API.
Caveats:
* for context-sensitive functions, we get the global runtime state
from the global C variable (_PyRuntime) rather than via the implicit
thread context
* for some of the runtime-dependent functions that target
_PyRuntimeState, we rely on the global C variable
All of this is the pattern we use currently. Using TSS to identify
the implicit runtime context has certain benefits and costs:
benefits:
* sticking with the status quo means no backward incompatibility for
existing C-extension code
* easier to distinguish the context-sensitive functions from the
runtime-dependent ones
* (debatable) callers don't have to track, nor pass through, an extra argument
costs:
* extra complexity in keeping TSS correct
* makes the C-API bigger (extra macros, etc.)
Alternative
=============
For every context-sensitive function we could add a new first
parameter, "context", that provides the runtime context to use. That
would be something like this:
struct {
PyThreadState *tstate;
...
} PyRuntimeContext;
The interpreter state and global runtime state would still be
accessible via the same indirection we have now.
Taking this alternative would eliminate the previous costs. Having a
consistent "PyRuntimeContext *context" first parameter would maintain
the easy distinction from runtime-dependent functions. Asking callers
to pass in the context explicitly is probably better regardless. As
to backward compatibility, we could maintain a shim to bridge between
the old way and the new.
About the C-global _PyRuntime
==============================
Currently the global runtime state (_PyRuntimeState) is stored in a
static global C variable, _PyRuntime. I added it at the time I
consolidated many of the existing C globals into a single struct.
Having a C global makes it easy to do the wrong thing, so it may be
good to do something else.
That would mean allocating a _PyRuntimeState on the heap early in
startup and pass that around where needed. I expect that would not
have any meaningful performance penalty. It would probably also
simplify some of the code we currently use to manage _PyRuntime
correctly.
As a bonus, this would be important if we decided that
multiple-runtimes-per-process were a desirable thing. That's a neat
idea, though I don't see a need currently. So on its own it's not
really a justification for dropping a static _PyRuntime. :) However,
I think the other reasons are enough.
Conclusions
====================
This issue has a specific objective that I think is premature. We
have an existing pattern and we should stick with that until we decide
to change to a new pattern. That said, a few things should get
corrected and we should investigate alternative patterns for the
context-sensitive C-API.
As to getting rid of the _PyRuntime global variable in favor of
putting it on the heap, I'm not opposed. However, doing so should
probably be handled in a separate issue.
Here are my thoughts on actionable items:
1. look for a better pattern for the context-sensitive C-API
2. clearly document which of the 3 groups each C-API function belongs to
3. add a "runtime" field to the PyInterpreterState pointing to the
parent _PyRuntimeState
4. (maybe) add a _PyRuntimeState_GET() macro, a la PyThreadState_GET()
5. for context-sensitive C-API that uses the global runtime state, get
it from the current PyInterpreterState
6. for runtime-dependent C-API that targets the global runtime state,
ensure the _PyRuntimeState is always an explicit parameter
7. (maybe) drop _PyRuntime and create a _PyRuntimeState on the heap
during startup to pass around
3 months, 3 weeks
- 7
- 17