Adding a threadlocal to the Python interpreter
I would like to take another stab at adding a threadlocal "str(bytes) raises an exception" to the Python interpreter, but I had a very hard time understanding both how to add a threadlocal value to either the interpreter state or the threadlocal dict that is part of that state, and then how to access the same value from both Python and CPython code. The structs were there but it was just hard to understand. Can someone explain it to me? Thanks, Daniel Holth
On 2016-05-18 15:20, Daniel Holth wrote:
I would like to take another stab at adding a threadlocal "str(bytes) raises an exception" to the Python interpreter, but I had a very hard time understanding both how to add a threadlocal value to either the interpreter state or the threadlocal dict that is part of that state, and then how to access the same value from both Python and CPython code. The structs were there but it was just hard to understand. Can someone explain it to me?
Python has a two important states related to threads. The PyInterpreterState contains the state of an interpreter instance: sys module, loaded modules and a couple of additional settings. Usually there is just one interpreter state in a Python process. Additional interpreter states are used to implement subinterpreters. Each C thread, that wants to run Python code, must have a PyThreadState. The thread state contains a reference to a PyInterpreterState. Each PyThreadState has a PyObject *dict member. You can stick Python objects into the dict. The interpreter cleans up the dict when it reaps a thread. How performance critical is your code? Does the interpreter have to check the value of the thread local frequently? In that case you should add a new member to typedef struct _ts PyThreadState in pystate.h right before /* XXX signal handlers should also be here */. Otherwise you can simply use PyThreadState_GetDict(). It returns a Python dict object that is local to the current thread. You can simply use a fixed key like in Modules/_decimal/_decimal.c. Christian
On 18 May 2016 at 23:20, Daniel Holth
I would like to take another stab at adding a threadlocal "str(bytes) raises an exception" to the Python interpreter, but I had a very hard time understanding both how to add a threadlocal value to either the interpreter state or the threadlocal dict that is part of that state, and then how to access the same value from both Python and CPython code. The structs were there but it was just hard to understand. Can someone explain it to me?
Christian covered the C aspects of the API, while the general purpose Python aspects live in the threading module. However, the Python level thread-local API doesn't provide direct access to the thread state dict. Instead, it provides access to subdicts stored under per-object keys in that dict, keyed as "thread.local.<id>": * Key definition in local_new: https://hg.python.org/cpython/file/tip/Modules/_threadmodule.c#l705 * Subdict creation in _ldict: https://hg.python.org/cpython/file/tip/Modules/_threadmodule.c#l810 Getting access to state stored that way from C is *possible*, but significantly less convenient than access the thread state directly. What that means is that any time we want to expose thread local state to both C and Python code, it will generally be the responsibility of the C code to both manage the key in the thread state dict (or the field in the thread state struct), and also to provide a Python API for access that state. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 2016-05-19 04:30, Nick Coghlan wrote:
On 18 May 2016 at 23:20, Daniel Holth
wrote: I would like to take another stab at adding a threadlocal "str(bytes) raises an exception" to the Python interpreter, but I had a very hard time understanding both how to add a threadlocal value to either the interpreter state or the threadlocal dict that is part of that state, and then how to access the same value from both Python and CPython code. The structs were there but it was just hard to understand. Can someone explain it to me?
Christian covered the C aspects of the API, while the general purpose Python aspects live in the threading module.
However, the Python level thread-local API doesn't provide direct access to the thread state dict. Instead, it provides access to subdicts stored under per-object keys in that dict, keyed as "thread.local.<id>":
In case you wonder about subdicts, they are required to provide multiple thread local objects. Each thread local instance has its own key in each thread state dict. The approach enables thread local to have independent storage objects. Christian
Thank you. I started my patch http://bugs.python.org/issue27134 to allow
Python code to set a flag causing the evil str(b'bytes') to raise an
exception. I wasn't sure exactly which module to put it in, so it's in
_string. Please let me know the best place to put the feature and what I
should polish to get it in.
Thanks,
Daniel Holth
On Thu, May 19, 2016 at 6:34 AM Christian Heimes
On 2016-05-19 04:30, Nick Coghlan wrote:
On 18 May 2016 at 23:20, Daniel Holth
wrote: I would like to take another stab at adding a threadlocal "str(bytes) raises an exception" to the Python interpreter, but I had a very hard time understanding both how to add a threadlocal value to either the interpreter state or the threadlocal dict that is part of that state, and then how to access the same value from both Python and CPython code. The structs were there but it was just hard to understand. Can someone explain it to me?
Christian covered the C aspects of the API, while the general purpose Python aspects live in the threading module.
However, the Python level thread-local API doesn't provide direct access to the thread state dict. Instead, it provides access to subdicts stored under per-object keys in that dict, keyed as "thread.local.<id>":
In case you wonder about subdicts, they are required to provide multiple thread local objects. Each thread local instance has its own key in each thread state dict. The approach enables thread local to have independent storage objects.
Christian
participants (3)
-
Christian Heimes
-
Daniel Holth
-
Nick Coghlan