The GIL and PyEval_RestoreThread

[MRAB]

On 2023-09-26 14:20, Peter Ebden via Python-list wrote:
> Hi all,
> 
> I've been working on embedding Python and have an interesting case around
> locking with PyEval_RestoreThread which wasn't quite doing what I expect,
> hoping someone can explain what I should expect here.
> 
> I have a little example (I'm running this in parallel from two different
> threads; I have some more C code for that but I don't think it's super
> interesting):
> 
> void run_python(PyThreadState* thread) {
>    LOG("Restoring thread %p...", thread);
>    PyEval_RestoreThread(thread);
>    LOG("Restored thread %p", thread);
>    PyRun_SimpleString("import time; print('sleeping'); time.sleep(3.0)");
>    LOG("Saving thread...");
>    PyThreadState* saved_thread = PyEval_SaveThread();
>    LOG("Saved thread %p", saved_thread);
> }
> 
> This produces output like
> 11:46:48.110058893: Restoring thread 0xabc480...
> 11:46:48.110121656: Restored thread 0xabc480
> 11:46:48.110166060: Restoring thread 0xabc480...
> sleeping
> 11:46:48.110464194: Restored thread 0xabc480
> sleeping
> 11:46:51.111307541: Saving thread...
> 11:46:51.111361075: Saved thread 0xabc480
> 11:46:51.113116633: Saving thread...
> 11:46:51.113177605: Saved thread 0xabc480
> 
> The thing that surprises me is that both threads seem to be able to pass
> PyEval_RestoreThread before either reaches the corresponding
> PyEval_SaveThread call, which I wasn't expecting to happen; I assumed that
> since RestoreThread acquires the GIL, that thread state would remain locked
> until it's released.
> 
> I understand that the system occasionally switches threads, which I guess
> might well happen with that time.sleep() call, but I wasn't expecting the
> same thread to become usable somewhere else. Maybe I am just confusing
> things by approaching the same Python thread from multiple OS threads
> concurrently and should be managing my own locking around that?
> 
Storing the result of PyEval_SaveThread in a local variable looks wrong 
to me.

In the source for the regex module, I release the GIL with 
PyEval_SaveThread and save its result. Then, when I want to claim the 
GIL, I pass that saved value to PyEval_RestoreThread.

You seem to be releasing the GIL and discarding the result, so which 
thread are you resuming when you call PyEval_RestoreThread?

It looks like you're resuming the same thread twice. As it's already 
resumed the second time, no wonder it's not blocking!