[Python-Dev] Sub-interpreters: importing numpy causes hang

[Nick Coghlan]

On Mon, 28 Jan 2019 at 19:36, Stephan Reiter <stephan.reiter at gmail.com> wrote:
>
> Reading through that post, I think I have everything covered but this here:
> - The third and final scenario, and the one where the extended GIL
> state functions for Ensure is still required, is where code doesn't
> have the GIL as yet and wants to make a call into sub interpreter
> rather than the main interpreter, where it already has a pointer to
> the sub interpreter and nothing more. In this case the new
> PyGILState_EnsureEx() function is used, with the sub interpreter being
> passed as argument.
>
> If I understand it correctly, it means the following in practice:
> Whenever I or a third-party library start a new thread, we need to
> query what interpreter we are running at the moment (in the thread
> that is starting the new thread) and pass that information on to the
> new thread so that it can initialize the GIL for itself.
>
> Pseudo code ahead:
> void do_in_thread(func_t *what) {
>   PyThreadState* state = PyThreadState_Get(); /// or new
> PyInterpreterState_Current();
>   PyInterpreterState *interpreter = state->interp;
>   std::thread t([what, interpreter] {
>     auto s = PyGILState_EnsureEx(interpreter);
>     what();
>     PyGILState_Release(s); // could also release before what() because
> TLS was updated and next PyGILState_Ensure() will work
>   });
> }
>
> Did I get that right?

Yeah, I think that's the essence of it, although the other case that
can come up is when the parent thread just created a new
subinterpreter (that only changes how it acquires the pointer though -
the challenge of getting a child thread to make proper use of that
pointer remains the same).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia