[Python-Dev] Sub-interpreters: importing numpy causes hang

[Nick Coghlan]

On Thu, 24 Jan 2019 at 05:45, Stephan Reiter <stephan.reiter at gmail.com> wrote:
> If we create a fresh OS thread and make it call PyGILState_Ensure, it
> won't have a PyThreadState saved under autoTLSkey. That means it will
> create one using the main interpreter. I, as the developer embedding
> Python into my application and using multiple interpreters, have no
> control here. Maybe I know that under current conditions a certain
> other interpreter should be used.
>
> I'll try to provoke this situation and then introduce a callback from
> Python into my application that will allow me to specify which
> interpreter should be used, e.g. code as follows:
>
> PyInterpreter *pickAnInterpreter() {
>   return activePlugin ? activePlugin->interpreter : nullptr; //
> nullptr maps to main interpreter
> }
>
> PyGILState_SetNewThreadInterpreterSelectionCallback(&pickAnInterpreter);
>
> Maybe rubbish. But I think a valuable experiment that will give me a
> better understanding.

That actually sounds like a pretty plausible approach to me, at least
for cases where the embedding application maintains some other state
that lets it know which interpreter a new thread should be associated
with. The best aspect of it is that it would let the embedding
application decide how to handle registration of previously unknown
threads with the Python runtime *without* requiring that all existing
extension modules switch to a new thread registration API first.

I'll pass the concept along to Graham Dumpleton (author of the
mod_wsgi module for Apache httpd) to see if an interface like this
might be enough to resolve some of the major compatibility issues
mod_wsgi currently encounters with subinterpreters.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia