[Python-Dev] Sub-interpreters: importing numpy causes hang

[Stephan Reiter]

Cool. Thanks, Nick!

I did experiments based on this idea (
https://github.com/stephanreiter/cpython/commit/3bca91c26ac81e517b4aa22302be1741b3315622)
and haven't rejected it yet. :-)

Together with the other fix (
https://github.com/stephanreiter/cpython/commit/c1afa0c8cdfab862f409f1c7ff02b189f5191cbe),
numpy at least is happy in my Python-hosting app.

I will pursue the idea of swapping sys.modules in a single Interpreter now
because that wouldn't require patching Python and I might get the mileage
out of this approach I need.

Still interested in improving sub-interpreters, though. I just need to
balance short and long term solution. :-)

Stephan


Den søn. 27. jan. 2019, 15.17 skrev Nick Coghlan <ncoghlan at gmail.com:

> On Thu, 24 Jan 2019 at 05:45, Stephan Reiter <stephan.reiter at gmail.com>
> wrote:
> > If we create a fresh OS thread and make it call PyGILState_Ensure, it
> > won't have a PyThreadState saved under autoTLSkey. That means it will
> > create one using the main interpreter. I, as the developer embedding
> > Python into my application and using multiple interpreters, have no
> > control here. Maybe I know that under current conditions a certain
> > other interpreter should be used.
> >
> > I'll try to provoke this situation and then introduce a callback from
> > Python into my application that will allow me to specify which
> > interpreter should be used, e.g. code as follows:
> >
> > PyInterpreter *pickAnInterpreter() {
> >   return activePlugin ? activePlugin->interpreter : nullptr; //
> > nullptr maps to main interpreter
> > }
> >
> > PyGILState_SetNewThreadInterpreterSelectionCallback(&pickAnInterpreter);
> >
> > Maybe rubbish. But I think a valuable experiment that will give me a
> > better understanding.
>
> That actually sounds like a pretty plausible approach to me, at least
> for cases where the embedding application maintains some other state
> that lets it know which interpreter a new thread should be associated
> with. The best aspect of it is that it would let the embedding
> application decide how to handle registration of previously unknown
> threads with the Python runtime *without* requiring that all existing
> extension modules switch to a new thread registration API first.
>
> I'll pass the concept along to Graham Dumpleton (author of the
> mod_wsgi module for Apache httpd) to see if an interface like this
> might be enough to resolve some of the major compatibility issues
> mod_wsgi currently encounters with subinterpreters.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190127/e4e0bb59/attachment.html>