[Python-Dev] Sub-interpreters: importing numpy causes hang

[Antoine Pitrou]

On Tue, 22 Jan 2019 15:32:22 +0100
Stephan Reiter <stephan.reiter at gmail.com> wrote:
> 
> Numpy's extension umath calls PyGILState_Ensure(), which in turn calls
> PyEval_RestoreThread on the (auto) threadstate of the main
> interpreter. And that's wrong.
> We are already holding the GIL with the threadstate of our current
> sub-interpreter, so there's no need to switch.
> 
> I know that the GIL API is not fully compatible with sub-interpreters,
> as issues #10915 and #15751 illustrate.

That's a pity.

Note that there is a patch on https://bugs.python.org/issue10915 that
could probably solve the issue if it had been applied some years ago ;-)

(yes, it needs C extension authors to use the new API, but Numpy is a
well-maintained library and would probably have accepted a patch for
that; so would Cython probably)

> Essentially, I'd like to ask for some guidance in how to tackle this
> problem while keeping the current GIL API unchanged (to avoid breaking
> modules).

I'm not aware of any solution which does not require designing a new
API, unfortunately.

> I am also wondering how I can test any changes I am proposing. Is
> there a test suite for interpreters, for example?

You'll find a couple of them in test_embed.py, test_capi.py and
test_threading.py.

Regards

Antoine.