[Python-Dev] Sub-interpreters: importing numpy causes hang

[Stephan Reiter]

Hi all!

I am new to the list and arriving with a concrete problem that I'd
like to fix myself.

I am embedding Python (3.6) into my C++ application and I would like
to run Python scripts isolated from each other using sub-interpreters.
I am not using threads; everything is supposed to run in the
application's main thread.

I noticed that if I create an interpreter, switch to it and execute
code that imports numpy (1.13), my application will hang.

  ntdll.dll!NtWaitForSingleObject() Unknown
  KernelBase.dll!WaitForSingleObjectEx() Unknown
> python36.dll!_PyCOND_WAIT_MS(_PyCOND_T * cv=0x00000000748a67a0, _RTL_CRITICAL_SECTION * cs=0x00000000748a6778, unsigned long ms=5) Line 245 C
  [Inline Frame] python36.dll!PyCOND_TIMEDWAIT(_PyCOND_T *) Line 275 C
  python36.dll!take_gil(_ts * tstate=0x0000023251cbc260) Line 224 C
  python36.dll!PyEval_RestoreThread(_ts * tstate=0x0000023251cbc260) Line 370 C
  python36.dll!PyGILState_Ensure() Line 855 C
  umath.cp36-win_amd64.pyd!00007ff8c6306ab2() Unknown
  umath.cp36-win_amd64.pyd!00007ff8c630723c() Unknown
  umath.cp36-win_amd64.pyd!00007ff8c6303a1d() Unknown
  umath.cp36-win_amd64.pyd!00007ff8c63077c0() Unknown
  umath.cp36-win_amd64.pyd!00007ff8c62ff926() Unknown
  [Inline Frame] python36.dll!_PyObject_FastCallDict(_object *) Line 2316 C
  [Inline Frame] python36.dll!_PyObject_FastCallKeywords(_object *) Line 2480 C
  python36.dll!call_function(_object * * *
pp_stack=0x00000048be5f5e40, __int64 oparg, _object * kwnames) Line
4822 C

Numpy's extension umath calls PyGILState_Ensure(), which in turn calls
PyEval_RestoreThread on the (auto) threadstate of the main
interpreter. And that's wrong.
We are already holding the GIL with the threadstate of our current
sub-interpreter, so there's no need to switch.

I know that the GIL API is not fully compatible with sub-interpreters,
as issues #10915 and #15751 illustrate.

But since I need to support calls to PyGILState_Ensure - numpy is the
best example -, I am trying to improve the situation here:
https://github.com/stephanreiter/cpython/commit/d9d3451b038af2820f500843b6a88f57270e1597

That change may be naive, but it does the trick for my use case. If
totally wrong, I don't mind pursuing another alley.

Essentially, I'd like to ask for some guidance in how to tackle this
problem while keeping the current GIL API unchanged (to avoid breaking
modules).

I am also wondering how I can test any changes I am proposing. Is
there a test suite for interpreters, for example?

Thank you very much,
Stephan