<div dir="auto">There are currently numerous incompatibilities between numpy and subinterpreters, and no concrete plan for fixing them. <span style="font-family:sans-serif">The numpy team does not consider subinterpreters to be a supported configuration, and can't help you with any issues you run into. I know the concept of subinterpreters is really appealing, but unfortunately the CPython implementation is not really mature or widely supported... ar</span><span style="font-family:sans-serif">e you absolutely certain you need to use subinterpreters for your application?</span></div><br><div class="gmail_quote"><div dir="ltr">On Tue, Jan 22, 2019, 08:27 Stephan Reiter <<a href="mailto:stephan.reiter@gmail.com">stephan.reiter@gmail.com</a> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi all!<br>
<br>
I am new to the list and arriving with a concrete problem that I'd<br>
like to fix myself.<br>
<br>
I am embedding Python (3.6) into my C++ application and I would like<br>
to run Python scripts isolated from each other using sub-interpreters.<br>
I am not using threads; everything is supposed to run in the<br>
application's main thread.<br>
<br>
I noticed that if I create an interpreter, switch to it and execute<br>
code that imports numpy (1.13), my application will hang.<br>
<br>
ntdll.dll!NtWaitForSingleObject() Unknown<br>
KernelBase.dll!WaitForSingleObjectEx() Unknown<br>
> python36.dll!_PyCOND_WAIT_MS(_PyCOND_T * cv=0x00000000748a67a0, _RTL_CRITICAL_SECTION * cs=0x00000000748a6778, unsigned long ms=5) Line 245 C<br>
[Inline Frame] python36.dll!PyCOND_TIMEDWAIT(_PyCOND_T *) Line 275 C<br>
python36.dll!take_gil(_ts * tstate=0x0000023251cbc260) Line 224 C<br>
python36.dll!PyEval_RestoreThread(_ts * tstate=0x0000023251cbc260) Line 370 C<br>
python36.dll!PyGILState_Ensure() Line 855 C<br>
umath.cp36-win_amd64.pyd!00007ff8c6306ab2() Unknown<br>
umath.cp36-win_amd64.pyd!00007ff8c630723c() Unknown<br>
umath.cp36-win_amd64.pyd!00007ff8c6303a1d() Unknown<br>
umath.cp36-win_amd64.pyd!00007ff8c63077c0() Unknown<br>
umath.cp36-win_amd64.pyd!00007ff8c62ff926() Unknown<br>
[Inline Frame] python36.dll!_PyObject_FastCallDict(_object *) Line 2316 C<br>
[Inline Frame] python36.dll!_PyObject_FastCallKeywords(_object *) Line 2480 C<br>
python36.dll!call_function(_object * * *<br>
pp_stack=0x00000048be5f5e40, __int64 oparg, _object * kwnames) Line<br>
4822 C<br>
<br>
Numpy's extension umath calls PyGILState_Ensure(), which in turn calls<br>
PyEval_RestoreThread on the (auto) threadstate of the main<br>
interpreter. And that's wrong.<br>
We are already holding the GIL with the threadstate of our current<br>
sub-interpreter, so there's no need to switch.<br>
<br>
I know that the GIL API is not fully compatible with sub-interpreters,<br>
as issues #10915 and #15751 illustrate.<br>
<br>
But since I need to support calls to PyGILState_Ensure - numpy is the<br>
best example -, I am trying to improve the situation here:<br>
<a href="https://github.com/stephanreiter/cpython/commit/d9d3451b038af2820f500843b6a88f57270e1597" rel="noreferrer noreferrer" target="_blank">https://github.com/stephanreiter/cpython/commit/d9d3451b038af2820f500843b6a88f57270e1597</a><br>
<br>
That change may be naive, but it does the trick for my use case. If<br>
totally wrong, I don't mind pursuing another alley.<br>
<br>
Essentially, I'd like to ask for some guidance in how to tackle this<br>
problem while keeping the current GIL API unchanged (to avoid breaking<br>
modules).<br>
<br>
I am also wondering how I can test any changes I am proposing. Is<br>
there a test suite for interpreters, for example?<br>
<br>
Thank you very much,<br>
Stephan<br>
_______________________________________________<br>
Python-Dev mailing list<br>
<a href="mailto:Python-Dev@python.org" target="_blank" rel="noreferrer">Python-Dev@python.org</a><br>
<a href="https://mail.python.org/mailman/listinfo/python-dev" rel="noreferrer noreferrer" target="_blank">https://mail.python.org/mailman/listinfo/python-dev</a><br>
Unsubscribe: <a href="https://mail.python.org/mailman/options/python-dev/njs%40pobox.com" rel="noreferrer noreferrer" target="_blank">https://mail.python.org/mailman/options/python-dev/njs%40pobox.com</a><br>
</blockquote></div>