PyThreadState_GET() returns NULL from within PyImport_GetModuleDict()
I'm working on Windows. I have the following dead simple embedding code that I'm using to test out the python interpreter: ============================================================ Py_SetProgramName(L"MyApp"); Py_SetPath( L"C:\\Users\\rutski\\Documents\\python\\PCBuild\\amd64\\python36.zip;" L"C:\\Users\\rutski\\Documents\\python\\DLLs;" L"C:\\Users\\rutski\\Documents\\python\\lib;" L"C:\\Users\\rutski\\Documents\\python\\PCBuild\\amd64;" L"C:\\Users\\rutski\\Documents\\python;" L"C:\\Users\\rutski\\Documents\\python\\lib\\site-packages"); Py_Initialize(); PyRun_SimpleString( "from time import time,ctime\n" "print('Today is', ctime(time()))\n"); ============================================================ This code crashes trying to access address 0x00000010 from within PyRun_SimpleString(). The sequence of event's is this: 1) PyRun_SimpleString() tries to do AddModule("__main__") 2) AddModule tries to do PyImport_GetModuleDict() 3) PyImport_GetModuleDict() tries to doPyThreadState_GET()->interp 4) PyThreadState_GET() returns NULL, so the ->interp part crashes. The weird thing is that calling PyImport_GetModuleDict() from within my application directly works just fine. Weirder still is that the whole thing actually executes fine if I build a windows command line application with the embed code in main(), and execute it from a terminal. The crash only happens when building a Windows GUI application and calling the embed code in WinMain(). This is a python interpreter that I built from source on windows using PCbuild\build.bat, so that I could track the crash. However, the exact same crash was happening with the stock interpreter provided by the python windows installer. Does anyone have any ideas here? -Patrick
On Mon, Aug 7, 2017 at 8:44 PM, Larry Hastings <larry@hastings.org> wrote:
My one idea: the GIL isn't initialized until you create a new thread.
That didn't seem to be it. I put a CreateThread() call right after Py_Initialize(), and that didn't fix it. I also moved it around to before Py_Initialize() and various other places, and nothing helped.
After much stumbling around in the dark I have the answer. The following table shows the results of tests which I performed pairing different build configurations of my WinMain() app with different python libraries. The terms "Release" and "Debug" refer to the configuration type in Visual Studio. All builds were done in x64 mode. Debug | python36.lib | Works Debug | python36_d.lib | Works Debug | python3_d.lib | Works Debug | python3.lib | !!! CRASHES !!! Release | python36.lib | Works Release | python36_d.lib | Works Release | python3_d.lib | !!! CRASHES !!! Release | python3.lib | Works So, it seems to be the case that picking a mismatched python binary causes the crash, __but only with python3, not with python36__. This makes me wonder what the differences is between the two in the first place. I was getting the crash to begin with because I was linking my Debug build with the release build python3.lib, since I thought it shouldn't matter. My problem is fixed now, but if anyone could sheld light on the details of why exactly it happened then I would certainy be interested. On Mon, Aug 7, 2017 at 8:11 PM, Patrick Rutkowski <patrick.rutkowski@gmail.com> wrote:
I'm working on Windows. I have the following dead simple embedding code that I'm using to test out the python interpreter:
============================================================
Py_SetProgramName(L"MyApp");
Py_SetPath( L"C:\\Users\\rutski\\Documents\\python\\PCBuild\\amd64\\python36.zip;" L"C:\\Users\\rutski\\Documents\\python\\DLLs;" L"C:\\Users\\rutski\\Documents\\python\\lib;" L"C:\\Users\\rutski\\Documents\\python\\PCBuild\\amd64;" L"C:\\Users\\rutski\\Documents\\python;" L"C:\\Users\\rutski\\Documents\\python\\lib\\site-packages");
Py_Initialize();
PyRun_SimpleString( "from time import time,ctime\n" "print('Today is', ctime(time()))\n");
============================================================
This code crashes trying to access address 0x00000010 from within PyRun_SimpleString(). The sequence of event's is this:
1) PyRun_SimpleString() tries to do AddModule("__main__") 2) AddModule tries to do PyImport_GetModuleDict() 3) PyImport_GetModuleDict() tries to doPyThreadState_GET()->interp 4) PyThreadState_GET() returns NULL, so the ->interp part crashes.
The weird thing is that calling PyImport_GetModuleDict() from within my application directly works just fine. Weirder still is that the whole thing actually executes fine if I build a windows command line application with the embed code in main(), and execute it from a terminal. The crash only happens when building a Windows GUI application and calling the embed code in WinMain().
This is a python interpreter that I built from source on windows using PCbuild\build.bat, so that I could track the crash. However, the exact same crash was happening with the stock interpreter provided by the python windows installer.
Does anyone have any ideas here? -Patrick
On 07Aug2017 2231, Patrick Rutkowski wrote:
So, it seems to be the case that picking a mismatched python binary causes the crash, __but only with python3, not with python36__. This makes me wonder what the differences is between the two in the first place. I was getting the crash to begin with because I was linking my Debug build with the release build python3.lib, since I thought it shouldn't matter.
My problem is fixed now, but if anyone could sheld light on the details of why exactly it happened then I would certainy be interested.
I'd love to be able to, but honestly I have basically no idea. There is only one implementation of PyThreadState_Get() (which is what the PyThreadState_GET macro should map to), and that should never return NULL without terminating the entire process. My best guess is that the API forwarding being done by python3.dll is failing somehow and is turning into a function that returns 0. Since you are embedding rather than building an extension module, I'd suggest linking directly to python36[_d].dll, since you will presumably be including the Python runtime with your application (this is exactly the use case for the embeddable package, in case you hadn't seen that). The python3[_d].dll is most useful when building a .pyd that may be used with multiple versions of Python 3. It has some limitations and many bugs though, which I don't know we can ever fully resolve, but all of which can be avoided in your case by not using it :) Hope that helps, Steve
participants (3)
-
Larry Hastings -
Patrick Rutkowski -
Steve Dower