[Python-Dev] thread issues when embedding Python

[Daniel Pocock]

I've successfully embedded Python for a single thread

I tried to extend the implementation for multiple threads (a worker
thread scenario) and I'm encountering either deadlocks or seg faults
depending upon how I got about it.

There seems to be some inconsistency between what is covered in the docs
here:

http://docs.python.org/2/c-api/init.html#non-python-created-threads

and the experiences of other users trying the same thing, e.g.

http://bugs.python.org/issue19576
http://wiki.blender.org/index.php/Dev:2.4/Source/Python/API/Threads

Can anybody comment on the situation, in particular,

Is the non-python-created-threads documentation accurate for v2.7?

If a main thread does things like importing a module and obtaining a
reference to a Python method, can those things be used by other C++
threads or do they have to repeat those lookups?

Is there any logic that needs to be executed once only as each thread is
started? (the doc suggests just PyGILState_Ensure/PyGILState_Release
each time a thread accesses Python methods - is there anything else?)

Given the bug 19576, what is the most portable way to code this to work
reliably on unfixed Python versions?  (e.g. should users always
explicitly call PyEval_InitThreads() in their main thread or worker
threads or both?)




Here is my actual source code:

https://svn.resiprocate.org/viewsvn/resiprocate/main/repro/plugins/pyroute/

  (see example.py for a trivial example of what it does)

The problem that I encounter:

- the init stuff runs fine in PyRoutePlugin.cxx,
    it calls Py_Initialize, PyEval_InitThreads, PyImport_ImportModule
      and looks up the "provide_route" method in the module
    it creates a PyRouteWorker object,
      giving it a reference to "provide_route"
    it creates a thread pool to run the worker

- the PyRouteWorker::process() method is invoked in one of those threads

- it crashes when trying to call the "provide_route" method
    PyRouteWorker.cxx:
      routes = mAction.apply(args);


Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff30b8700 (LWP 23965)]
0x00007ffff3d6ad06 in PyObject_Call () from /usr/lib/libpython2.7.so.1.0
(gdb) bt
#0  0x00007ffff3d6ad06 in PyObject_Call () from /usr/lib/libpython2.7.so.1.0
#1  0x00007ffff3d6b647 in PyEval_CallObjectWithKeywords ()
   from /usr/lib/libpython2.7.so.1.0
#2  0x00007ffff414885a in apply (args=..., this=<optimized out>)
    at /usr/include/python2.7/CXX/Python2/Objects.hxx:3215
#3  repro::PyRouteWorker::process (this=0x6f00a0, msg=<optimized out>)
    at PyRouteWorker.cxx:98
#4  0x00007ffff7b879e1 in repro::WorkerThread::thread (this=0x68e110)
    at WorkerThread.cxx:36
#5  0x00007ffff70b7a2f in threadIfThreadWrapper (threadParm=<optimized out>)
    at ThreadIf.cxx:51
#6  0x00007ffff65ffb50 in start_thread (arg=<optimized out>)
    at pthread_create.c:304
#7  0x00007ffff5999a7d in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#8  0x0000000000000000 in ?? ()