[Python-Dev] thread issues when embedding Python

[Daniel Pocock]

Another link that fills in some gaps and finally helped me make this work:

http://www.codevate.com/blog/7-concurrency-with-embedded-python-in-a-multi-threaded-c-application

In particular, I found that PyGILState_Ensure/PyGILState_Release as
described in the Python docs is not sufficient - as described in that
blog link, I had to

a) obtain PyInterpreterState from the first thread where Py_Initialize()
was called

b) when each worker thread starts, call
PyThreadState_New(mInterpreterState)  and save the result in a thread
local mPyThreadState

c) use the mPyThreadState with PyEval_RestoreThread and
PyEval_SaveThread before and after calling Python methods

Is this a bug in PyGILState_Ensure or is it a deficiency in the
documentation?

I also found one bug in my own code, although that was not related to
the problem just described with PyGILState_Ensure and I had to fix both
problems to make it work.  Specifically, the PyWorkerThread constructor
was taking an object argument when it should have taken a reference
argument and this was creating an invalid Py::Callable member in my worker.

On 18/12/13 00:19, Daniel Pocock wrote:
>
> I've successfully embedded Python for a single thread
>
> I tried to extend the implementation for multiple threads (a worker
> thread scenario) and I'm encountering either deadlocks or seg faults
> depending upon how I got about it.
>
> There seems to be some inconsistency between what is covered in the docs
> here:
>
> http://docs.python.org/2/c-api/init.html#non-python-created-threads
>
> and the experiences of other users trying the same thing, e.g.
>
> http://bugs.python.org/issue19576
> http://wiki.blender.org/index.php/Dev:2.4/Source/Python/API/Threads
>
> Can anybody comment on the situation, in particular,
>
> Is the non-python-created-threads documentation accurate for v2.7?
>
> If a main thread does things like importing a module and obtaining a
> reference to a Python method, can those things be used by other C++
> threads or do they have to repeat those lookups?
>
> Is there any logic that needs to be executed once only as each thread is
> started? (the doc suggests just PyGILState_Ensure/PyGILState_Release
> each time a thread accesses Python methods - is there anything else?)
>
> Given the bug 19576, what is the most portable way to code this to work
> reliably on unfixed Python versions?  (e.g. should users always
> explicitly call PyEval_InitThreads() in their main thread or worker
> threads or both?)
>
>
>
>
> Here is my actual source code:
>
> https://svn.resiprocate.org/viewsvn/resiprocate/main/repro/plugins/pyroute/
>
>   (see example.py for a trivial example of what it does)
>
> The problem that I encounter:
>
> - the init stuff runs fine in PyRoutePlugin.cxx,
>     it calls Py_Initialize, PyEval_InitThreads, PyImport_ImportModule
>       and looks up the "provide_route" method in the module
>     it creates a PyRouteWorker object,
>       giving it a reference to "provide_route"
>     it creates a thread pool to run the worker
>
> - the PyRouteWorker::process() method is invoked in one of those threads
>
> - it crashes when trying to call the "provide_route" method
>     PyRouteWorker.cxx:
>       routes = mAction.apply(args);
>
>
> Program received signal SIGSEGV, Segmentation fault.
> [Switching to Thread 0x7ffff30b8700 (LWP 23965)]
> 0x00007ffff3d6ad06 in PyObject_Call () from /usr/lib/libpython2.7.so.1.0
> (gdb) bt
> #0  0x00007ffff3d6ad06 in PyObject_Call () from /usr/lib/libpython2.7.so.1.0
> #1  0x00007ffff3d6b647 in PyEval_CallObjectWithKeywords ()
>    from /usr/lib/libpython2.7.so.1.0
> #2  0x00007ffff414885a in apply (args=..., this=<optimized out>)
>     at /usr/include/python2.7/CXX/Python2/Objects.hxx:3215
> #3  repro::PyRouteWorker::process (this=0x6f00a0, msg=<optimized out>)
>     at PyRouteWorker.cxx:98
> #4  0x00007ffff7b879e1 in repro::WorkerThread::thread (this=0x68e110)
>     at WorkerThread.cxx:36
> #5  0x00007ffff70b7a2f in threadIfThreadWrapper (threadParm=<optimized out>)
>     at ThreadIf.cxx:51
> #6  0x00007ffff65ffb50 in start_thread (arg=<optimized out>)
>     at pthread_create.c:304
> #7  0x00007ffff5999a7d in clone ()
>     at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> #8  0x0000000000000000 in ?? ()
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/daniel%40pocock.com.au