Embeded Multi-Threaded Python: PyEval_InitThread(), PyEval_SaveThread(),...

[Titus Brown]

In article <8e0ac4fb.0109191028.5c2c1cda at posting.google.com>,
Nicolas Duchastel <nicolas at otelnet.com> wrote:
>I am trying to use the Python/C API function to get an embeded Interpreter
>into a multi-threaded C++ application. We are using Pthreads on Solaris 7.
>My application works fine, but only with 1 thread; and then, when I stop it,
>it core dumps.
>I am using Python 2.1.1 built on Solaris 7 with the SUN's C++ 5.2 compiler.

Hi, Nicolas,

you might be interested in the PyWX source code: pywx.idyll.org/.  It's
an LGPL'd project that put an embedded Python interpreter into AOLserver,
and it (correctly ;) deals with all of the issues you are facing.

Keeping the intricacies of the Python threading model in my head doesn't
seem to have worked too well, but here goes:

>I have an C++ application which creates a bunch of threads and then
>accepts messages. Each messages is a request to execute some Python code.
>Here's some pseudo-code of a 4 threaded system:
>   thread A: // INIT thread
>       do init stuff;
>       init Python interpreter:
>           Py_Initializ();
>           PyImport_ImportModule("MyModule");
>           ...
>       start thread B;
>       start thread C;
>       end this thread;

I would do things in this order:

--
Py_Initialize();
PyEval_InitThreads();			// grabs global lock

/* start pthreads stuff -- actual OS threading -- here */
--

Before you do things with threads B and C, you'll want to have a separate
Interpreter or ThreadState.

>   thread B and C: // WORKER thread
>       wait and receive a new message;
>       if (msg is exit_message)
>          start Thread D;
>          stop other worker thread (B or C);
>          end this thread;
>       else
>          call python code:
>             PyRun_SimpleString(...);
>            OR
>             PyObbject_CallMethod(...);

--
// grab Python lock
// if exit message, tell all other threads to exit and alert D;
// else call Python code
// release Python lock
--
>   thread D: // CLEAN-UP thread
>      wait for threads B and C to be stopped;
>      clean-up stuff
>      Py_Finalize();

what you have above, roughly.

>QUESTIONS:
>---------
> 1) what are the calls that I should be making !? in what order ?
>    any examples ? (i.e. not example of Python code executing some
>    multi-threaded stuff using the thread module, but rather some C/C++
>    code using the API C functions to implement a multi-threaded Interpreter).

See PyWX for actual examples...

> 2) what is the init sequence ? i.e. should PyEval_InitThreads be called ?
>    if so, before or after Py_Initialized() ? What objects need to be created ?
>    How ? (e.g. how do you create the locks ? what about the Thread State ?)

Py_Initialize should be followed by InitThreads.

> 3) does the Python interpreter remember which OS thread is running what ?

No.

>    i.e. one of my hypothesis for explaining the core dump is that the
>         interpreter is saying something like "Hey! You called Py_Initialize()
>         on thread t at 1, you called PyObject_CallMethod() on thread t at 4 and
>         now you are calling Py_Finalize() on thread t at 7; what the #@$ are
>         you trying to do!?"

That sounds pretty reasonable to me ;).

>    ---> do I need to create a ThreadState object for every OS thread which
>         could possibly use the Python Interpreter ? or is it, on per
>         concurent thread ?

I believe, technically, that you'll want to create a ThreadState object
for every Python execution process whose execution you want interleaved.
ThreadStates execute Python code in sequence...

> 4) when executing my message (i.e. logic in threads B or C), what do I
>    have to do to lock ? When do I need to lock ? or unlock ?
>    Which methods: PyEval_SavedThread(), PyEval_AcquireThread(),
>                   PyEval_AcquireLock(), Py_BEGIN_ALLOW_THREADS,.... ?

You'll want to allow threads *after* your C++ code is finished doing
C API calls into Python, and grab the lock *before* your C++ code starts
doing C API calls into Python.  Python code itself is sprinkled with the
statements so you don't need to worry about executing bodies of Python code.

Note also that PyEval_InitThreads grabs the lock, so you'll need to do a
lock release before the end of your A function.

It took me a few weeks to get PyWX working well with all of this; I'd be happy
to correspond with you after you take a look at that, to help explain stuff.
I find that the C API in Python for this stuff is confusing unless you have
a handle on what functions exist ;), so that's why I'd suggest starting to play.

Once it all works, it's great, though ;).

cheers,
--titus