Problems with Threads

[David Bolen]

"Alexandre Perrin" <payou at free.fr> writes:

> I had the same problem recently... In fact, it mainly depends on the
> kind of thread you are using.  I'm using Windows and I know that
> Win32 threads are incompatible with Python (because the Python API
> is single-threaded), but I heard that POSIX threads work fine.

There's nothing wrong with calling into the Python interpreter from
multiple native windows threads created outside of Python.  After all,
under Windows (native Python, not Cygwin) Python threads are in fact
created as native Win32 threads.

However, you do have to pay attention to the global interpreter lock.
Thus, if you are going to create a thread outside of Python (say from
an environment into which the Python interpreter is embedded) you need
to obtain a thread context, and then acquire/release the GIL around
any calls into the interpreter.  I think that's probably what you
mistakenly interpreted as the Python API being single-threaded - it's
that the core Python code is not thread safe, so Python uses a global
lock to ensure only one thread can be executing in that native code
simultaneously.  But that doesn't mean it isn't multi-threading, nor
that multiple threads might not be executing the core code.

Handling the GIL will let your own independent native threads
interoperate within the same interpreter with any other Python-created
threads, or just with a single threaded Python script.

For example, here's a snippet of some C code that initializes an
embedded interpreter as part of a newly created thread (begun via the
_beginthread call from Visual C):

-----------------------------------------------------------------------
Py_Initialize();
PyEval_InitThreads();
ecnaPy_RcvThreadState = PyThreadState_New(PyThreadState_Get()->interp);

PyImport_AddModule("ecna");
modobj = Py_InitModule("ecna",ecnaPy_Methods);

(...)

PyRun_SimpleFile(OptScriptFile,OptScriptName);
-----------------------------------------------------------------------

This sets up the interpreter, but also creates a new thread state to
be used by another thread in the application (a background data
receiving thread).  It then declares the "ecna" module to the embedded
interpreter to use for callbacks into the master C application, and
finally starts the embedded interpreter running a script.  While the
embedded interpreter is in control of the script, it handles the GIL,
releasing it periodically, or whenever explicitly instructed such as
by other extensions.

In the separate data reception thread, at the point that it has enough
data to send over to the Python interpreter, it does something like:

-----------------------------------------------------------------------
PyEval_AcquireThread(ecnaPy_RcvThreadState);

(... Call into script ...)

PyEval_ReleaseThread(ecnaPy_RcvThreadState);
-----------------------------------------------------------------------

There are macros for managing the GIL, but I found them more
logical/readable at the time I wrote the above module for use with
extensions since they work for releasing the GIL around an operation
rather than acquiring it around one.

> Anyway, there is a solution to your problem. Instead of creating your thread
> in C, you can create a Python thread BEFORE calling your C function. This
> works fine because Python threads are safe.

They're not really any more "safe" then natively created threads -
what you're probably seeing is that by creating the thread in Python
the GIL is being handled by Python thus relieving your extension of
worrying about it.

However, if you aren't relinquishing the GIL within your extension,
then you aren't being "nice" either - since Python won't be able to
run other threads as long as your extension is in control.

For an extension, Python provides some straight-forward macros that
you should bracket around any lengthy or I/O related call - this
permits other Python threads to keep executing in the interpreter
while your extension code is waiting for the operation to complete.
For example:

--------------------------------------------------
Py_BEGIN_ALLOW_THREADS
(... perform operation ...)
Py_END_ALLOW_THREADS
--------------------------------------------------

--
-- David
-- 
/-----------------------------------------------------------------------\
 \               David Bolen            \   E-mail: db3l at fitlinxx.com  /
  |             FitLinxx, Inc.            \  Phone: (203) 708-5192    |
 /  860 Canal Street, Stamford, CT  06902   \  Fax: (203) 316-5150     \
\-----------------------------------------------------------------------/