PyEval_CallObject and threads

[Tim Peters]

[Phlip]
> My colleague came over and asked me to write this:
>
> "We got multiple C++ pthreads calling PyEval_CallObject. How threadsafe
> is the Python API? A C++ module using Boost BPL creates multiple threads,
> which then call back into Python. When two threads simultaneously call
> PyEval_CallObject we go boom."
>
> I will now go examine his serialization primitives. But if
> anyone has any better ideas...

[Tim]
> With very few exceptions, all calls to Python C API functions
> must be made while holding the global interpreter lock.  The exceptions
> have mostly to do with a handful of functions related to initialization
> and shutdown.  See the section "Thread State and the Global Interpreter
> Lock" in the Python/C API Reference Manual for details.

[Philip]
> Further inspection reveals our architect(s) think we need to call Python
> script asynchronously from multiple threads.

You started this talking about pthreads.  I don't know what "C++ pthreads"
might mean (C++ doesn't define any threading facilities last I knew), but do
know about POSIX pthreads.  Whatever, any thread calling a Python C API
function must acquire the global interpreter lock before making that call.

> By 'lock' we assume both you and the section you cite mean 'block'.

I don't know what you mean by "block".  The global interpreter lock is a
specific Python lock, implemented in various ways on various platforms using
the platform thread facilities (under POSIX pthreads it happens to be
implemented via a combination of a pthreads mutex and a pthreads condition
variable, for example).  It must be acquired and released according to the
protocols, and using the macros, explained in the referenced manual.

> Isn't this what Stackless Python is for?

Stackless has nothing to do with "real" threads.  If you're using real
threads, they must acquire the global interpreter lock before doing anything
with the C API.  Are you or are you not using platform OS threads (if you're
using POSIX pthreads, or using Python threads via the thread or threading
modules, the answer is yes)?  If yes, you must do what the manual says to do.

It is flatly illegal, for example, that "two threads simultaneously call
PyEval_CallObject".  The calls MUST be serialized via acquiring the global
interpreter lock.  Else "we go boom" is one of the more pleasant possible
outcomes.

If you're using microthreads within a single OS-level thread, then the latter
is always holding the global interpreter lock so this shouldn't be an issue.
But then your use of the word "pthreads" is a mystery.

you-already-know-the-answers-we're-just-trying-to-extract-them<wink>-ly
    y'rs  - tim