[Python-Dev] Extension modules, Threading, and the GIL

[Scott Gilbert]

--- Mark Hammond <mhammond@skippinet.com.au> wrote:
> 
> [...] I believe it is clear and would prefer to see a single
> other person with a problem [...]
> 

I've been reading this thread with interest since we've recently fought
(and lost) this battle at my company.  Here is our use case:

We use Python in primarily two ways, the first and obvious use is as a
scripting language with a small group of us creating extensions to talk to
existing libraries.  There are no relevant problems here.

The second use is as a data structures library used from C++.  We created a
very easy to use C++ class that has a bazillion operator overloads and
handles all the reference counting and what not for the user.  It used to
handle threading too, but that proved to be very difficult.  Think of this
C++ class as something similar to what boost::python::{object, dict, list,
tuple, long, numeric} provides, but intended for users who don't really
like or want to know C++.

Most of our users write small C++ processes that communicate amongst
themselves via an assortment of IPC mechanisms.  Occasionally these C++
processes are threaded, and we wanted to handle that.  Our model was that
C++ code would never hold the GIL, and that before we entered the Python
API we would use pthread_getspecific (thread local storage) to see if there
was a valid PyThreadState to use.  If there wasn't a thread state, we would
create one.  Since C++ code never held the GIL, we'd always acquire it. 
This strategy allows all Python threads to take turns running, and allows
any C++ threads to enter into Python when needed.  Performance lagged a
little this way, but not so much that we cared.

The problem came when our users started to write generic libraries to be
used from C++ and also wanted these libraries as Python extensions.  In one
case, their library would be used up in a standalone C++ process (where the
GIL was not held), and in another they would use boost to try and export
their library as an extension to Python (where the GIL was held).  The same
C++ library couldn't know in advance if the GIL was held.

The way boost templatizes on your functions and classes, it is not at all
clear when you can safely release the GIL for the benefit of the C++
library being wrapped up that expects the GIL is not held.

Since being able to support writing generic libraries easily is more
important to us than supporting multithreaded C++ processes (using Python
as a data structure library), we changed our strategy and made it so that
in C++ the GIL was held by default.  Since for these types of processes
"most" of our time is spent in C++, no Python threads ever get a chance to
run without additional work from the C++ author.  It also requires
additional work to have multiple C++ threads use Python.  This was pretty
unsatisfying to those of us who like to work with threads.

It's too late to make this long story short, but what would have made our
situation much easier would be something like:

   void *what_happened = Py_AcquireTheGilIfIDontAlreadyHaveIt();

   // Can safely call Python API functions here, no matter what the
   // context is...

   Py_ReleaseTheGilIfImSupposedTo(what_happened);


I hope seeing another side of this is of some use.

Cheers,
    -Scott








__________________________________________________
Do you Yahoo!?
Yahoo! Mail Plus - Powerful. Affordable. Sign up now.
http://mailplus.yahoo.com