[Python-Dev] Extension modules, Threading, and the GIL

[David Abrahams]

Recently an issue has come up on the C++-sig which I think merits a
little attention here.  To boil it down, the situation looks like
this:

Shared library Q uses threading but not Python.  It supplies a an
interface by which users can supply callback functions.  Some of these
callbacks will be invoked directly in response to external calls into
Q; others will be invoked on threads started by calls into Q.

Python extension module A calls shared library Q, but doesn't use its
callback interface.  It works fine by itself.

Python extension module B calls shared library Q and uses Q's callback
interface.  Because some of the callbacks need to use the Python API,
and *might* be invoked by threads, they must all acquire the GIL.
Because they also might be invoked by direct calls into Q, B must
always release the GIL before calling anything in Q.

Problem: using B while A is loaded breaks A: because B has installed
callbacks in Q that acquire the GIL, A must also release the GIL
before calling into Q.

Notice that the author of A may have had no reason to believe anyone
would install Python callbacks in Q!

It seems to me that for situations like these, where a function may or
may not be called on Python's main thread, it would be helpful if
Python supplied a "recursive mutex" GIL acquisition/release pair, for
which acquisition and release on the main thread would simply bump a
counter.  Is this something that was considered and rejected?

TIA,
Dave

-- 
                       David Abrahams
   dave@boost-consulting.com * http://www.boost-consulting.com
Boost support, enhancements, training, and commercial distribution