[Python-Dev] Extension modules, Threading, and the GIL
Sun, 29 Dec 2002 20:29:19 -0500
Recently an issue has come up on the C++-sig which I think merits a
little attention here. To boil it down, the situation looks like
Shared library Q uses threading but not Python. It supplies a an
interface by which users can supply callback functions. Some of these
callbacks will be invoked directly in response to external calls into
Q; others will be invoked on threads started by calls into Q.
Python extension module A calls shared library Q, but doesn't use its
callback interface. It works fine by itself.
Python extension module B calls shared library Q and uses Q's callback
interface. Because some of the callbacks need to use the Python API,
and *might* be invoked by threads, they must all acquire the GIL.
Because they also might be invoked by direct calls into Q, B must
always release the GIL before calling anything in Q.
Problem: using B while A is loaded breaks A: because B has installed
callbacks in Q that acquire the GIL, A must also release the GIL
before calling into Q.
Notice that the author of A may have had no reason to believe anyone
would install Python callbacks in Q!
It seems to me that for situations like these, where a function may or
may not be called on Python's main thread, it would be helpful if
Python supplied a "recursive mutex" GIL acquisition/release pair, for
which acquisition and release on the main thread would simply bump a
counter. Is this something that was considered and rejected?
firstname.lastname@example.org * http://www.boost-consulting.com
Boost support, enhancements, training, and commercial distribution