Aahz <aahz@pythoncraft.com> writes:
On Sun, Dec 29, 2002, David Abrahams wrote:
Python extension module B calls shared library Q and uses Q's callback interface. Because some of the callbacks need to use the Python API, and *might* be invoked by threads, they must all acquire the GIL. Because they also might be invoked by direct calls into Q, B must always release the GIL before calling anything in Q.
So you're saying that the callback functions in B acquire the GIL?
Yes.
Problem: using B while A is loaded breaks A: because B has installed callbacks in Q that acquire the GIL, A must also release the GIL before calling into Q.
Why? The callbacks in B will simply hang until they acquire the GIL.
If A doesn't release the GIL, one of its direct calls into Q from Python may invoke a callback in B, which tries to acquire the lock when it is already held. This is a no-no. I realize that the docs for PyEval_AcquireLock() say: "If this thread already has the lock, a deadlock ensues", but the behavior we're seeing is consistent with a scenario where trying to acquire an already-held is a no-op and releasing it is unconditional. Eventually the GIL release in B's callback takes effect and when A returns to Python there is no thread state.
I think this thread might be better handled on c.l.py, at least until it's understood well enough to be clear whether something does need to change in Python.
I'm pretty certain I understand what's going on. If it would be better for you to take this to c.l.py, I'm happy to do so, but AFAICT there _is_ a Python core issue here: there's no way to find out whether you've already got the GIL**, so if you _might_ have to acquire it, you must always acquire it. **If I'm wrong about that, please let me know. It isn't obvious from the documentation. -- David Abrahams dave@boost-consulting.com * http://www.boost-consulting.com Boost support, enhancements, training, and commercial distribution