Deadlock in cpyext callback
We have an extension module which works fine with CPython but deadlocks every time in PyPy. The problem is induced when the C++ code in the module (having been invoked from a Python thread) prepares to call back into Python again by calling PyGILState_Ensure. It seems at this point we try to acquire the GIL mutex while already (presumably) holding it - any other threads at this point are also waiting on the mutex so it isn't held elsewhere. This seems to occur from a call to rffi.aroundstate.after() - I assume that the invariant here is that the before() has already executed but that seems not to be the case here. What is supposed to trigger that? Thanks for any assistance in advance. Nick
Hi Nick, On 22 May 2014 11:23, Nick Williams <nhw@me.com> wrote:
The problem is induced when the C++ code in the module (having been invoked from a Python thread) prepares to call back into Python again by calling PyGILState_Ensure.
The documentation of PyGILState_Ensure() says, indeed, that it's ok to call this function even if the current thread already owns the GIL. If PyPy's version deadlocks in this case, then it's a bug of PyPy. Please file it as a bug (https://bitbucket.org/pypy/pypy/issues; also look a bit in case a similar bug would be already reported and waiting there). You can also try to look in the implementation of PyGILState_Ensure() to make sure. (Sorry for the delay in answering, this is part of "cpyext", a part of PyPy that is missing a more dedicated person that would improve it. We still try to fix bugs if they come with a detailed step-by-step guide on how to reproduce them or a test case.) A bientôt, Armin.
Armin,
The documentation of PyGILState_Ensure() says, indeed, that it's ok to call this function even if the current thread already owns the GIL. If PyPy's version deadlocks in this case, then it's a bug of PyPy. Please file it as a bug (https://bitbucket.org/pypy/pypy/issues; also look a bit in case a similar bug would be already reported and waiting there). You can also try to look in the implementation of PyGILState_Ensure() to make sure.
Thanks for the response. We found that by adding a 'with nogil' to our Cython we were able to resolve the problem. However, it's a bit error prone. The callback is for logging and the implication is that we have to release the GIL at any point where we call into C++ and log may be output. Since the same callback may also be called from a non Python thread we do need to ensure the GIL from there, so we can't just drop it.
From my point of view, it does seem like a PyPy bug; will check the tracker and open a bug if there's not already one.
Thanks again Nick
participants (2)
-
Armin Rigo -
Nick Williams