Hi Charles-François, 2011/8/27 Charles-François Natali <neologix@free.fr>:
The problem is that many locks are actually acquired implicitely. For example, `print` to a buffered stream will acquire the fileobject's mutex.
Indeed. After looking more at the kind of locks used throughout the stdlib, I notice that in many cases a lock is acquired by code in the following simple pattern: Py_BEGIN_ALLOW_THREADS PyThread_acquire_lock(self->lock, 1); Py_END_ALLOW_THREADS If one thread is waiting in the END_ALLOW_THREADS for another one to release the GIL, but the other one is in a "with atomic" block and tries to acquire the same lock, deadlock. But the issue can be resolved: the first thread in the above example needs to notice that the other thread is in a "with atomic" block, and "be nice" and release the lock again. Then it waits until the "with atomic" block finishes, and tries again from the start. We could do this by putting the above pattern it own function (which makes some sense anyway, because the pattern is repeated left and right, and is often complicated by an additional "if (!PyThread_acquire_lock(self->lock, 0))" before); and then allowing that function to be overridden by the external 'stm' module. I suspect that I need to do a more thorough review of the stdlib to make sure (at least more than now) that all potential deadlocking places can be avoided with a similar refactoring. All in all, it seems that the patch to CPython itself will need to be more than just the few lines in ceval.c --- but still very reasonable both in size and in content. A bientôt, Armin.