[Python-Dev] Software Transactional Memory for Python

[Armin Rigo]

Hi Charles-François,

2011/8/27 Charles-François Natali <neologix at free.fr>:
> The problem is that many locks are actually acquired implicitely.
> For example, `print` to a buffered stream will acquire the fileobject's mutex.

Indeed.  After looking more at the kind of locks used throughout the
stdlib, I notice that in many cases a lock is acquired by code in the
following simple pattern:

    Py_BEGIN_ALLOW_THREADS
    PyThread_acquire_lock(self->lock, 1);
    Py_END_ALLOW_THREADS

If one thread is waiting in the END_ALLOW_THREADS for another one to
release the GIL, but the other one is in a "with atomic" block and
tries to acquire the same lock, deadlock.  But the issue can be
resolved: the first thread in the above example needs to notice that
the other thread is in a "with atomic" block, and "be nice" and
release the lock again.  Then it waits until the "with atomic" block
finishes, and tries again from the start.

We could do this by putting the above pattern it own function (which
makes some sense anyway, because the pattern is repeated left and
right, and is often complicated by an additional "if
(!PyThread_acquire_lock(self->lock, 0))" before); and then allowing
that function to be overridden by the external 'stm' module.

I suspect that I need to do a more thorough review of the stdlib to
make sure (at least more than now) that all potential deadlocking
places can be avoided with a similar refactoring.  All in all, it
seems that the patch to CPython itself will need to be more than just
the few lines in ceval.c --- but still very reasonable both in size and
in content.


A bientôt,

Armin.