[Python-Dev] Experimenting with STM on CPython

[Armin Rigo]

Hi all,

This is an update on the (so far PyPy-only) project of adding "Automatic
Mutual Exclusion" to Python, via STM (Software Transactional Memory).
For the motivation, see here:

http://morepypy.blogspot.com/2012/03/call-for-donations-for-software.html
"""The point is that [with STM/AME] your program is always correct,
and can be tweaked to improve performance. This is the opposite from
what explicit threads and locks give you, which is a performant
program which you need to tweak to remove bugs. Arguably, this
approach is the reason for why you use Python in the first place
:-)"""

The update is: I now believe that it might be (reasonably) possible to
apply the same techniques to CPython, and not only to PyPy.  For now I
am experimenting with applying them in a simple CPython-like
interpreter.  If it works, it might end up as a patch to the core parts
of CPython.  The interesting property is that it would still be able to
run unmodified C extension modules --- the Python code gets the benefits
of multi-core STM/AME only if it involves only the patched parts of the
C code, but in all cases it still works correctly, falling back to
single-core usage.

I did not try to hack CPython so far, but only this custom interpreter
for a Lisp language, whose implementation should be immediately familiar
to anyone who knows CPython C code: https://bitbucket.org/arigo/duhton .
The non-standard built-in function is "transaction", which schedules a
transaction to run later (see test/test_transaction.py).

The code contains the necessary tweaks to reference counting, and seems
to work on all examples, but leaks some of the objects so far.  Fixing
this directly might be possible, but I'm not sure yet (it might require
interaction with the cycle-detecting GC of CPython).  Moreover the
performance hit is well below 2x, more like 20%.

If anyone is interested, I could create a dedicated mailing list in
order to discuss this in more details.  From experience I would think
that this has the potential to become a Psyco-like experiment, but
unlike 10 years ago, today I'm not ready any more to dive completely
alone into a project of that scale :-)


A bientôt,

Armin.