Armin, thanks for weighing in on this. I'm keen to see a CPython making use of STM, maybe I'll give it a try over Christmas break. I'm willing to take the single threaded performance hit, as I have several applications that degrade due to significant contention with the GIL. The other benefits of STM you describe make it a lot more appealing. I actually tried out Haskell recently to make use of many of the advanced features but came crawling back. If anyone else is keen to try this, I'm happy to receive patches for testing and review. On Thu, Dec 1, 2011 at 10:01 PM, Armin Rigo <arigo@tunes.org> wrote:
Hi,
On Thu, Dec 1, 2011 at 07:06, Matt Joiner <anacrolix@gmail.com> wrote:
I saw this, I believe it just exposes an STM primitive to user code. It doesn't make use of STM for Python internals.
That's correct.
Explicit STM doesn't seem particularly useful for a language that doesn't expose raw memory in its normal usage.
In my opinion, that sentence could not be more wrong.
It is true that, as I discuss on the blog post cited a few times in this thread, the first goal I see is to use STM to replace the GIL as an internal way of keeping the state of the interpreter consistent. This could quite possibly be achieved using the new GCC __transaction_atomic keyword, although I see already annoying issues (e.g. the keyword can only protect a _syntactically nested_ piece of code as a transaction).
However there is another aspect: user-exposed STM, which I didn't explore much. While it is potentially even more important, it is a language design question, so I'm happy to delegate it to python-dev. In my opinion, explicit STM (like Clojure) is not only *a* way to write multithreaded Python programs, but it seems to be *the only* way that really makes sense in general, for more than small examples and more than examples where other hacks are enough (see http://en.wikipedia.org/wiki/Software_transactional_memory#Composable_operat... ). In other words, locks are low-level and should not be used in a high-level language, like direct memory accesses, just because it forces the programmer to think about increasingly complicated situations.
And of course there is the background idea that TM might be available in hardware someday. My own guess is that it will occur, and I bet that in 5 to 10 years all new Intel and AMD CPUs will have Hybrid TM. On such hardware, the performance penalty mostly disappears (which is also, I guess, the reasoning behind GCC 4.7, offering a future path to use Hybrid TM).
If python-dev people are interested in exploring the language design space in that direction, I would be most happy to look in more detail at GCC 4.7. If we manage to make use of it, then we could get a version of CPython using STM internally with a very minimal patch. If it seems useful we can then turn that patch into #ifdefs into the normal CPython. It would of course be off by default because of the performance hit; still, it would give an optional alternate "CPythonSTM" to play with in order to come up with good user-level abstractions. (This is what I'm already trying to do with PyPy without using GCC 4.7, and it's progressing nicely.) (My existing patch to CPython emulating user-level STM with the GIL is not really satisfying, also for the reason that it cannot emulate some other potentially useful user constructs, like abort_and_retry().)
A bientôt,
Armin.
-- ಠ_ಠ