[Python-Dev] STM and python

[Matt Joiner]

Armin, thanks for weighing in on this. I'm keen to see a CPython
making use of STM, maybe I'll give it a try over Christmas break. I'm
willing to take the single threaded performance hit, as I have several
applications that degrade due to significant contention with the GIL.

The other benefits of STM you describe make it a lot more appealing. I
actually tried out Haskell recently to make use of many of the
advanced features but came crawling back.

If anyone else is keen to try this, I'm happy to receive patches for
testing and review.

On Thu, Dec 1, 2011 at 10:01 PM, Armin Rigo <arigo at tunes.org> wrote:
> Hi,
>
> On Thu, Dec 1, 2011 at 07:06, Matt Joiner <anacrolix at gmail.com> wrote:
>> I saw this, I believe it just exposes an STM primitive to user code.
>> It doesn't make use of STM for Python internals.
>
> That's correct.
>
>> Explicit STM doesn't seem particularly useful for a language that
>> doesn't expose raw memory in its normal usage.
>
> In my opinion, that sentence could not be more wrong.
>
> It is true that, as I discuss on the blog post cited a few times in
> this thread, the first goal I see is to use STM to replace the GIL as
> an internal way of keeping the state of the interpreter consistent.
> This could quite possibly be achieved using the new GCC
> __transaction_atomic keyword, although I see already annoying issues
> (e.g. the keyword can only protect a _syntactically nested_ piece of
> code as a transaction).
>
> However there is another aspect: user-exposed STM, which I didn't
> explore much.  While it is potentially even more important, it is a
> language design question, so I'm happy to delegate it to python-dev.
> In my opinion, explicit STM (like Clojure) is not only *a* way to
> write multithreaded Python programs, but it seems to be *the only* way
> that really makes sense in general, for more than small examples and
> more than examples where other hacks are enough (see
> http://en.wikipedia.org/wiki/Software_transactional_memory#Composable_operations
> ).  In other words, locks are low-level and should not be used in a
> high-level language, like direct memory accesses, just because it
> forces the programmer to think about increasingly complicated
> situations.
>
> And of course there is the background idea that TM might be available
> in hardware someday.  My own guess is that it will occur, and I bet
> that in 5 to 10 years all new Intel and AMD CPUs will have Hybrid TM.
> On such hardware, the performance penalty mostly disappears (which is
> also, I guess, the reasoning behind GCC 4.7, offering a future path to
> use Hybrid TM).
>
> If python-dev people are interested in exploring the language design
> space in that direction, I would be most happy to look in more detail
> at GCC 4.7.  If we manage to make use of it, then we could get a
> version of CPython using STM internally with a very minimal patch.  If
> it seems useful we can then turn that patch into #ifdefs into the
> normal CPython.  It would of course be off by default because of the
> performance hit; still, it would give an optional alternate
> "CPythonSTM" to play with in order to come up with good user-level
> abstractions.  (This is what I'm already trying to do with PyPy
> without using GCC 4.7, and it's progressing nicely.)  (My existing
> patch to CPython emulating user-level STM with the GIL is not really
> satisfying, also for the reason that it cannot emulate some other
> potentially useful user constructs, like abort_and_retry().)
>
>
> A bientôt,
>
> Armin.



-- 
ಠ_ಠ