[Python-Dev] Pythonic concurrency

[Phillip J. Eby]

At 10:48 AM 9/29/2005 -0600, Bruce Eckel wrote:
>I haven't spent the weekends on the paper yet (but it looks like that
>is what it would take), but I had the impression that they were
>talking about the lock-free techniques such as the ones used in Java
>5. Basically, you start a write operation "in the background" without
>locking the data structure, so reads can continue while the
>calculation is taking place but before the result is committed. When
>the result is ready, an atomic "test and write" operation is used to
>determine whether any other task has modified the value in the
>meantime, and if not to commit the new value. If another task did
>modify the value, then the calculation begins anew.
>
>That was my take, but I haven't studied everything about STM yet, so
>I'm probably missing something.

No, that's certainly the general idea.  The issue for an imperative 
language like Python is that side-effects are the norm, rather than an 
exception.  The Haskell implementation of the idea effectively relies on 
the fact that in Haskell you have to jump through monadic hoops to get side 
effects, so if you have a low-level transactional memory facility, you can 
create monadic combinators that restrict those side effects to occuring in 
the way you want them to.  This is so utterly alien to Python's execution 
model - or indeed most imperative languages' execution models - that I 
don't see how it can be modelled in Python in a way that retains the 
significant benefits of the approach.

Now, in PyPy, I suppose it might be possible to create a custom 
object-space that works that way, and I hadn't thought of that 
before.  Maybe you should run that paper past one of the PyPy wizards and 
see if you can get them interested in the idea, then stand back and see 
what happens.  :)


>The one thing about this paper is that it seems to be an orthogonal
>perspective to anything about concurrency that *I* have seen before.

Yes - it's dramatically different, and seems like the One True Way to do 
imperative concurrency with side-effects.  Certainly transactions work 
nicely for databases, anyway.  :)


>So with concurrency, I would like to know when I do something wrong,
>but if I am told at runtime that's OK with me as long as I'm told.

Yeah, but in current Python implementations - with the possible exception 
of PyPY - detecting the problem at all is virtually impossible.  PyPy is an 
extensible VM, though, so in principle you could create some kind of 
transactional object space that could keep track of things and tell you if 
you mess up.