I've received some great feedback since the initial beta release of the minimalistic STM code I discussed and released 2 weeks ago. I've incorporated the feedback, and created a couple of examples based on the canonical dining philosophers example. (One based on normal python threads, one based on Kamaelia)
It turns out that there was a potential race hazard during "using" and "usevar" which I'd missed - many thanks to Richard Taylor for pointing out this issue.
You can download this release version here: http://thwackety.com/Axon.STM-1.0.1.tar.gz
tar zxf Axon.STM-1.0.1.tar.gz cd Axon.STM-1.0.1/ sudo python setup.py install
Software Transactional Memory (STM) is a technique for allowing multiple threads to share data in such a way that they know when something has gone wrong. It's been used in databases (just called transactions there really) for some time and is also very similar to version control. Indeed, you can think of STM as being like variable level version control.
Note: Because this is NOT intended to be persistent, this is not an ACID store because it doesn't support the D - durability across a crash. (after all, we don't save the state to disk) (The other aspects atomicity, consistency & isolation are supported though)
I've written this to allow a part of Kamaelia to share & manage a dictionary of atomic values between threads simply, and as a result this code is also going into mainline Kamaelia. (Specifically into Axon Kamaelia's core)
However STM is something that should hopefully be of use to others doing concurrent things whether or not they're using kamaelia, hence this stand alone release.
This stand alone release should not be used alongside mainline Axon yet. (Well you can, as long as you reinstall your Axon over the top, but that's icky :-)
[ please skip this (or correct me :) if you understand concurrency already :-) ]
Why do you need it? Well, in normal code, Global variables are generally shunned because it can make your code a pain to work with and a pain to be certain if it works properly. Even with linear code, you can have 2 bits of code manipulating a structure in surprising ways - but the results are repeatable. Not-properly-managed-shared-data is to threaded systems as not-properly-managed-globals are to normal code. (This code is one way of helping manage shared data)
Well, with code where you have multiple threads active, having shared data is like an even nastier version of globals. Why? Well, when you have 2 (or more) running in parallel, the results of breakage can become hard to repeat as two pieces of code "race" to update values.
With STM you make it explicit what the values are you want to update, and only once you're happy with the updates do you publish them back to the shared storage. The neat thing is, if someone else changed things since you last looked, you get told (your commit fails), and you have to redo the work. This may sound like extra work (you have to be prepared to redo the work), but it's nicer than your code breaking :-)
The way you get that message is the .commit raises a ConcurrentUpdate exception.
Also, it's designed to work happily in code that requires non-blocking usage - which means you may also get a "BusyRetry" exception under load. If you do, you should as the exception suggests retry the action that you just tried. (With or without restarting the transaction)
Apologies if that sounds too noddy :)
# Initialising a Store
from Axon.STM import Store
S = Store()
# Single values
greeting = S.usevar("hello") print repr(greeting.value) greeting.set("Hello World") greeting.commit() S.dump()
# Groups of values
D = S.using("account_one", "account_two", "myaccount") D["account_one"].set(50) D["account_two"].set(100) D.commit() S.dump()
D = S.using("account_one", "account_two", "myaccount") D["myaccount"].set(D["account_one"].value+D["account_two"].value) D["account_one"].set(0) D["account_two"].set(0) D.commit() S.dump()
Pure python version: https://kamaelia.svn.sourceforge.net/svnroot/kamaelia/branches/private_MPS_S...
import time import Axon from Axon.STM import Store import random
def all(aList, value): for i in aList: if value != i: return False return True
class Philosopher(Axon.ThreadedComponent.threadedcomponent): forks = ["fork.1", "fork.2"] # default for testing :-) def main(self): # start here :-) while 1: X = self.getforks() time.sleep(0.2) self.releaseforks(X) time.sleep(0.3+random.random())
def getforks(self): gotforks = False while not gotforks: try: X = self.store.using(*self.forks) if all([ X[fork].value for fork in self.forks], None): for fork in self.forks: X[fork].value = self.name X.commit() gotforks = True else: time.sleep(random.random()) except Axon.STM.ConcurrentUpdate: time.sleep(random.random()) print "Got forks!", self.name, self.forks return X def releaseforks(self,X): print "releasing forks", self.name for fork in self.forks: X[fork].value = None X.commit()
S = Store() N = 5 for i in range(1,N): Philosopher(store=S,forks=["fork.%d" % i ,"fork.%d" % (i+1)]).activate()
Philosopher(store=S,forks=["fork.%d" % N ,"fork.%d" % 1]).run()
Feedback is very welcome, preferably via email to the Kamaelia List * email@example.com
Feedback especially regarding bugs and logical errors is particularly welcome. (hopefully there aren't any - but it's always hard to spot your own)
Many thanks to Fuzzyman, Duncan Booth, John J Lee & Sylvain Hellegouarch for feedback whilst I was prototyping this.
Further thanks go to Richard Taylor for detailed feedback and discussion regarding locking and for pointing me at MASCOT which made me think of doing the dining philosophers this way :-)
This will be merged onto the mainline of Kamaelia with some auxillary functions , as another feather aimed at making concurrency easy to work with :-)