[python-uk] [pyconuk] Minimalistic software transactional memory

[Michael Sparks]

Hi Richard,


On Tuesday 11 December 2007 13:36, Richard Taylor wrote:
> I don't think that you can rely on the threadsafety of these functions.
> Even if they are threadsafe in C Python (which I doubt that 'set' is), the
> locking in Jython in more fine grained and would likely catch you out.

It's perhaps worth noting in the version of the code I posted, in this thread, 
where it said...

   """What key areas appear least threadsafe, and any general suggestions
      around that."""

...I knew that set and using were not threadsafe, but wondered about other 
parts. I perhaps should've been more explicit on that point. (I wanted to 
simply post some ideas which showed the core logic without locking. Perhaps a 
mistake :)

Anyhow, the current version is here:

https://kamaelia.svn.sourceforge.net/svnroot/kamaelia/branches/private_MPS_Scratch/Bindings/STM/Axon/STM.py

In that version, "set" now looks like this:
    def set(self, key, value):
        success = False
        if self.lock.acquire(0):
            try:
                if not (self.store[key].version > value.version):
                    self.store[key] = Value(value.version+1,
                                            copy.deepcopy(value.value),
                                            self, key)
                    value.version= value.version+1
                    success = True
            finally:
                self.lock.release()
        else:
            raise BusyRetry

        if not success:
            raise ConcurrentUpdate

and "using" has changed to "usevar: (using now relates to a collection)

    def usevar(self, key):
        try:
            return self.get(key)
        except KeyError:
            if self.lock.acquire(0):
                try:
                    self.store[key] = Value(0, None,self,key)
                finally:
                    self.lock.release()
            else:
                raise BusyRetry

            return self.get(key)

Since mutations of the store rely on acquiring the lock on the store, that 
should be safe(r). User code doesn't have to worry about locks however - 
which is course the point of the code :-)

The reason for specifically using the acquire(0) call rather than acquire() 
call is because I want it to fail hard if the lock can't be acquired. I know 
it'd be nicer to have a finer grained lock here, but I'm personally primarily 
going to be using this for rare operations rather than common operations.

These locks above are of course in relation to write locking. I'll think about 
the read locking you've suggested.

Your locking looks incorrect on using since it both allows reading and writing 
of the store. (retrieve value & if not present create & initialise)

I also think the independent locks are a misnomer, but they're useful for 
thinking about it.

> I would suggest that you should routinely wrap shared datamodels like these
> in thread locks to be certain about things.

Indeed. It makes the code look worse, so for this example I was really after 
suggestions (like yours :-) of "OK, where does this break badly" as well as 
"does the logic look sane?".

> I would also suggest that a small change to the Value class would make it
> possible for client code to subclass it, which might make it more flexible.

I'm not convinced by the changes to Value - its there for storing arbitrary 
values, rather than extending Value itself. It's probably worth noting 
that .clone has changed in my version to this:

    def clone(self):
        return Value(self.version,
                     copy.deepcopy(self.value),self.store,self.key)

Which includes deepcopy on the value stored by Value. I'm beginning to think 
that Value should be called "Variable" to make this clearer...

> Here is my suggestion, bare in mind that I have not tested the thread
> locking code beyond making sure that it runs :-)

The feedback is much appreciated - it's making me think more about the read
locking aspect. I suspect the GIL in CPython *may* make the reads safe, but
the lack of a GIL in jython & ironpython probably renders the reads in using &
get unsafe. (and I would like this to be safe in jython & ironpython)

It's interesting though, after having developed large amounts of code of code 
based on no-shared-data & read-only/write-only pipes with data handoff and 
not having had any major concurrency issues (despite mixing threads and non 
threads) switching to a shared data model instantly causes problems. 

The difference is really stark. One is simple, natural and easy and the other 
is subtle & problematic. I'm not shocked, but find it amusing :-)

Many thanks for the feedback!


Michael.