[Python-Dev] Pythonic concurrency

[Jim Fulton]

Shane Hathaway wrote:
> Antoine Pitrou wrote:
> 
>>A relational database has a very strict and regular data model. Also, it
>>has transactions. This makes it easy to precisely define concurrency at
>>the engine level.
>>
>>To apply the same thing to Python you would at least need :
>>  1. a way to define a subset of the current bag of reachable objects
>>which has to stay consistent w.r.t. transactions that are applied to it
>>(of course, you would have several such subsets in any non-trivial
>>application)
>>  2. a way to start and end a transaction on a bag of objects (begin /
>>commit / rollback)
>>  3. a precise definition of the semantics of "consistency" here : for
>>example, only one thread could modify a bag of objects at any given
>>time, and other threads would continue to see the frozen, stable version
>>of that bag until the next version is committed by the writing thread
>>
>>For 1), a helpful paradigm would be to define an object as being the
>>"root" of a bag, and all its properties would automatically and
>>recursively (or not ?) belong to this bag. One has to be careful that no
>>property "leaks" and makes the bag become the set of all reachable
>>Python objects (one could provide a means to say that a specific
>>property must not be transitively put in the bag). Then, use
>>my_object.begin_transaction() and my_object.commit_transaction().
>>
>>The implementation of 3) does not look very obvious ;-S
> 
> 
> Well, I think you just described ZODB. ;-)  I'd be happy to explain how 
> ZODB solves those problems, if you're interested.
> 
> However, ZODB doesn't provide locking, and that bothers me somewhat.  If 
> two threads try to modify an object at the same time, one of the threads 
> will be forced to abort, unless a method has been defined for resolving 
> the conflict.  If there are too many writers, ZODB crawls.  ZODB's 
> strategy works fine when there aren't many conflicting, concurrent 
> changes, but the complex locking done by relational databases seems to 
> be required for handling a lot of concurrent writers.

I don't think it would be all that hard to use a locking (rather than
a time-stamp) strategy for ZODB, although ZEO would make this
extra challenging.

In any case, the important thing to agree on here is that transactions
provide a useful approach to concurrency control in the case where

- separate control flows are independent, and

- we need to mediate access to shared resources.

Someone else pointed out essentially the same thing at the beginning
of this thread.

Jim

-- 
Jim Fulton           mailto:jim at zope.com       Python Powered!
CTO                  (540) 361-1714            http://www.python.org
Zope Corporation     http://www.zope.com       http://www.zope.org