[Python-Dev] Pythonic concurrency

[Shane Hathaway]

Antoine Pitrou wrote:
> A relational database has a very strict and regular data model. Also, it
> has transactions. This makes it easy to precisely define concurrency at
> the engine level.
> 
> To apply the same thing to Python you would at least need :
>   1. a way to define a subset of the current bag of reachable objects
> which has to stay consistent w.r.t. transactions that are applied to it
> (of course, you would have several such subsets in any non-trivial
> application)
>   2. a way to start and end a transaction on a bag of objects (begin /
> commit / rollback)
>   3. a precise definition of the semantics of "consistency" here : for
> example, only one thread could modify a bag of objects at any given
> time, and other threads would continue to see the frozen, stable version
> of that bag until the next version is committed by the writing thread
> 
> For 1), a helpful paradigm would be to define an object as being the
> "root" of a bag, and all its properties would automatically and
> recursively (or not ?) belong to this bag. One has to be careful that no
> property "leaks" and makes the bag become the set of all reachable
> Python objects (one could provide a means to say that a specific
> property must not be transitively put in the bag). Then, use
> my_object.begin_transaction() and my_object.commit_transaction().
> 
> The implementation of 3) does not look very obvious ;-S

Well, I think you just described ZODB. ;-)  I'd be happy to explain how 
ZODB solves those problems, if you're interested.

However, ZODB doesn't provide locking, and that bothers me somewhat.  If 
two threads try to modify an object at the same time, one of the threads 
will be forced to abort, unless a method has been defined for resolving 
the conflict.  If there are too many writers, ZODB crawls.  ZODB's 
strategy works fine when there aren't many conflicting, concurrent 
changes, but the complex locking done by relational databases seems to 
be required for handling a lot of concurrent writers.

Shane