[Python-Dev] Pythonic concurrency

[Antoine Pitrou]

Hi,

(my 2 cents, probably not very constructive)

> Recently, I've been simulating high concurrency on a PostgreSQL 
> database, and I've discovered that the way you reason about row and 
> table locks is very similar to the way you reason about locking among 
> threads.  The big difference is the consequence of incorrect locking: in 
> PostgreSQL, using the serializable mode, incorrect locking generally 
> only leads to aborted transactions; while in Python and most programming 
> languages, incorrect locking instantly causes corruption and chaos. 
> That's what hurts developers.  I want a concurrency model in Python that 
> acknowledges the need for locking while punishing incorrect locking with 
> an exception rather than corruption.  *That* would be cool, IMHO.

A relational database has a very strict and regular data model. Also, it
has transactions. This makes it easy to precisely define concurrency at
the engine level.

To apply the same thing to Python you would at least need :
  1. a way to define a subset of the current bag of reachable objects
which has to stay consistent w.r.t. transactions that are applied to it
(of course, you would have several such subsets in any non-trivial
application)
  2. a way to start and end a transaction on a bag of objects (begin /
commit / rollback)
  3. a precise definition of the semantics of "consistency" here : for
example, only one thread could modify a bag of objects at any given
time, and other threads would continue to see the frozen, stable version
of that bag until the next version is committed by the writing thread

For 1), a helpful paradigm would be to define an object as being the
"root" of a bag, and all its properties would automatically and
recursively (or not ?) belong to this bag. One has to be careful that no
property "leaks" and makes the bag become the set of all reachable
Python objects (one could provide a means to say that a specific
property must not be transitively put in the bag). Then, use
my_object.begin_transaction() and my_object.commit_transaction().

The implementation of 3) does not look very obvious ;-S
 
Regards

Antoine.