[Python-ideas] Python and Concurrency

[Talin]

Josiah Carlson wrote:
> These examples are all what are generally referred to as "embarassingly
> parallel" in literature.  One serious issue with programming parallel
> algorithms generally is that not all algorithms are necessarily
> parallelizable. Some are, certainly, but not all.  The task is to
> discover those alternate algorithms that *are* parallelizable in such a
> way to offer gains that are "worth it".

A couple of ideas I wanted to explore:

-- A base class or perhaps metaclass that makes python objects 
transactional. This wraps all attribute access so what you see is your 
transaction's view of the current state of the object.

Something like:

    with atomic():
       obj1.attribute = 1

       # Other threads can't 'see' the new value until the
       # transaction commits.
       value = obj1.attribute

'atomic()' starts a new transaction in your thread, which is stored in 
thread-local data; Any objects that you mutate become part of the 
transaction automatically (will have to be careful about built-in 
mutable objects such as lists). At the end, the transaction either 
commits, or if there was a conflict, it rolls back and you get to do it 
over again.

This would be useful for large networks of objects, where you want to 
make large numbers of local changes, where each local change affects an 
object and perhaps its surrounding objects. What I am describing is very 
similar to many complex 3D game worlds.

ZODB already has a transactional object mechanism, although it's 
oriented towards database-style transactions and object persistence.

-- A way to partition the space of python objects, such that objects in 
each partition cannot have references outside of the partition without 
going through some sort of synchronization mechanism, perhaps via some 
proxy. The idea is to be able to guarantee that shared state is only 
accessible in certain ways that you can reason about.

-- Talin