[Python-ideas] Python and Concurrency
talin at acm.org
Fri Mar 23 08:12:39 CET 2007
Josiah Carlson wrote:
> These examples are all what are generally referred to as "embarassingly
> parallel" in literature. One serious issue with programming parallel
> algorithms generally is that not all algorithms are necessarily
> parallelizable. Some are, certainly, but not all. The task is to
> discover those alternate algorithms that *are* parallelizable in such a
> way to offer gains that are "worth it".
A couple of ideas I wanted to explore:
-- A base class or perhaps metaclass that makes python objects
transactional. This wraps all attribute access so what you see is your
transaction's view of the current state of the object.
obj1.attribute = 1
# Other threads can't 'see' the new value until the
# transaction commits.
value = obj1.attribute
'atomic()' starts a new transaction in your thread, which is stored in
thread-local data; Any objects that you mutate become part of the
transaction automatically (will have to be careful about built-in
mutable objects such as lists). At the end, the transaction either
commits, or if there was a conflict, it rolls back and you get to do it
This would be useful for large networks of objects, where you want to
make large numbers of local changes, where each local change affects an
object and perhaps its surrounding objects. What I am describing is very
similar to many complex 3D game worlds.
ZODB already has a transactional object mechanism, although it's
oriented towards database-style transactions and object persistence.
-- A way to partition the space of python objects, such that objects in
each partition cannot have references outside of the partition without
going through some sort of synchronization mechanism, perhaps via some
proxy. The idea is to be able to guarantee that shared state is only
accessible in certain ways that you can reason about.
More information about the Python-ideas