[Python-Dev] Concurrency and Python

[James Y Knight]

On Jan 7, 2005, at 2:44 PM, Edward C. Jones wrote:
> Is Python a suitable language for concurrent programming?

Depends on what you mean. Python is not very good for shared-memory  
concurrent programming. (That is, threads). The library doesn't have  
enough good abstractions for locks/synchronization/etc, and, of course  
the big issue of CPython only allowing one thread to execute bytecode  
at a time.

At the moment, threads are the fad, but I don't believe that will be  
scaling very well. As you scale up the number of CPUs, the amount of  
time wasted on memory synchronization similarly goes up, until you're  
wasting more time on memory consistency than doing actual work.

Thus, I expect the trend to be more towards async message passing  
architectures (that is, multiple processes each with their own memory),  
instead, and I think Python is about as good for that as any existing  
language. Which is to say: reasonable, but not insanely great.

> What is the state of programming theory for concurrency?

For an example of the kind of new language being developed around a  
asynchronous message passing model, see IBM's poorly-named "X10"  
language. I saw a talk on it and thought it sounded very promising.  
What it adds over the usual message passing system is an easier way to  
name and access remote data and to spawn parallel activities that  
operates on that data. The part about arrays of data spread out over a  
number of different "places" (roughly, a CPU and its own memory) and  
how to operate on them I found especially interesting.

I tried to find their project website, but since their name conflicts  
with the home automation system, it's hard to google for. Or perhaps  
they don't have a website.

Short summary information:
http://www.csail.mit.edu/events/eventcalendar/calendar.php? 
show=event&id=131

Talk slides:
http://www.cs.ualberta.ca/~amaral/cascon/CDP04/slides/sarkar.pdf

More talk slides, and a video:
http://www.research.ibm.com/vee04/video.html#sarkar
"Vivek Sarkar, Language and Virtual Machine Challenges for Large-Scale  
Parallel Systems"

James