[Python-ideas] Python and Concurrency

[Talin]

Aahz wrote:
> On Sun, Mar 25, 2007, Talin wrote:
>> Thinking more about this, it seems to me that discussions of syntax for 
>> doing parallel operations and nifty classes for synchronization are a 
>> bit premature. The real question, it seems to me, is how to get Python 
>> to operate concurrently at all.
> 
> Maybe that's what it seems to you; to others of us who have been looking
> at this problem for a while, the real question is how to get a better
> multi-process control and IPC library in Python, preferably one that is
> cross-platform.  You can investigate that right now, and you don't even
> need to discuss it with other people.

If you mean some sort of inter-process messaging system, there are a 
number that already exist; I'd look at IPython and py-globus for starters.

My feeling is that while such an approach is vastly easier from the 
standpoint of Python developers, and may be easier from the standpoint 
of a typical Python programmer, it doesn't actually solve the problem 
that I'm attempting to address, which is figuring out how to write 
client-side software that dynamically scales to the number of processors 
on the system.

My view is that while the number of algorithms that we have that can be 
efficiently parallelized in a fine-grained threading environment is 
small (compared to the total number of strictly sequential algorithms), 
the number of algorithms that can be adapted to heavy-weight, 
coarse-grained processes is much smaller still.

For example, it is easy to imagine a quicksort routine where different 
threads are responsible for sorting various sub-partitions of the array. 
If this were to be done via processes, the overhead of marshalling and 
unmarshalling the array elements would completely swamp the benefits of 
making it concurrent.

-- Talin