Distributing simple tasks

[Terry Reedy]

Noam Aigerman wrote:
> Hi,
> 
> Suppose I have an array of functions which I execute in threads (each 
> thread get a slice of the array, iterates over it and executes each 
> function in it’s slice one after the other). Now I want to distribute 
> these tasks between two machines, i.e give each machine half of the 
> slices and let it run them in threads as described above. Is there an 
> easy way, or an article on this matter you can point me to?

Here is a standard  approach that is slightly more general than you 
specified.  It will work with more than two cores or machines.

Write master/server program that reads data, listens to socket for calls 
from worker/client programs, splits work into multiple tasks, sends them 
out to workers, and collects and re-assembles the results.  After a 
timeout, it should be able to reassign tasks not returned.

Write worker/client program than calls master, gets task, completes it, 
and returns result.

Assume all data is on machine A.
Start master and worker on machine A.
Start worker(s) on other cores or machines.

You can either use socket module directly or use networking modules 
built on top of it.  Twisted and pyro are two that come to mind 
immediately.  I have seen code examples but do not remember where.

Note: I do not see any advantage to having multiple compute-bound 
threads, as you seem to describe.

Terry Jan Reedy