Parallelization with Python: which, where, how?

[Fredrik Lundh]

"Mathias" <no_sp at m_please.cc> wrote:

> I have a (pretty much) "emberassingly parallel" problem and look for the right toolbox to 
> parallelize it over a cluster of homogenous linux workstations. I don't need automatic 
> loop-parallelization or the like since I prefer to prepare the work packets "by hand".
> I simply need
> - to specify a list of clients
> - a means of sending a work packet to a free client and receiving the
>    result (hopefully automatically without need to login to each one)
> - optionally a timeout mechanism if a client doesn't respond
> - optionally help for debugging of remote clients
>
> So far I've seen scipy's COW (cluster of workstation) package, but couldn't find documentation or 
> even examples for it (and the small example in the code crashes...).
> I've noticed PYRO as well, but didn't look too far yet.
>
> Can someone recommend a parallelization approach? Are there examples or documentation? Has someone 
> got experience with stability and efficiency?

googling for "parallel python" brings up lots of references; tools like

    http://pympi.sourceforge.net/
    http://datamining.anu.edu.au/~ole/pypar/

    (see https://geodoc.uchicago.edu/climatewiki/DiscussPythonMPI for
    a comparision)

seem to be commonly used.

</F>