Parallelization with Python: which, where, how?

Mathias no_sp at m_please.cc
Mon Dec 20 14:03:09 CET 2004


Dear NG,

I have a (pretty much) "emberassingly parallel" problem and look for the 
right toolbox to parallelize it over a cluster of homogenous linux 
workstations. I don't need automatic loop-parallelization or the like 
since I prefer to prepare the work packets "by hand".
I simply need
- to specify a list of clients
- a means of sending a work packet to a free client and receiving the
    result (hopefully automatically without need to login to each one)
- optionally a timeout mechanism if a client doesn't respond
- optionally help for debugging of remote clients

So far I've seen scipy's COW (cluster of workstation) package, but 
couldn't find documentation or even examples for it (and the small 
example in the code crashes...).
I've noticed PYRO as well, but didn't look too far yet.

Can someone recommend a parallelization approach? Are there examples or 
documentation? Has someone got experience with stability and efficiency?

Thanks a lot,
    Mathias




More information about the Python-list mailing list