[IPython-dev] IPython parallel "education"

Aron Ahmadia aron at ahmadia.net
Thu Dec 18 15:22:21 EST 2014


On Thu, Dec 18, 2014 at 11:13 AM, Jose Gomez-Dans <jgomezdans at gmail.com>
wrote:
>
> # parallel run
> Y = run_func.map ( [(X[(2000000*i):(i+1)*2000000], the_func) for i in
> xrange(nodes)] )
>

What happens if instead of partitioning the data, you create a list of work
units and map those?
Something like:

def apply_the_func(i):
      return the_func(X[N*i):X[(i+1)*N])

Y = run_func.map ( [xrange(i), apply_the_func) for i in range(nodes)] )

You'd instantiate X ahead of time on your nodes.

I suggest this because if "the_func" is not particularly complex, a good
fraction of your time will be spent just on sending input and results back
and forth.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20141218/2963f0c2/attachment.html>


More information about the IPython-dev mailing list