[IPython-dev] IPython parallel and objects

[Michael McKerns]

If you are looking to convert nested for-loops to hierarchical parallel maps,
this is the kind of thing that `pathos` is built to do.  If you are
interested
in an optimizer that can utilize hierarchical parallelism to compute
expectation
values, this is the kind of thing that `mystic` is designed to do.

If you want to get the code, the best place to get them is here:
https://github.com/uqfoundation

Examples:
http://stackoverflow.com/questions/28203774/how-to-do-hierarchical-parallelism-in-ipython-parallel
http://stackoverflow.com/questions/27751023/converting-serial-to-parallel-to-multi-machine

Neither `pathos` nor `mystic` use IPython-parallel, but I hope to make them
more compatible with it in the near future.


> Hi,
>
> On Wed, Jan 28, 2015 at 1:40 PM, thwiouz <maidos93 at laposte.net> wrote:
>
>> Hi guys,
>>
>> I'd like to parallelize the following loop:
>>
>> m1 = np.arange(0, 10, 100)
>> m2 = np.arange(0, 10, 100)
>>
>> for m_ in m1:
>>     for n_2 in m2:
>>         tmp = optimizer.compute_expectation([m_, n_])
>>         res.append(tmp)
>>
>> with type(optimizer) some class that I've defined. How could I handle
>> it?
>>
>> I tried to do the same as in
>>
>> http://stackoverflow.com/questions/9363118/parallelise-nested-for-loop-in-ipython
>> so I created a lambda function f
>>
>> f = lambda x, y: optimizer.compute_expectation([x, y])
>>
>> but it's not working with a load balanced view, saying that optimizer is
>> not
>> defined.
>>
>
> IPython only automatically transmits the function definition used in a
> `map` call or similar. Your optimizer is inside your lambda function and
> thus unknown on the remote kernels. In order to overcome this, take a look
> here:
>
> http://ipython.org/ipython-doc/2/parallel/parallel_multiengine.html#remote-function-decorators
>
> and here:
>
> http://ipython.org/ipython-doc/2/parallel/parallel_task.html#dependencies
>
> this could be helpful, too:
>
> http://ipython.org/ipython-doc/2/parallel/parallel_multiengine.html#moving-python-objects-around
>
>
> HTH,
> Moritz
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-dev
>


---

Mike McKerns
California Institute of Technology
TEL: (626)395-5773 or (626)590-8470
http://www.its.caltech.edu/~mmckerns
mmckerns at caltech.edu