[IPython-dev] IPython parallel "education"

[Jose Gomez-Dans]

Hello,

I'm trying to figure out the use of the IPython parallel scheme for some of
the processing we are doing. In effect, what we have is a bunch of
calculations on a vector, and we have tons of these vectors, stored in an
(N,11) matrix, where N is typically >10E6. The calculations involve an
euclidean distance and a few vector/matix, matrix/vector products (these
are typically reasonable matrices, say 200x200 or so). First idea is to
approach the problem by splitting the starting (N,11) matrix into (N/s, 11)
matrices (s being the number of nodes) and evaluate them in parallel.

from IPython.parallel import *
import numpy as np

client = Client(profile="default")
ids    = client.ids
nodes  = len(ids)
view   = client.load_balanced_view()
dview  = client[:]

with dview.sync_imports():
    import numpy as np


def the_func ( x ):
    """Does some numerical stuff with a vector `x`"""


@dview.parallel(block=True)
def run_func ( inputx ):
    (x,f) = inputx
    return f(x)


X=np.random.randn(8000000,2)
# parallel run
Y = run_func.map ( [(X[(2000000*i):(i+1)*2000000], the_func) for i in
xrange(nodes)] )

# serial run
YY = the_func(X)



The above code works OK. htop tells me it's using all cores on my laptop.
If I just run the_func(X), it appears to use only one core (htop dixit),
and it takes a little bit longer (~10-15%), which maybe is a lot to do with
parallel linear algebra inside of the local BLAS version or internal
optimisations on scipy's euclidean distance calculations.

We plan to use this on an IPython cluster connected with SSH, to which I
don't yet have access. Are there some obvious guidelines about optimising
for this new setup that I need to know? And yes, hoping to get a bigger
speed up as we increase the size of our problem and nodes!

Thanks!
Jose
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20141218/f6c6d29b/attachment.html>