[IPython-dev] Parallel map

[Gael Varoquaux]

On Fri, Mar 07, 2008 at 08:42:26PM +0100, Gael Varoquaux wrote:
> On Fri, Mar 07, 2008 at 08:10:36PM +0100, Gael Varoquaux wrote:
> > I am trying to do a parallel map using ipython1. Is there a really simple
> > way to do this, or a tutorial somewhere telling me how? I can probably
> > figure it out, but I have to dig through a fair amount of
> > tutorial/doc/wiki articles/reading source code to move forward.

> > My requirement is that I want the code to be purely valid self sustained
> > Python code.

> OK, making some progress at this.

> I found out I need to create a MultiEngineClient

> rc = client.MultiEngineClient(('127.0.0.1', 10105))

> and I can use its map method.

I succeeded (I had a good night(s sleep, in between), by piggy backing
the ipcluster script. It is a bit ugly, but I post the code here for
future reference.

What made my task hard was both the fact that there is no obvious way of
creating a cluster from Python, and the fact that ipython1.kernel.api was
suppressed and that all the information I can find on the web uses
ipython1.kerenl.api.RemoteControler.

Now the irony is that I ended up not beeing able to use ipython1 for the
problem I was interested in, as the objects I wanted to send to my
parallel map where not picklable. I wrote a small hack using threading
and os.system to do the work. I suspect this is a limitation people are
going to bump into quite often. Ideas to make a workaround more or less
part of ipython1 natively would be great. In my case, the object I had to
scatter where directly imported from a module, so scattering a module
path as a string (eg 'ipython1.kernel.client.MultiEngineClient') waould 
have been an option. I have no hindsight on these problems, so I don't
pretend suggesting a good solution.

Anyway, thanks for ipython1, keep the good work up, it is a difficult but
import task,

Cheers,

Gaël

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
"""
Provides a simple parallel map.
"""

# Piggy-back the ipcluster script to start the engines.
from ipython1.kernel.scripts import ipcluster as cluster
from ipython1.kernel.client import MultiEngineClient
from threading import Thread
from time import sleep
import sys

##############################################################################
def guess_ncpu():
    """ Parses /proc/cpuinfo to guess the number of CPU on the box.
        This has been tested only under Linux.
    """
    ncpu = 0
    cpuinfo = file('/proc/cpuinfo')
    for line in cpuinfo.readlines():
        if line[:10]  == 'processor\t':
            ncpu += 1
    return ncpu

##############################################################################
# Code to start the engine and create the controller
def start_cluster(ncpu=guess_ncpu()):
    """ Starts a cluster on the local computer and returns a controller
        to the cluster.
    """
    # We use ipcluster.main, but it takes its instructions from sys.argv,
    # thus we overide it
    orig_argv = sys.argv
    sys.argv = ['foo', '-n', str(ncpu)]
    # Starting the cluster is a blocking operation. We thus need a
    # thread to do the work.
    Thread(target=cluster.main).start()
    # There is a sleep(3) in ipcluster
    sleep(4)
    sys.argv = orig_argv 
    return MultiEngineClient(('127.0.0.1',10105))

##############################################################################
# This code is so trivial you should really use directly the controller
# method if you are going to do anything more than running pmap once
# (keep in mind that there is an overhead of creating the cluster).
def pmap(func, seq, ncpu=guess_ncpu()):
    """ Creates a cluster of ipython1 engines and runs a parallel map on
        it.
    """
    mec = start_cluster(ncpu=ncpu)
    outseq = mec.map(func, seq)
    mec.kill(controler=True)
    return outseq
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++