[IPython-dev] IPython parallel "education"

Erik Hvatum ice.rikh at gmail.com
Mon Dec 22 17:15:09 EST 2014

On Mon, Dec 22, 2014 at 2:50 PM, Moritz Beber <moritz.beber at gmail.com>

> Hi Jose,
> Just wanted to share my experience with parallel:
> On Mon, Dec 22, 2014 at 6:19 PM, Jose Gomez-Dans <jgomezdans at gmail.com>
> wrote:
>> Hi Aron,
>> On 18 December 2014 at 20:22, Aron Ahmadia <aron at ahmadia.net> wrote:
>>> What happens if instead of partitioning the data, you create a list of
>>> work units and map those?
>>> Something like:
>>> def apply_the_func(i):
>>>       return the_func(X[N*i):X[(i+1)*N])
>>> Y = run_func.map ( [xrange(i), apply_the_func) for i in range(nodes)] )
>> This provides a substantial speed-up. I also tested other approaches
>> (scatter&gather), but all in all, "pushing" X to the engines seems & using
>> your suggestion seems to work. A question I have is what is going on behind
>> the scenes when I push X around: do all the engines get a copy of the full
>> X? In my case, X can be quite large, and it seems expensive to send lots
>> and lots of data to engines that will only operate on a small fraction of
>> the data...
> I've been working with up to 2 GB of data and using the push mechanism is
> not really feasible at that size. Also, the transmission time increases
> linearly (super linearly?) with more target engines. So I've tried a few
> solutions:
> 1.) If you're working on the same host and don't expect to expand that
> switch to multiprocessing. It's very fast in transmitting data.
> 2.) Store your data on the file system and have each engine access that.
> Either you have a shared file system for the remote kernels to access or
> you'll need to copy the data beforehand/use paramiko.
> 3.) Having a database server is quite a bit of work to invest at the
> beginning (especially if you don't know how) but really lends itself to
> this sort of task. A database server usually has a connection pool so that
> it can automatically handle many workers accessing it concurrently.
>> Thanks for your help
>> Jose
> Just my thoughts/experience. Best of luck with your project,
> Moritz

It is very informative to see that other folks are interested in getting
Python-multiprocessing-style shared memory in other contexts.  I think it
would be a massive benefit for IPython clustering to opportunistically
share data backing Numpy arrays between processes, defaulting to either
mutex locking or copy-on-modify, with the ability to specify the desired

This is actually quite doable.

As Moritz noted, Python multiprocessing transmission is extremely fast.
That's because it simply stashes shared array data in anonymous,
non-file-backed mmap regions shared between processes, rather than
transmitting it.  The thing about the anonymous, non-file-backed mmap
regions used by the Python multiprocessing module is that they are
anonymous - they are identified strictly by file descriptor.  This is no
problem in the case where the other process was forked: it inherits the
file descriptor.  Informing an unrelated process of such a region requires
piping the file descriptor through a Unix domain socket (the Python
multiprocessing module does, in fact, do this for you when you request
spawn rather than fork).

The main advantage of keeping the regions anonymous: the OS maintains a
reference count and automatically destroys and deallocates the region when
the final process with that handle open closes it or otherwise stops
existing (the region is destroyed even if that final process segfaults or
is kill -9ed).  A mmaped named region using shm must be explicitly
destroyed, requiring a watchdog process in order to ensure destruction in
the case of ungraceful exit, even if a reference count is kept in the
shared region.  This duty could perhaps be assumed by the IPython cluster

Zach Pincus and I put together a Numpy shared memory thingy along these
lines, using named, shared, non-file-backed regions with a locked reference
count and automatic deallocation:
https://github.com/erikhvatum/py_interprocess_shared_memory_blob  It's MIT
licensed, so any and all are welcome to use it for any purpose, without

I know how to go about doing the Windows side of the implementation
(currently, only POSIX and Darwin support are complete), but haven't had a
need thus far.  If Windows support is something you want (even just for
reference), drop me a line and I'll finish up the Windows parts.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20141222/99cdbd96/attachment.html>

More information about the IPython-dev mailing list