<div dir="ltr"><div><div class="gmail_quote">On Mon, Dec 22, 2014 at 2:50 PM, Moritz Beber <span dir="ltr"><<a href="mailto:moritz.beber@gmail.com" target="_blank">moritz.beber@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi Jose,<br><div><div class="gmail_extra"><br></div><div class="gmail_extra">Just wanted to share my experience with parallel:<br></div><div class="gmail_extra"><br><div class="gmail_quote"><div><div class="h5">On Mon, Dec 22, 2014 at 6:19 PM, Jose Gomez-Dans <span dir="ltr"><<a href="mailto:jgomezdans@gmail.com" target="_blank">jgomezdans@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi Aron,<div class="gmail_extra"><br><div class="gmail_quote"><span>On 18 December 2014 at 20:22, Aron Ahmadia <span dir="ltr"><<a href="mailto:aron@ahmadia.net" target="_blank">aron@ahmadia.net</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><span><div class="gmail_extra">What happens if instead of partitioning the data, you create a list of work units and map those?<br></div></span><div class="gmail_extra">Something like:<br><br></div><div class="gmail_extra">def apply_the_func(i):<br></div><div class="gmail_extra"> return the_func(X[N*i):X[(i+1)*N])<br></div><div class="gmail_extra"><br><font face="monospace">Y = run_func.map ( [xrange(i), apply_the_func) for i in range(nodes)] ) </font></div></div></blockquote><div><br></div></span><div>This
provides a substantial speed-up. I also tested other approaches
(scatter&gather), but all in all, "pushing" X to the engines seems
& using your suggestion seems to work. A question I have is what is
going on behind the scenes when I push X around: do all the engines get a
copy of the full X? In my case, X can be quite large, and it seems
expensive to send lots and lots of data to engines that will only
operate on a small fraction of the data...</div></div></div></div></blockquote><div><br><br></div></div></div><div>I've
been working with up to 2 GB of data and using the push mechanism is
not really feasible at that size. Also, the transmission time increases
linearly (super linearly?) with more target engines. So I've tried a few
solutions:<br><br></div><div>1.) If you're working on the same host and
don't expect to expand that switch to multiprocessing. It's very fast
in transmitting data.<br></div><div>2.) Store your data on the file
system and have each engine access that. Either you have a shared file
system for the remote kernels to access or you'll need to copy the data
beforehand/use paramiko.<br></div><div>3.) Having a database server is
quite a bit of work to invest at the beginning (especially if you don't
know how) but really lends itself to this sort of task. A database
server usually has a connection pool so that it can automatically handle
many workers accessing it concurrently.<br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span class=""><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><br></div><div>Thanks for your help</div><span><font color="#888888"><div>Jose </div></font></span></div></div></div>
</span><br></blockquote></div><br></div><div class="gmail_extra">Just my thoughts/experience. Best of luck with your project,<br></div><div class="gmail_extra">Moritz<br></div></div></div>
<br></blockquote></div><br><br>It is very informative to see that other folks are interested in getting Python-multiprocessing-style shared memory in other contexts. I think it would be a massive benefit for IPython clustering to opportunistically share data backing Numpy arrays between processes, defaulting to either mutex locking or copy-on-modify, with the ability to specify the desired behavior.<br><br></div>This is actually quite doable.<br><div><br>As Moritz noted, Python multiprocessing transmission is extremely fast. That's because it simply stashes shared array data in anonymous, non-file-backed mmap regions shared between processes, rather than transmitting it. The thing about the anonymous, non-file-backed mmap regions used by the Python multiprocessing module is that they are anonymous - they are identified strictly by file descriptor. This is no problem in the case where the other process was forked: it inherits the file descriptor. Informing an unrelated process of such a region requires piping the file descriptor through a Unix domain socket (the Python multiprocessing module does, in fact, do this for you when you request spawn rather than fork).<br><br></div><div>The main advantage of keeping the regions anonymous: the OS maintains a reference count and automatically destroys and deallocates the region when the final process with that handle open closes it or otherwise stops existing (the region is destroyed even if that final process segfaults or is kill -9ed). A mmaped named region using shm must be explicitly destroyed, requiring a watchdog process in order to ensure destruction in the case of ungraceful exit, even if a reference count is kept in the shared region. This duty could perhaps be assumed by the IPython cluster controller.<br></div><div><br>Zach Pincus and I put together a Numpy shared memory thingy along these lines, using named, shared, non-file-backed regions with a locked reference count and automatic deallocation: <a href="https://github.com/erikhvatum/py_interprocess_shared_memory_blob">https://github.com/erikhvatum/py_interprocess_shared_memory_blob</a> It's MIT licensed, so any and all are welcome to use it for any purpose, without limitation.<br><br></div><div>I know how to go about doing the Windows side of the implementation (currently, only POSIX and Darwin support are complete), but haven't had a need thus far. If Windows support is something you want (even just for reference), drop me a line and I'll finish up the Windows parts.<br><br></div><div>Cheers,<br></div><div>Erik<br></div></div>