[IPython-dev] pushAll and executeAll speed issue
fullung at gmail.com
Wed Nov 8 22:25:57 EST 2006
I'm doing some experiments with IPython1 on Windows. I'm using DeinoMPI's
GUI on top of mpi4py to start two kernels on my Core 2 Duo system. They
connect to the controller without problems. I make a RemoteController and
off I go. This is too cool.
(By the way: as I understand it, MPI is currently only being used to start
the kernels. From what I can see, it is not yet being used inside things
like pushAll, etc. Is this the next step to use the MPI functions if they
are available? Anybody working on this? If not, maybe I can try my hand at
it. Any hints or suggestions on where to start appreciated.)
Anyway, back to my experiment. The script I'm using is attached.
It starts off with some local benchmarks running on a single core. I get:
cPickle dumps time 4.95649055955
cPickle loads time 2.39126201794
local map time 10.3841574837
Then I push to all and time it. Here I get:
total pushAll time 70.6287321433
I was expecting something proportional to dumpsTime+N*loadsTime (if I
understand correctly that it is serially pushing out the data to each of the
N kernels) or maybe even less. I noticed that this section gets dramatically
slower even with a relatively small increase in data size. While this is
busy, I see Python using up all the CPU on one core as expected, but I also
see that the Windows "System" process is very busy.
Next up, the timing over the executeAll call that does the map:
executeAll time for map 139.186242386
but the kernels print:
engine map time 12.8248084603
engine map time 11.3900467797
which is only slightly more than the single core time. However, I have to
wonder where the heck the rest of the time went?! :-)
Do you guys see similar timings on Linux? Any thoughts on where my time is
going? Could it be Twisted? I tried with the latest from SVN and the
official 2.4.0 release for Windows.
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
More information about the IPython-dev