[SciPy-User] Poor scalability of embarrassingly parallel code with multiprocessing

[Brian Granger]

I haven't looked at your algorithm, but using "time" to time parallel codes
like this
can give very unreliable results.  I have seen many times people get
prematurely excited or
depressed about the parallel scaling of a program using time.  So the first
step would be to figure out
a better way of timing your code.

Brian

On Sat, Oct 24, 2009 at 6:31 AM, Rohit Garg <rpg.314 at gmail.com> wrote:

> Hi,
>
> I am attaching a very simple, embarrassingly parallel code. The
> problem is that it shows practically no speedup at all on my dual-core
> machine.
>
> 64 bit, python 2.6,
> ==============
> ~/Documents/lingo/python/multiprocessing at rpg> time python pool-test.py 3
> 3
> done
>
> real    1m5.686s
> user    1m41.881s
> sys     0m2.846s
> ~/Documents/lingo/python/multiprocessing at rpg> time python pool-test.py 2
> 2
> done
>
> real    1m6.052s
> user    1m45.550s
> sys     0m2.797s
> ~/Documents/lingo/python/multiprocessing at rpg> time python pool-test.py 1
> 1
> done
>
> real    1m21.696s
> user    1m58.327s
> sys     0m2.426s
> ~/Documents/lingo/python/multiprocessing at rpg> time python pool-test.py 4
> 4
> done
>
> real    1m3.422s
> user    1m39.742s
> sys     0m3.179s
> ========================
>
> And the following tests were done on a quad core machine over ssh, (so
> network latencies *may* be part of the problem here).
>
> 32 bit, python 2.6,
> =============================
> ~@pixel> time python pool-test.py 1
> 1
> done
>
> real    0m58.772s
> user    1m24.549s
> sys     0m1.720s
> ~@pixel> time python pool-test.py 2
> 2
> done
>
> real    0m44.396s
> user    1m18.525s
> sys     0m2.000s
> ~@pixel> time python pool-test.py 3
> 3
> done
>
> real    0m45.898s
> user    1m21.293s
> sys     0m1.904s
> ~@pixel> time python pool-test.py 4
> 4
> done
>
> real    0m44.196s
> user    1m19.561s
> sys     0m2.056s
> ~@pixel> time python pool-test.py 5
> 5
> done
>
> real    0m47.451s
> user    1m24.997s
> sys     0m2.416s
> ~@pixel> time python pool-test.py 6
> 6
> done
>
> real    0m49.181s
> user    1m25.881s
> sys     0m2.356s
> ==============================
>
> The code which delivered these results is here
>
>
> ===============================
> #test of Pool's map capablities
> from multiprocessing import Pool
> import numpy
> import sys
>
> procs=int(sys.argv[1])
> print procs
> def f(x):
>    index,probs=x
>    return index,2.0*probs
>
> prob_samples=1000000
>
> probX=numpy.linspace(0.2, 0.3, prob_samples)
>
> Input=[(i,probX[i]) for i in xrange(prob_samples) ]
>
> pool = Pool(processes=procs)
>
> pool.map(f, Input)
>
> print 'done'
> ===================
>
>
> Did I make a mistake somewhere? What have been your experiences with
> multiprocessing module in general and pool in particular? What
> approaches would you suggest to improve the speed scaling.
>
> Regards,
> --
> Rohit Garg
>
> http://rpg-314.blogspot.com/
>
> Senior Undergraduate
> Department of Physics
> Indian Institute of Technology
> Bombay
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20091024/a5b1ce2b/attachment.html>