[pypy-dev] I was talking with Russel Winder at PyCON UK.

[Josh Ayers]

Here's a further modified version.  In this case, when using the pool for
the first time, it uses an n of 10, instead of 100 million.  Even with such
a low precision, the first execution takes 1.3 seconds.  It seems some
significant warm up time is needed the first time a multiprocessing.Pool
object is used.

See the attachment or this link for the code: http://pastie.org/2614925



On Thu, Sep 29, 2011 at 8:24 PM, Josh Ayers <josh.ayers at gmail.com> wrote:

> I think the slowdown you're seeing is due to the time it takes to create
> new processes.  This seems to be quite a bit slower in PyPy than in
> CPython.  However, once the process pool is created and has been used once,
> the execution time vs. process count behaves as expected.
>
> I attached a modified version of your code to demonstrate the behavior.  It
> calculates Pi once without using multiprocessing as a baseline for
> comparison.  Then a multiprocessing.Pool object is created with 8 processes,
> and the same pool is used multiple times.  On my machine, creating the 8 new
> processes takes 0.60 seconds in PyPy and only 0.20 seconds in CPython.
>
> The pool is first used two times in a row with only a single process
> active.  For some reason, the second run is a factor of 2 faster than the
> first.  Is this just warmup of the JIT, or some other behavior?
>
> Next, it repeats using 2, 4, and 8 processes.  This was run on a 4 core
> machine, and as expected there was an improvement in run time with 2 and 4
> processes.  Using 8 processes gives approximately the same run time as 4.
>
> The output is pasted below.  I also pasted the modified code here in case
> the attached file doesn't come through: http://pastie.org/2614751.  For
> reference, I'm running PyPy 1.6 on Windows 7.
>
> Sincerely,
> Josh
>
>
> C:\Users\jayers\Documents\SVN\randomStuff\pypy_comparisons>pypy-c
> pi_python2_multiprocessing_pool.py
>
> 3.14159265359
> non parallel execution time: 1.52899980545
> pool creation time: 0.559000015259
> ==== Python Multiprocessing Pool pi = 3.14159265359
> ==== Python Multiprocessing Pool iteration count = 100000000
> ==== Python Multiprocessing Pool elapse = 3.1930000782
> ==== Python Multiprocessing Pool process count = 1
> ==== Python Multiprocessing Pool processor count = 4
>
>
> ==== Python Multiprocessing Pool pi = 3.14159265359
> ==== Python Multiprocessing Pool iteration count = 100000000
> ==== Python Multiprocessing Pool elapse = 1.53900003433
> ==== Python Multiprocessing Pool process count = 1
> ==== Python Multiprocessing Pool processor count = 4
>
>
> ==== Python Multiprocessing Pool pi = 3.14159265359
> ==== Python Multiprocessing Pool iteration count = 100000000
> ==== Python Multiprocessing Pool elapse = 0.802000045776
> ==== Python Multiprocessing Pool process count = 2
> ==== Python Multiprocessing Pool processor count = 4
>
>
> ==== Python Multiprocessing Pool pi = 3.14159265359
> ==== Python Multiprocessing Pool iteration count = 100000000
> ==== Python Multiprocessing Pool elapse = 0.441999912262
> ==== Python Multiprocessing Pool process count = 4
> ==== Python Multiprocessing Pool processor count = 4
>
>
> ==== Python Multiprocessing Pool pi = 3.14159265359
> ==== Python Multiprocessing Pool iteration count = 100000000
> ==== Python Multiprocessing Pool elapse = 0.457000017166
> ==== Python Multiprocessing Pool process count = 8
> ==== Python Multiprocessing Pool processor count = 4
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110929/0a726f89/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: pi_python2_multiprocessing_pool_v2.py
Type: application/octet-stream
Size: 1577 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20110929/0a726f89/attachment.obj>