I don't think it's due to the warmup of the JIT.  Here's a simpler example.

import time
import multiprocessing

def do_nothing(): pass

if __name__ == '__main__':
   time1 = time.time()
   do_nothing()
   time2 = time.time()
   pool = multiprocessing.Pool(processes=1)
   time3 = time.time()
   result = pool.apply_async(do_nothing)
   result.get()
   time4 = time.time()
   result = pool.apply_async(do_nothing)
   result.get()
   time5 = time.time()
   pool.close()

   print('not multiprocessing: ' + str(time2 - time1))
   print('create pool: ' + str(time3 - time2))
   print('run first time: ' + str(time4 - time3))
   print('run second time: ' + str(time5 - time4))

Here are the results in PyPy.  The first call to do_nothing() using multiprocessing.Pool takes 0.57 seconds.

not multiprocessing: 0.0
create pool: 0.30999994278
run first time: 0.575999975204
run second time: 0.00100016593933

Here are the results in CPython.  It also appears to be have some overhead the first time the pool is used, but it's less severe than PyPy.

not multiprocessing: 0.0
create pool: 0.00500011444092
run first time: 0.134000062943
run second time: 0.0



On Fri, Sep 30, 2011 at 6:25 AM, Maciej Fijalkowski <fijall@gmail.com> wrote:
On Fri, Sep 30, 2011 at 10:20 AM, Armin Rigo <arigo@tunes.org> wrote:
> Hi,
>
> Is the conclusion just the fact that, again, the JIT's warm-up time is
> important, which we know very well?  Or is there some other effect
> that cannot be explained just by that?  (BTW, Laura, it's unrelated to
> multithreading if it's based on the multiprocessing module.)
>

I guess what people didn't realize is that if you spawn a new process,
you have to warmup the JIT *again* for each of the worker (at least in
the worst case scenario).

>
> A bientôt,
>
> Armin.
> _______________________________________________
> pypy-dev mailing list
> pypy-dev@python.org
> http://mail.python.org/mailman/listinfo/pypy-dev
>