I don't think it's due to the warmup of the JIT. Here's a simpler example. import time import multiprocessing def do_nothing(): pass if __name__ == '__main__': time1 = time.time() do_nothing() time2 = time.time() pool = multiprocessing.Pool(processes=1) time3 = time.time() result = pool.apply_async(do_nothing) result.get() time4 = time.time() result = pool.apply_async(do_nothing) result.get() time5 = time.time() pool.close() print('not multiprocessing: ' + str(time2 - time1)) print('create pool: ' + str(time3 - time2)) print('run first time: ' + str(time4 - time3)) print('run second time: ' + str(time5 - time4)) Here are the results in PyPy. The first call to do_nothing() using multiprocessing.Pool takes 0.57 seconds. not multiprocessing: 0.0 create pool: 0.30999994278 run first time: 0.575999975204 run second time: 0.00100016593933 Here are the results in CPython. It also appears to be have some overhead the first time the pool is used, but it's less severe than PyPy. not multiprocessing: 0.0 create pool: 0.00500011444092 run first time: 0.134000062943 run second time: 0.0 On Fri, Sep 30, 2011 at 6:25 AM, Maciej Fijalkowski <fijall@gmail.com>wrote:
On Fri, Sep 30, 2011 at 10:20 AM, Armin Rigo <arigo@tunes.org> wrote:
Hi,
Is the conclusion just the fact that, again, the JIT's warm-up time is important, which we know very well? Or is there some other effect that cannot be explained just by that? (BTW, Laura, it's unrelated to multithreading if it's based on the multiprocessing module.)
I guess what people didn't realize is that if you spawn a new process, you have to warmup the JIT *again* for each of the worker (at least in the worst case scenario).
A bientôt,
Armin. _______________________________________________ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev