I don't think it's due to the warmup of the JIT. Here's a simpler example.<br><br>import time<br>import multiprocessing<br><br>def do_nothing(): pass<br><br>if __name__ == '__main__':<br> time1 = time.time()<br>
do_nothing()<br> time2 = time.time()<br> pool = multiprocessing.Pool(processes=1)<br> time3 = time.time()<br> result = pool.apply_async(do_nothing)<br> result.get()<br> time4 = time.time()<br> result = pool.apply_async(do_nothing)<br>
result.get()<br> time5 = time.time()<br> pool.close()<br><br> print('not multiprocessing: ' + str(time2 - time1))<br> print('create pool: ' + str(time3 - time2))<br> print('run first time: ' + str(time4 - time3))<br>
print('run second time: ' + str(time5 - time4))<br><br>Here are the results in PyPy. The first call to do_nothing() using multiprocessing.Pool takes 0.57 seconds.<br><br>not multiprocessing: 0.0<br>create pool: 0.30999994278<br>
run first time: 0.575999975204<br>run second time: 0.00100016593933<br><br>Here are the results in CPython. It also appears to be have some overhead the first time the pool is used, but it's less severe than PyPy.<br>
<br>not multiprocessing: 0.0<br>create pool: 0.00500011444092<br>run first time: 0.134000062943<br>run second time: 0.0<br><br><br><br><div class="gmail_quote">On Fri, Sep 30, 2011 at 6:25 AM, Maciej Fijalkowski <span dir="ltr"><<a href="mailto:fijall@gmail.com" target="_blank">fijall@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div></div><div>On Fri, Sep 30, 2011 at 10:20 AM, Armin Rigo <<a href="mailto:arigo@tunes.org" target="_blank">arigo@tunes.org</a>> wrote:<br>
> Hi,<br>
><br>
> Is the conclusion just the fact that, again, the JIT's warm-up time is<br>
> important, which we know very well? Or is there some other effect<br>
> that cannot be explained just by that? (BTW, Laura, it's unrelated to<br>
> multithreading if it's based on the multiprocessing module.)<br>
><br>
<br>
</div></div>I guess what people didn't realize is that if you spawn a new process,<br>
you have to warmup the JIT *again* for each of the worker (at least in<br>
the worst case scenario).<br>
<br>
><br>
> A bientôt,<br>
><br>
> Armin.<br>
<div><div></div><div>> _______________________________________________<br>
> pypy-dev mailing list<br>
> <a href="mailto:pypy-dev@python.org" target="_blank">pypy-dev@python.org</a><br>
> <a href="http://mail.python.org/mailman/listinfo/pypy-dev" target="_blank">http://mail.python.org/mailman/listinfo/pypy-dev</a><br>
><br>
</div></div></blockquote></div><br>