![](https://secure.gravatar.com/avatar/964204bccbce1be3610faccec94d7ca6.jpg?s=120&d=mm&r=g)
On 6/12/2020 2:17 PM, Chris Angelico wrote:
On Sat, Jun 13, 2020 at 3:50 AM Edwin Zimmerman <edwin@211mainstreet.net> wrote:
My previous timings were slightly inaccurate, as they compared spawning processes on Windows to forking on Linux. Also, I changed my timing code to run all process synchronously, to avoid hitting resource limits.
Updated Windows (Windows 7 this time, on a four core processor):
timeit.timeit('x=multiprocessing.Process(target=exit);x.start();x.join()', number=1000,globals = globals()) 84.7111053659259
Thanks, I was actually going to ask about joining the processes, since you don't really get a good indication of timings from asynchronous operations like that. Another interesting data point is that starting and joining in batches makes a fairly huge difference to performance, at least on my Linux system. Starting with your example and rescaling the number by ten to compensate for performance differences:
timeit.timeit('x=multiprocessing.Process(target=exit);x.start();x.join()', number=10000,globals = globals()) 14.261007152497768
Just for completeness and consistency, confirmed that adding a list comp around it doesn't change the timings:
timeit.timeit('xx=[multiprocessing.Process(target=exit) for _ in range(1)];[x.start() for x in xx];[x.join() for x in xx]', number=10000,globals = globals()) 14.030426062643528
But doing a hundred at a time and then joining them all cuts the time in half!
timeit.timeit('xx=[multiprocessing.Process(target=exit) for _ in range(100)];[x.start() for x in xx];[x.join() for x in xx]', number=100,globals = globals()) 5.470761131495237
The difference is even more drastic with spawn, although since it's slower, I also lowered the number of iterations.
ctx = multiprocessing.get_context('spawn') timeit.timeit('x=ctx.Process(target=exit);x.start();x.join()', number=1000,globals = globals()) 40.82687543518841 timeit.timeit('xx=[ctx.Process(target=exit) for _ in range(100)];[x.start() for x in xx];[x.join() for x in xx]', number=10,globals = globals())8.566341979429126 8.566341979429126
Would be curious to know if that's the same on Windows. Yea, it's the same. Watch your cpu utilization, and you will realize that your list comprehension is parallelizing the process startups. ChrisA _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/YFRM3LNO... Code of Conduct: http://python.org/psf/codeofconduct/