On Sat, Jun 13, 2020 at 3:50 AM Edwin Zimmerman <edwin@211mainstreet.net> wrote:
My previous timings were slightly inaccurate, as they compared spawning processes on Windows to forking on Linux. Also, I changed my timing code to run all process synchronously, to avoid hitting resource limits.
Updated Windows (Windows 7 this time, on a four core processor):
timeit.timeit('x=multiprocessing.Process(target=exit);x.start();x.join()', number=1000,globals = globals()) 84.7111053659259
Thanks, I was actually going to ask about joining the processes, since you don't really get a good indication of timings from asynchronous operations like that. Another interesting data point is that starting and joining in batches makes a fairly huge difference to performance, at least on my Linux system. Starting with your example and rescaling the number by ten to compensate for performance differences:
timeit.timeit('x=multiprocessing.Process(target=exit);x.start();x.join()', number=10000,globals = globals()) 14.261007152497768
Just for completeness and consistency, confirmed that adding a list comp around it doesn't change the timings:
timeit.timeit('xx=[multiprocessing.Process(target=exit) for _ in range(1)];[x.start() for x in xx];[x.join() for x in xx]', number=10000,globals = globals()) 14.030426062643528
But doing a hundred at a time and then joining them all cuts the time in half!
timeit.timeit('xx=[multiprocessing.Process(target=exit) for _ in range(100)];[x.start() for x in xx];[x.join() for x in xx]', number=100,globals = globals()) 5.470761131495237
The difference is even more drastic with spawn, although since it's slower, I also lowered the number of iterations.
ctx = multiprocessing.get_context('spawn') timeit.timeit('x=ctx.Process(target=exit);x.start();x.join()', number=1000,globals = globals()) 40.82687543518841 timeit.timeit('xx=[ctx.Process(target=exit) for _ in range(100)];[x.start() for x in xx];[x.join() for x in xx]', number=10,globals = globals())8.566341979429126 8.566341979429126
Would be curious to know if that's the same on Windows. ChrisA