My previous timings were slightly inaccurate, as they compared spawning processes on Windows to forking on Linux. Also, I changed my timing code to run all process synchronously, to avoid hitting resource limits. Updated Windows (Windows 7 this time, on a four core processor):
timeit.timeit('x=multiprocessing.Process(target=exit);x.start();x.join()', number=1000,globals = globals())
84.7111053659259 Updated Linux with spawn (single core processor):
ctx = multiprocessing.get_context('spawn')
timeit.timeit('x=ctx.Process(target=exit);x.start();x.join()', number=1000,globals = globals())
60.01154333699378 Updated Linux with fork:
timeit.timeit('x=multiprocessing.Process(target=exit);x.start();x.join()', number=1000,globals = globals())
4.402019854984246 Compare this to subinterpreters on my linux machine:
timeit.timeit('s=_xxsubinterpreters.create();_xxsubinterpreters.destroy(s)',number=1000, globals=globals())
13.47043095799745 This shows that is speed is all that matters, multiprocessing comes out way ahead of subinterpreters on linux, but way behind on Windows. I need to time subinterpreters on Windows yet for the full picture, but that will be tomorrow till I get that done. --Edwin From: Emily Bowman [mailto:silverbacknet@gmail.com] Sent: Friday, June 12, 2020 12:44 PM To: Mark Shannon <mark@hotpy.org> Cc: Python Dev <python-dev@python.org> Subject: [Python-Dev] Re: My take on multiple interpreters (Was: Should we be making so many changes in pursuit of PEP 554?) On Fri, Jun 12, 2020 at 7:19 AM Mark Shannon <mark@hotpy.org <mailto:mark@hotpy.org> > wrote: Hi Edwin, Thanks for providing some concrete numbers. Is it expected that creating 100 processes takes 6.3ms per process, but that creating 1000 process takes 40ms per process? That's over 6 times as long in the latter case. Cheers, Mark. On 12/06/2020 11:29 am, Edwin Zimmerman wrote:
On 6/12/2020 6:18 AM, Edwin Zimmerman wrote:
On 6/12/2020 5:08 AM, Paul Moore wrote:
On Fri, 12 Jun 2020 at 09:47, Mark Shannon <mark@hotpy.org <mailto:mark@hotpy.org> > wrote:
Starting a new process is cheap. On my machine, starting a new Python process takes under 1ms and uses a few Mbytes. Is that on Windows or Unix? Traditionally, process creation has been costly on Windows, which is why threads, and in-process solutions in general, tend to be more common on that platform. I haven't done experiments recently, but I do tend to avoid multiprocess-type solutions on Windows "just in case". I know that evaluating a new feature based on unsubstantiated assumptions informed by "it used to be like this" is ill-advised, but so is assuming that everything will be OK based on experience on a single platform :-) Here's a test on Windows 10, 4 logical cpus, 8 GB of ram:
timeit.timeit("""multiprocessing.Process(target=exit).start()""",number=100, globals=globals()) 0.6297528999999997 timeit.timeit("""multiprocessing.Process(target=exit).start()""",number=1000, globals=globals()) 40.281721199999964
Or this way:
timeit.timeit("""os.system('python.exe -c "exit()"')""",number=100, globals=globals()) 17.461259299999995
--Edwin For comparison, on a single core linux cloud server with 512 mb of ram:
timeit.timeit("""multiprocessing.Process(target=exit).start()""",number=100, globals=globals()) 0.354354709998006
timeit.timeit("""multiprocessing.Process(target=exit).start()""",number=1000, globals=globals()) 3.847851719998289
So yeah, process creation is still rather costly on Windows.
I was wondering that too, some tests show that process creation/destruction starts to seriously bog down after a few hundred in a row. I guess it's hitting some resource limits it has to clean up, though creating hundreds of processes at once sounds like an antipattern that doesn't really deserve too much consideration. It would be rare that fork is more than a negligible part of any workload. (With A/V on, though, it's _much_ slower out the gate. I'm seeing over 100ms per process with Kaspersky running.) Em