Hi Edwin, Thanks for providing some concrete numbers. Is it expected that creating 100 processes takes 6.3ms per process, but that creating 1000 process takes 40ms per process? That's over 6 times as long in the latter case. Cheers, Mark. On 12/06/2020 11:29 am, Edwin Zimmerman wrote:
On 6/12/2020 6:18 AM, Edwin Zimmerman wrote:
On 6/12/2020 5:08 AM, Paul Moore wrote:
On Fri, 12 Jun 2020 at 09:47, Mark Shannon wrote:
Starting a new process is cheap. On my machine, starting a new Python process takes under 1ms and uses a few Mbytes. Is that on Windows or Unix? Traditionally, process creation has been costly on Windows, which is why threads, and in-process solutions in general, tend to be more common on that platform. I haven't done experiments recently, but I do tend to avoid multiprocess-type solutions on Windows "just in case". I know that evaluating a new feature based on unsubstantiated assumptions informed by "it used to be like this" is ill-advised, but so is assuming that everything will be OK based on experience on a single platform :-) Here's a test on Windows 10, 4 logical cpus, 8 GB of ram:
timeit.timeit("""multiprocessing.Process(target=exit).start()""",number=100, globals=globals()) 0.6297528999999997 timeit.timeit("""multiprocessing.Process(target=exit).start()""",number=1000, globals=globals()) 40.281721199999964
Or this way:
timeit.timeit("""os.system('python.exe -c "exit()"')""",number=100, globals=globals()) 17.461259299999995
--Edwin For comparison, on a single core linux cloud server with 512 mb of ram:
timeit.timeit("""multiprocessing.Process(target=exit).start()""",number=100, globals=globals()) 0.354354709998006
timeit.timeit("""multiprocessing.Process(target=exit).start()""",number=1000, globals=globals()) 3.847851719998289
So yeah, process creation is still rather costly on Windows. _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/BLBNMZXK... Code of Conduct: http://python.org/psf/codeofconduct/