On 23/06/15 01:29, Gregory P. Smith wrote:
While I understand that Windows adds some overhead there, startup time for Python worker processes is high on all OSes.
No it is not. A fork() will clone the process. You don't need to run any initialization code after that. You don't need to start a new Python interpreter -- you already have one. You don't need to run module imports -- they are already imported. You don't need to pickle and build Python objects -- they are already there. Everything you had in the parent process is ready to use the child process. This magic happens so fast it is comparable to the time it takes Windows to start a thread. On Windows, CreateProcess starts an "almost empty" process. You therefore have a lot of setup code to run. This is what makes starting Python processes with multiprocessing so much slower on Windows. It is not that Windows processes are more hevy-weight than threads, they are, but the real issue is all the setup code you need to run. On Linux and Mac, you don't need to run any setup code code after a fork(). Sturla