[Python-ideas] solving multi-core Python

[Sturla Molden]

On 23/06/15 01:29, Gregory P. Smith wrote:

> While I understand that Windows adds some overhead there, startup time
> for Python worker processes is high on all OSes.

No it is not.

A fork() will clone the process. You don't need to run any 
initialization code after that. You don't need to start a new Python 
interpreter -- you already have one. You don't need to run module 
imports -- they are already imported. You don't need to pickle and build 
Python objects -- they are already there. Everything you had in the 
parent process is ready to use the child process. This magic happens so 
fast it is comparable to the time it takes Windows to start a thread.

On Windows, CreateProcess starts an "almost empty" process. You 
therefore have a lot of setup code to run. This is what makes starting 
Python processes with multiprocessing so much slower on Windows. It is 
not that Windows processes are more hevy-weight than threads, they are, 
but the real issue is all the setup code you need to run.

On Linux and Mac, you don't need to run any setup code code after a fork().


Sturla