On Mon, Jun 22, 2015 at 4:29 PM, Gregory P. Smith <greg@krypto.org> wrote:
On Mon, Jun 22, 2015 at 3:51 PM Sturla Molden <sturla.molden@gmail.com> wrote:
"Gregory P. Smith" <greg@krypto.org> wrote:
What this means for subinterpreters in this case is not much different from starting up multiple worker processes: You need to start them up and wait for them to be ready to serve, then reuse them as long as feasible before recycling them to start up a new one. The startup cost is high.
The statup cost for worker processes is high on Windows. It is very small on nearly any other OS.
While I understand that Windows adds some overhead there, startup time for Python worker processes is high on all OSes.
Python startup is slow in general. It slows down further based on the modules you must import before you can begin work.
Python does *very* little work on fork, which is what Sturla is alluding to. (Fork doesn't exist on Windows.) The only part I've found forking to be slow with is if you need to delay initialization of a thread pool and everything that depends on a thread pool until after the fork. This could hypothetically be made faster with subinterpreters if the thread pool was shared among all subinterpreters (e.g. if it was written in C.), but I would *expect* fork to be faster overall. That said, worker startup time is not actually very interesting anyway, since workers should restart rarely. I think its biggest impact is probably the time it takes to start your entire task from scratch. -- Devin