Re: [Python-Dev] python process creation overhead

Yes right, sorry I didn’t mean to imply that all that time was spent in the Python start up time. I’ve personally never actually spent time to figure out which part of that was slow because getting visibility inside of a subprocess.Popen is a pain and I’m slowly trying to rewrite our tests to not require subprocesses at all. I was only trying to chime in with a me too here about slow subprocess tests since our test suite ends up starting a couple thousand subprocesses as well and those tests are definitely our slowest. On May 10, 2014, at 6:50 PM, Guido van Rossum wrote:

Yeah, but 200 test in 30 minutes is 9 *seconds* per test -- the Python startup time is only a tiny fraction of that (20-40 *milliseconds*).

On Sat, May 10, 2014 at 3:33 PM, Donald Stufft wrote:

On May 10, 2014, at 5:46 PM, Victor Stinner wrote:

...

Le 10 mai 2014 22:51, "Gregory Szorc" a écrit :

...

Furthermore, Python 3 appears to be >50% slower than Python 2.

Please mention the minor version. It looks like you compared 2.7 and 3.3. Please test 3.4, we made interesting progress on the startup time.

There is still something to do, especially on OS X. Depending on the OS, different modules are loaded and some functions are implemented differently.

For what it's worth pip is the same way, about half of our test suite involves invoking (multiple) python processes. This has historically be really slow (~30 minutes to run ~200 tests of that type). We've been able to get the wall clock run time down by parallelizing these but the sequential time is still really slow.

----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org

-- --Guido van Rossum (python.org/~guido)

----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA