[Python-Dev] python process creation overhead

[Donald Stufft]

Yes right, sorry I didn’t mean to imply that all that time was spent in the
Python start up time. I’ve personally never actually spent time to figure out
which part of that was slow because getting visibility inside of a
subprocess.Popen is a pain and I’m slowly trying to rewrite our tests to not
require subprocesses at all.

I was only trying to chime in with a me too here about slow subprocess tests
since our test suite ends up starting a couple thousand subprocesses as well
and those tests are definitely our slowest.

On May 10, 2014, at 6:50 PM, Guido van Rossum <guido at python.org> wrote:

> Yeah, but 200 test in 30 minutes is 9 *seconds* per test -- the Python startup time is only a tiny fraction of that (20-40 *milliseconds*).
> 
> 
> On Sat, May 10, 2014 at 3:33 PM, Donald Stufft <donald at stufft.io> wrote:
> 
> On May 10, 2014, at 5:46 PM, Victor Stinner <victor.stinner at gmail.com> wrote:
> 
>> Le 10 mai 2014 22:51, "Gregory Szorc" <gregory.szorc at gmail.com> a écrit :
>> > Furthermore, Python 3 appears to be >50% slower than Python 2.
>> 
>> Please mention the minor version. It looks like you compared 2.7 and 3.3. Please test 3.4, we made interesting progress on the startup time.
>> 
>> There is still something to do, especially on OS X. Depending on the OS, different modules are loaded and some functions are implemented differently.
>> 
> 
> For what it's worth pip is the same way, about half of our test suite involves
> invoking (multiple) python processes. This has historically be really slow
> (~30 minutes to run ~200 tests of that type). We've been able to get the wall
> clock run time down by parallelizing these but the sequential time is still
> really slow.
> 
> -----------------
> Donald Stufft
> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
> 
> 
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org
> 
> 
> 
> 
> -- 
> --Guido van Rossum (python.org/~guido)


-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140510/37e1e4b2/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140510/37e1e4b2/attachment.sig>