[issue11314] Subprocess suffers 40% process creation overhead penalty

[Charles-Francois Natali]

Charles-Francois Natali <neologix at free.fr> added the comment:

> If you stop to think about it, though, this is actually a shockingly huge percent increase. In any process creation scenario I'm familiar with, its overhead should be so small that you could bump it up several orders of magnitude and still not compete with executing a shell and asking it to do anything, even just exit.

os.popen just calls the popen(3) library call, which just performs a fork/execve and some dup/close in between.
subprocess.Popen is implemented in Python, so it doesn't come as a surprise that it's slower in your example.
But I think there's a point you're missing: you're not just creating a child process, but a shell.
subprocess is actually faster than popen to spawn executables (not scripts), because you don't have to spawn a shell.

For example:

$ cat /tmp/test.c 
int main(int argc, char *argv[])
{
        return 0;
}

$ cat /tmp/test_subprocess.py 
import subprocess

for i in range(10000):
    f = subprocess.Popen('/tmp/test')
    f.wait()

$ cat /tmp/test_popen.py 
import os

for i in range(10000):
    f = os.popen('/tmp/test')
    f.close()

$ time ./python /tmp/test_subprocess.py

real    0m13.933s
user    0m3.083s
sys     0m12.441s

$ time ./python /tmp/test_popen.py 

real    0m18.235s
user    0m4.293s
sys     0m15.176s

Given of important the subprocess overhead seems to you, I guess that the processes you're launching are not shell scripts, and thus you're probably better off using subprocess.

----------
nosy: +neologix

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11314>
_______________________________________