[issue35823] Use vfork() in subprocess on Linux

[Gregory P. Smith]

Gregory P. Smith <greg at krypto.org> added the comment:

Performance improvement measured on a 1.4Ghz quad aarch64 A57 (nvidia jetson nano):

#define VFORK_USABLE 1
 test_subprocess: 36.5 seconds

#undef VFORK_USABLE
 test_subprocess: 45 seconds

Nice.  I really didn't expect a 20% speedup on its testsuite alone!

Lets dive into that with a microbenchmark:

$ ./build-novfork/python ten_seconds_of_truth.py
Launching /bin/true for 10 seconds in a loop.
Launched 2713 subprocesses in 10.00194378796732 seconds.
271.247275281014 subprocesses/second.
Increased our mapped pages by 778240KiB.
Launching /bin/true for 10 seconds in a loop.
Launched 212 subprocesses in 10.006392606999725 seconds.
21.186456331095847 subprocesses/second.

$ ./build/python ten_seconds_of_truth.py
Launching /bin/true for 10 seconds in a loop.
Launched 3310 subprocesses in 10.001623224001378 seconds.
330.94628000551285 subprocesses/second.
Increased our mapped pages by 778240KiB.
Launching /bin/true for 10 seconds in a loop.
Launched 3312 subprocesses in 10.001519071985967 seconds.
331.1496959773679 subprocesses/second.

Demonstrating perfectly the benefit of vfork().  The more mapped pages, the slower regular fork() becomes.  With vfork() the size of the process's memory map does not matter.

ten_seconds_of_truth.py:

```python
from time import monotonic as now
import subprocess

def benchmark_for_ten():
    print('Launching /bin/true for 10 seconds in a loop.')

    count = 0
    start = now()
    while now() - start < 10.:
        subprocess.run(['/bin/true'])
        count += 1
    end = now()
    duration = end-start

    print(f'Launched {count} subprocesses in {duration} seconds.')
    print(f'{count/duration} subprocesses/second.')

benchmark_for_ten()

map_a_bunch_of_pages = '4agkahglahaa^#\0ag3\3'*1024*1024*40

print(f'Increased our mapped pages by {len(map_a_bunch_of_pages)//1024}KiB.')

benchmark_for_ten()
```

----------

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue35823>
_______________________________________