[New-bugs-announce] [issue30198] distutils build_ext: don't run newer_group() in parallel in multiple threads when using parallel

[STINNER Victor]

New submission from STINNER Victor:

Since Python 3.5, distutils is able to build extensions in parallel, nice enhancement! But setup.py of CPython is 2x slower in parallel mode when all modules are already built: (1 sec vs 500 ms).

Building extensions calls newer_group() which calls os.stat() 6,856 times. I wrote a cache for os.stat() but it has no impact on performance.

It seems like threads are fighting to death for the GIL in the os.stat() race...

Attached pull request calls newer_group() before spawning threads in parallel mode, so "setup.py build" takes the same time with and without parallel module, when all extensions are already built.

I didn't measure performance when all extensions must be built.

----------
components: Distutils
messages: 292530
nosy: dstufft, haypo, merwok
priority: normal
severity: normal
status: open
title: distutils build_ext: don't run newer_group() in parallel in multiple threads when using parallel
type: performance
versions: Python 3.7

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue30198>
_______________________________________