[New-bugs-announce] [issue43459] Race conditions when the same source file used to build mutliple extensions

[Michał Górny]

New submission from Michał Górny <mgorny at gentoo.org>:

There is a race condition in distutils' build_ext implementation.  When the same source file is used to build multiple extensions, distutils attempts to build it multiple times using the same output file, in parallel.  This means that the link editor can grab the file while another compiler instance is overwriting it.  The results vary from compile errors to cryptic dyld failures when attempting to load the module.

I've created a trivial reproducer that I've attached in a patch form.  For convenience, it's also available on my GitHub: https://github.com/mgorny/distutils-build_ext-race

The reproducer consists of two extension modules sharing the same file.  The race.sh script attempts to build the extension and then import it.  The process is repeated until something fails, e.g.:

+ python3.10 setup.py build_ext -i -j4
running build_ext
building 'bar' extension
creating build
building 'foo' extension
creating build/temp.linux-x86_64-3.10
creating build/temp.linux-x86_64-3.10
x86_64-pc-linux-gnu-gcc-10.2.0 -pthread -fPIC -I/usr/include/python3.10 -c bar.c -o build/temp.linux-x86_64-3.10/bar.o
x86_64-pc-linux-gnu-gcc-10.2.0 -pthread -fPIC -I/usr/include/python3.10 -c foo.c -o build/temp.linux-x86_64-3.10/foo.o
x86_64-pc-linux-gnu-gcc-10.2.0 -pthread -fPIC -I/usr/include/python3.10 -c shared.c -o build/temp.linux-x86_64-3.10/shared.o
x86_64-pc-linux-gnu-gcc-10.2.0 -pthread -fPIC -I/usr/include/python3.10 -c shared.c -o build/temp.linux-x86_64-3.10/shared.o
x86_64-pc-linux-gnu-gcc-10.2.0 -pthread -shared -Wl,-O1 -Wl,--as-needed -Wl,--hash-style=gnu build/temp.linux-x86_64-3.10/foo.o build/temp.linux-x86_64-3.10/shared.o -L/usr/lib64 -o /home/mgorny/git/distutils-build_ext-race/foo.cpython-310-x86_64-linux-gnu.so
x86_64-pc-linux-gnu-gcc-10.2.0 -pthread -shared -Wl,-O1 -Wl,--as-needed -Wl,--hash-style=gnu build/temp.linux-x86_64-3.10/bar.o build/temp.linux-x86_64-3.10/shared.o -L/usr/lib64 -o /home/mgorny/git/distutils-build_ext-race/bar.cpython-310-x86_64-linux-gnu.so
+ python3.10 -c 'import foo; import bar'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: /home/mgorny/git/distutils-build_ext-race/foo.cpython-310-x86_64-linux-gnu.so: undefined symbol: call_shared
+ echo 'Reproduced at iteration 256'
Reproduced at iteration 256
+ break

----------
components: Distutils
files: 0001-A-reproducer-for-distutils-build_ext-race-condition.patch
keywords: patch
messages: 388410
nosy: dstufft, eric.araujo, mgorny
priority: normal
severity: normal
status: open
title: Race conditions when the same source file used to build mutliple extensions
type: compile error
versions: Python 3.10, Python 3.6, Python 3.7, Python 3.8, Python 3.9
Added file: https://bugs.python.org/file49863/0001-A-reproducer-for-distutils-build_ext-race-condition.patch

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue43459>
_______________________________________