[New-bugs-announce] [issue41116] build on macOS 11 (beta) does not find system-supplied third-party libraries

[Ned Deily]

New submission from Ned Deily <nad at python.org>:

When building on macOS 11 (beta), a number of modules that should normally build on macOS fail because the system-supplied third-party libraries are not found.

The necessary bits to build these optional modules were not found:
_bz2                  _curses               _curses_panel
_gdbm                 _hashlib              _lzma
_ssl                  ossaudiodev           readline
spwd                  zlib

The list should look like this (with no additional third-party libs supplied from another source like Homebrew or MacPorts):

The necessary bits to build these optional modules were not found:
_gdbm                 _hashlib              _ssl
ossaudiodev           spwd

The problem is due to a change in the 11 beta versus 10.15 or earlier systems: 
"New in macOS Big Sur 11 beta, the system ships with a built-in dynamic linker cache of all system-provided libraries. As part of this change, copies of dynamic libraries are no longer present on the filesystem. Code that attempts to check for dynamic library presence by looking for a file at a path or enumerating a directory will fail. Instead, check for library presence by attempting to dlopen() the path, which will correctly check for the library in the cache."

This breaks tests in setup.py using find_library_file() to determine if a library is present and in what directory it exists.  setup.py depends on Lib/distutils/unixccompiler.py to do the dirty work. A similar problem arose on earlier macOS releases when header files could no longer be installed in the systems /usr/include; setup.py had to be taught to look in the SDK being used implicitly or explicitly by the compiler preprocessor.  We could probably do something like that here while trying to avoid changes that might break downstream supplements/replacements to distutils, for example, setuptools.

There is a workaround: explicitly specify the SDK location to ./configure (you also need to specify the universal archs setting):

./configure \
    --enable-universalsdk=$(xcodebuild -version -sdk macosx Path) \
    --with-universal-archs=intel-64 \
    ...

----------
components: macOS
messages: 372379
nosy: ned.deily, ronaldoussoren
priority: high
severity: normal
status: open
title: build on macOS 11 (beta) does not find system-supplied third-party libraries
versions: Python 3.10, Python 3.7, Python 3.8, Python 3.9

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue41116>
_______________________________________