How official binaries are built?
Hi, all. I want Homebrew uses `--enable-optimizations` and `--with-lto` option for building Python. But maintainer said:
Given this is not a default option, probably not, unless it is done in upstream (“official”) binaries.
https://github.com/Homebrew/homebrew-core/pull/45337
Are these options used for official macOS binaries?
Is there official information about the build step of official binaries?
Regards,
--
Inada Naoki
On Oct 15, 2019, at 04:54, Inada Naoki
I want Homebrew uses `--enable-optimizations` and `--with-lto` option for building Python. But maintainer said:
Given this is not a default option, probably not, unless it is done in upstream (“official”) binaries.
https://github.com/Homebrew/homebrew-core/pull/45337
Are these options used for official macOS binaries? Is there official information about the build step of official binaries?
We currently do not use those options to build the binaries for the python.org macOS installers. The main reason is that the Pythons we provide are built to support a wide-range of macOS releases and to do so safely we build the binaries on the oldest version of macOS supported by that installer. So, for example, the 10.9+ installer variant is built on a 10.9 system. Some of the optimization features either aren't available or are less robust on older build tools. And I believe it is more important for the python.org macOS installers to continue to provide a single installer that is usable on many systems and can be used in a broad range of applications and by a broad range of users rather than trying to optimize performance for a specific application: you can always build your own Python. As far as what other distributors of Python for macOS do, what we do shouldn't necessarily constrain them. I don't see any problem with Homebrew optimizing for a particular user's installation. I see that MacPorts, another distributor of Python on macOS, provides a non-default variant that uses --enable-optimizations. https://github.com/macports/macports-ports/blob/master/lang/python37/Portfil... -- Ned Deily nad@python.org -- []
Hi Inada-san, You can query the sysconfig module to check how Python has been built. Example: pyvstinner@apu$ python3 Python 3.7.4 (default, Jul 9 2019, 16:32:37) [GCC 9.1.1 20190503 (Red Hat 9.1.1-1)] on linux Type "help", "copyright", "credits" or "license" for more information.
import sysconfig
sysconfig.get_config_var('PY_CFLAGS') '-Wno-unused-result -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv'
sysconfig.get_config_var('PY_CFLAGS_NODIST') '-O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -flto -fuse-linker-plugin -ffat-lto-objects -flto-partition=none -g -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Wno-cast-function-type -Werror=implicit-function-declaration -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -fprofile-use -fprofile-correction'
sysconfig.get_config_var('PY_LDFLAGS') '-Wl,-z,relro -Wl,--as-needed -Wl,-z,now -g'
sysconfig.get_config_var('PY_LDFLAGS_NODIST') '-Wl,-z,relro -Wl,--as-needed -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -g -flto -fuse-linker-plugin -ffat-lto-objects -flto-partition=none -g'
import shlex [opt for opt in shlex.split(sysconfig.get_config_var('PY_CFLAGS') + sysconfig.get_config_var('PY_CFLAGS_NODIST')) if opt.startswith('-O') or 'profile' in opt] ['-O2', '-O2', '-fprofile-use', '-fprofile-correction']
[opt for opt in shlex.split(sysconfig.get_config_var('PY_LDFLAGS') + sysconfig.get_config_var('PY_LDFLAGS_NODIST')) if 'lto' in opt] ['-flto', '-ffat-lto-objects', '-flto-partition=none']
You can see that Fedora 30 /usr/bin/python3.7 is built using -O2 and
has been optimized with PGO (compiler flag -fprofile-use) and LTO
(linker flag -flto).
Victor
Le mar. 15 oct. 2019 à 11:02, Inada Naoki
Hi, all.
I want Homebrew uses `--enable-optimizations` and `--with-lto` option for building Python. But maintainer said:
Given this is not a default option, probably not, unless it is done in upstream (“official”) binaries.
https://github.com/Homebrew/homebrew-core/pull/45337
Are these options used for official macOS binaries? Is there official information about the build step of official binaries?
Regards, -- Inada Naoki
_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/EILILECN... Code of Conduct: http://python.org/psf/codeofconduct/
-- Night gathers, and now my watch begins. It shall not end until my death.
On Tue, Oct 15, 2019 at 10:57 PM Victor Stinner
Hi Inada-san,
You can query the sysconfig module to check how Python has been built.
Thank you for pointing it out. It seems official macOS binary doesn't use --enable-optimizations and --with-lto options... Python 3.8.0 (v3.8.0:fa919fdf25, Oct 14 2019, 10:23:27) [Clang 6.0 (clang-600.0.57)] on darwin Type "help", "copyright", "credits" or "license" for more information.
import sysconfig sysconfig.get_config_var('PY_CFLAGS') '-Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -arch x86_64 -g' sysconfig.get_config_var('PY_CFLAGS_NODIST') '-std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -I/Users/sysadmin/build/v3.8.0/Include/internal' sysconfig.get_config_var('PY_LDFLAGS_NODIST') '' sysconfig.get_config_var('PY_LDFLAGS') '-arch x86_64 -g'
--
Inada Naoki
Thank you for your response.
And I'm sorry about ignoring this. Gmail marked it as spam.
On Tue, Oct 15, 2019 at 6:20 PM Ned Deily
We currently do not use those options to build the binaries for the python.org macOS installers. The main reason is that the Pythons we provide are built to support a wide-range of macOS releases and to do so safely we build the binaries on the oldest version of macOS supported by that installer. So, for example, the 10.9+ installer variant is built on a 10.9 system. Some of the optimization features either aren't available or are less robust on older build tools.
It makes sense.
And I believe it is more important for the python.org macOS installers to continue to provide a single installer that is usable on many systems and can be used in a broad range of applications and by a broad range of users rather than trying to optimize performance for a specific application: you can always build your own Python.
As far as what other distributors of Python for macOS do, what we do shouldn't necessarily constrain them. I don't see any problem with Homebrew optimizing for a particular user's installation. I see that MacPorts, another distributor of Python on macOS, provides a non-default variant that uses --enable-optimizations.
https://github.com/macports/macports-ports/blob/master/lang/python37/Portfil...
-- Ned Deily nad@python.org -- []
--
Inada Naoki
On 17 Oct 2019, at 08:13, Inada Naoki
wrote: Thank you for your response. And I'm sorry about ignoring this. Gmail marked it as spam.
On Tue, Oct 15, 2019 at 6:20 PM Ned Deily
wrote: We currently do not use those options to build the binaries for the python.org macOS installers. The main reason is that the Pythons we provide are built to support a wide-range of macOS releases and to do so safely we build the binaries on the oldest version of macOS supported by that installer. So, for example, the 10.9+ installer variant is built on a 10.9 system. Some of the optimization features either aren't available or are less robust on older build tools.
It makes sense.
Somewhat. I would be better to build with the latest compiler because that’s generally better than older compilers, but that requires some source changes and testing to ensure that the build still works on older versions of macOS. At this point in time nobody seems to have available time and need to work on this, including myself. Ronald
participants (4)
-
Inada Naoki
-
Ned Deily
-
Ronald Oussoren
-
Victor Stinner