[issue36044] PROFILE_TASK for PGO build is not a good workload

[Gregory P. Smith]

Gregory P. Smith <greg at krypto.org> added the comment:

In my experience, people overthink what needs to go into a CPython profiling run.  Sure, our default PROFILE_TASK is rather unfortunate because it takes a very long time by including runs of super slow tests that won't meaningfully contribute profile data (multiprocessing, subprocess, concurrent_futures, heavy I/O, etc).

But despite somewhat popular belief, it is not actually a problem that the suite exercises other sides of branches, because by and large just by executing Python code at all and exercising C extension module code, it still acts like most any Python program and spends most of the time on the common critical paths - regardless tests that trigger specific number of executions of other paths.  Those executions pale in comparison to the ordinary ones in anywhere critical.

I don't recommend making any claim about something "harming" the profile without reliable data to prove it.

Feel free to tune what test.regrtest --pgo enables or disables by default.  But try to do it in a scientific data based manner rather than guessing.  Decreasing the total wall time for a default --enable-optimizations build would be a good thing for everyone, provided the resulting interpreter remains "effectively similar" in speed.  If you somehow manage to find something that actually speeds up the resulting interpreter, amazing!

https://github.com/python/pyperformance is your friend for real world performance measurement.  Patience is key.  The builds and benchmark runs are slow.

One thing --enable-optimizations does _not_ do today is enable link time optimization by default.  Various toolchain+platform versions were having problems successfully generating a working interpreter with LTO enabled.  If you want a potential large speed up in the interpreter, figuring out how to get link time optimization on top of the existing PGO working reliably, detecting when toolchains+platform combos where it will be reliable, and enabling it by default on such systems is _likely_ the largest possible gain still to be had.

----------
components: +Build
nosy: +gregory.p.smith
versions: +Python 3.8, Python 3.9

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue36044>
_______________________________________