Would it be possible rerun the tests with the current setup for say the last 1000 revisions or perhaps a subset of these (e.g. every 10th revision) to try to binary search for the revision which introduced the change ?
Every run takes 1-2 h so doing 1000 would be certainly time-consuming :)
That's why from now on I am trying to invest in daily builds for master, so we can answer that exact question if we detect regressions in the future.
On Wed, 14 Oct 2020 at 15:04, M.-A. Lemburg <mal@egenix.com> wrote:
Would it be possible to get the data for older runs back, so that it's easier to find the changes which caused the slowdown ?
Unfortunately no. The reasons are that that data was misleading because different points were computed with a different version of pyperformance and therefore with different packages (and therefore different code). So the
On 14.10.2020 16:00, Pablo Galindo Salgado wrote: points
could not be compared among themselves.
Also, past data didn't include 3.9 commits because the data gathering was not automated and it didn't run in a long time :(
Make sense.
Would it be possible rerun the tests with the current setup for say the last 1000 revisions or perhaps a subset of these (e.g. every 10th revision) to try to binary search for the revision which introduced the change ?
On Wed, 14 Oct 2020 at 14:57, M.-A. Lemburg <mal@egenix.com <mailto:mal@egenix.com>> wrote:
Hi Pablo, thanks for pointing this out. Would it be possible to get the data for older runs back, so that it's easier to find the changes which caused the slowdown ? Going to the timeline, it seems that the system only has data for Oct 14 (today):In addition to unpack_sequence, the regex_dna test has slowed down a lot compared to Py3.8.https://github.com/python/pyperformance/blob/master/pyperformance/benchmarks...
https://github.com/python/pyperformance/blob/master/pyperformance/benchmarks...
Thanks. On 14.10.2020 15:16, Pablo Galindo Salgado wrote: > Hi! > > I have updated the branch benchmarks in the pyperformance serverand now they
> include 3.9. There are > some benchmarks that are faster but on the other hand somebenchmarks are
> substantially slower, pointing > at a possible performance regression in 3.9 in some aspects. Inparticular
some > tests like "unpack sequence" are > almost 20% slower. As there are some other tests were 3.9 isfaster, is
not fair > to conclude that 3.9 is slower, but > this is something we should look into in my opinion. > > You can check these benchmarks I am talking about by: > > * Go here: https://speed.python.org/comparison/ > * In the left bar, select "lto-pgo latest in branch '3.9'" and"lto-pgo latest
> in branch '3.8'" > * To better read the plot, I would recommend to select a"Normalization"
to the > 3.8 branch (this is in the top part of the page) > and to check the "horizontal" checkbox. > > These benchmarks are very stable: I have executed them severaltimes over the
> weekend yielding the same results and, > more importantly, they are being executed on a server speciallyprepared to
> running reproducible benchmarks: CPU affinity, > CPU isolation, CPU pinning for NUMA nodes, CPU frequency is fixed,CPU
governor > set to performance mode, IRQ affinity is > disabled for the benchmarking CPU nodes...etc so you can trustthese numbers.
> > I kindly suggest for everyone interested in trying to improve the3.9 (and
> master) performance, to review these benchmarks > and try to identify the problems and fix them or to find whatchanges
introduced > the regressions in the first place. All benchmarks > are the ones being executed by the pyperformance suite > (https://github.com/python/pyperformance) so you can execute them > locally if you need to. > > --- > > On a related note, I am also working on the speed.python.org <http://speed.python.org> > <http://speed.python.org> server to provide more automation and > ideally some integrations with GitHub to detect performanceregressions. For
> now, I have done the following: > > * Recompute benchmarks for all branches using the same version of > pyperformance (except master) so they can > be compared with each other. This can only be seen in the"Comparison"
> tab: https://speed.python.org/comparison/ > * I am setting daily builds of the master branch so we can detectperformance
> regressions with daily granularity. These > daily builds will be located in the "Changes" and "Timeline"tabs
> (https://speed.python.org/timeline/). > * Once the daily builds are working as expected, I plan to work ontrying to
> automatically comment or PRs or on bpo if > we detect that a commit has introduced some notable performanceregression.
> > Regards from sunny London, > Pablo Galindo Salgado. > > _______________________________________________ > python-committers mailing list -- python-committers@python.org <mailto:python-committers@python.org> > To unsubscribe send an email to python-committers-leave@python.org <mailto:python-committers-leave@python.org> >https://mail.python.org/mailman3/lists/python-committers.python.org/
> Message archived athttps://mail.python.org/archives/list/python-committers@python.org/message/G...
> Code of Conduct: https://www.python.org/psf/codeofconduct/ > -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Oct 14
>>> Python Projects, Coaching and Support ...>>> Python Product Development ...https://consulting.egenix.com/
::: We implement business ideas - efficiently in both time and costs:::
eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/-- Marc-Andre Lemburg eGenix.com
Professional Python Services directly from the Experts (#1, Oct 14 2020)
Python Projects, Coaching and Support ... https://www.egenix.com/ Python Product Development ... https://consulting.egenix.com/
::: We implement business ideas - efficiently in both time and costs :::
eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/