Speed January 2018

speed@python.org

10 participants
21 discussions

Re: [Speed] adding a pypy runner for speed.python.org

by Miquel Torres

Hi Matti, I'll gladly help setting up the speed site El El mié, 31 ene 2018 a las 13:47, Victor Stinner <victor.stinner(a)gmail.com> escribió: > Hi, > > I tried but failed to find someone in PyPy to adjust performance > benchmarks for PyPy. Currently, the JIT is not properly warmed up, and > the results can be dishonnest or not reliable. > > My latest attempt to support is PyPy is: > http://vstinner.readthedocs.io/pypy_warmups.html > > IMHO we need to develop a statistical methodology in perf to compute > when values become stable. That's hard to define and it was proven > that "performance stability" doesn't exist (see " Virtual Machine > Warmup Blows Hot and Cold" paper). > > Fijal from PyPy would like to use hardcoded configuration for the > number of warmup values. I like the idea, but nobody implemented it > yet. > > Victor > > 2018-01-30 21:37 GMT+01:00 Matti Picus <matti.picus(a)gmail.com>: > > I am cross-posting to both speed and pypy-dev to ask what needs to be > done > > to get pypy2 and pypy3 benchmark runners onto speed.python.org > > I am willing to be the contact person from the PyPy side, who do we need > to > > talk to on the speed maintainers' side? > > Instead of spamming the lists again, we could discuss this off-line, and > > report back with a plan, required resources, and a timeline. > > > > Thanks, > > Matti Picus > > _______________________________________________ > > Speed mailing list > > Speed(a)python.org > > https://mail.python.org/mailman/listinfo/speed > _______________________________________________ > Speed mailing list > Speed(a)python.org > https://mail.python.org/mailman/listinfo/speed >

2 years, 10 months

adding a pypy runner for speed.python.org

by Matti Picus

I am cross-posting to both speed and pypy-dev to ask what needs to be done to get pypy2 and pypy3 benchmark runners onto speed.python.org I am willing to be the contact person from the PyPy side, who do we need to talk to on the speed maintainers' side? Instead of spamming the lists again, we could discuss this off-line, and report back with a plan, required resources, and a timeline. Thanks, Matti Picus

2 years, 10 months

Re: [Speed] Update Django from 1.11 to 2.0? What about Python 2.7 and PyPy2?

by Nick Coghlan

On 19 January 2018 at 20:39, Victor Stinner <victor.stinner(a)gmail.com> wrote: > 2018-01-19 11:28 GMT+01:00 Stefan Behnel <stefan_ml(a)behnel.de>: > > That suggests adding Django 2 as a new Py3-only benchmark. > > Again, the practical issue is to install Django 2 and Django 1.11 in > the same virtual environment. I'm not sure that it's doable. > > I would prefer to not have to create a different virtualenv for > Python3-only dependencies. > > I needed to release quickly a bugfix release, fix --track-memory, > feature asked by Xiang Zhang, so I released performance 0.6.1 which > only updated Django from 1.11.3 to 1.11.9. > > Or we need to redesign how performance install dependencies, but > that's a larger project :-) > It may be worth looking at using pew to set up a separate virtual environment for each benchmark, but then use `pew add` to share common components (like perf itself) between them. That way you won't have conflicting dependencies between benchmarks (since they'll be in separate venvs), without having to duplicate *all* the common components. Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

2 years, 10 months

Re: [Speed] Update Django from 1.11 to 2.0? What about Python 2.7 and PyPy2?

by Stefan Behnel

INADA Naoki schrieb am 16.01.2018 um 12:37: >> Even though it's not good for comparing interpreter performance, it's good >> for people comparing Python 2 and 3. >> >> If Django 2.0 on Python 3.7 is much faster than Django 1.11 on Python 2.7, >> it's nice carrot for people moving forward. >> > > FYI, Django 2 is about 2x faster than 1.11 on django_template benchmark! > It's because Django 1.11 calls force_text() many times for Python 2 > compatibility. > > https://github.com/django/django/blob/419705bbe84e27c3d5be85f198a0352a67249… > https://github.com/django/django/blob/419705bbe84e27c3d5be85f198a0352a67249… > > Actually, dropping Python 2 support makes Django faster. It's nice news! That suggests adding Django 2 as a new Py3-only benchmark. Stefan

2 years, 10 months

Re: [Speed] Update Django from 1.11 to 2.0? What about Python 2.7 and PyPy2?

by INADA Naoki

> Even though it's not good for comparing interpreter performance, it's good > for people comparing Python 2 and 3. > > If Django 2.0 on Python 3.7 is much faster than Django 1.11 on Python 2.7, > it's nice carrot for people moving forward. > FYI, Django 2 is about 2x faster than 1.11 on django_template benchmark! It's because Django 1.11 calls force_text() many times for Python 2 compatibility. https://github.com/django/django/blob/419705bbe84e27c3d5be85f198a0352a67249… https://github.com/django/django/blob/419705bbe84e27c3d5be85f198a0352a67249… Actually, dropping Python 2 support makes Django faster. It's nice news! Bests,

2 years, 10 months

Re: [Speed] Impact of Meltdown/Spectre OS patches on benchmark results?

by Antoine Pitrou

On Thu, 11 Jan 2018 10:36:16 +1000 Nick Coghlan <ncoghlan(a)gmail.com> wrote: > Hi folks, > > Reading https://medium.com/implodinggradients/meltdown-c24a9d5e254e > prompts me to ask: are speed.python.org benchmark results produced now > actually going to be comparable with those executed last year? > > Or will the old results need to be backfilled again with the new > baseline OS performance? AFAIU the Meltdown patches will only significantly affect workloads that do a lot of system calls. *Ideally* our benchmark suite doesn't, since it's meant to benchmark Python, not the outside environment :-) Spectre seems to be more uncertain, as it can affect any code (userspace code as well as kernel code), and will apparently require careful auditing of sensitive routines to avoid potential information leaks. This all depends which particular routines (e.g. something in glibc) get patched. In any case we can only tell with actual numbers :-) Regards Antoine.

2 years, 10 months

performance 0.6.1 released

by Victor Stinner

Hi, I'm happy to announce the release of performance 0.6.1: http://pyperformance.readthedocs.io/ Changes: * Fix inherit-environ: propagate to recursive invocations of ``performance`` in ``compile`` and ``compile_all`` commands. * Fix the ``--track-memory`` option thanks to the update to perf 1.5. * Update requirements - certifi: 2017.4.17 => 2017.11.5 - Chameleon: 3.1 => 3.2 - Django: 1.11.3 => 1.11.9 - docutils: 0.13.1 => 0.14 - dulwich: 0.17.3 => 0.18.6 - html5lib: 0.999999999 => 1.0.1 - Mako: 1.0.6 => 1.0.7 - mercurial: 4.2.2 => 4.4.2 - mpmath: 0.19 => 1.0.0 - perf: 1.4 => 1.5.1 (fix ``--track-memory`` option) - psutil: 5.2.2 => 5.4.3 - pyaes: 1.6.0 => 1.6.1 - six: 1.10.0 => 1.11.0 - SQLAlchemy: 1.1.11 => 1.2.0 - sympy: 1.0 => 1.1.1 - tornado: 4.5.1 => 4.5.3 Note: Django wasn't upgraded to 2.0 to keep Python 2 compatibility. Victor

2 years, 10 months

Re: [Speed] Impact of Meltdown/Spectre OS patches on benchmark results?

by Paul Graydon

That's not a great comparison there. They've gone from a 2+ year old, somewhat customised kernel, and then compared it to a bleeding edge vanilla upstream kernel. It's worth pointing out that Ubuntu has currently opted to only put in Meltdown patches with the kernel they released yesterday. Spectre stuff is still to come. (https://wiki.ubuntu.com/SecurityTeam/KnowledgeBase/SpectreAndMeltdown <https://wiki.ubuntu.com/SecurityTeam/KnowledgeBase/SpectreAndMeltdown>) so that benchmark probably won't reflect what anyone actually sees anyway. Impact is going to be very variable based on what is being benchmarked. I've been doing a whole bunch of benchmarking at work as part of some broad investigations of the impact for our customers, leveraging a few different resources. I haven't grabbed and run the speed.python.org <http://speed.python.org/> benchmarks but I may be able to spin up a bare metal CentOS instance tomorrow and run through the paces with a pre/post Spectre and Meltdown patched kernel, if it's straightforward to do. (are there any documents people could point me to with instructions?) I'd consider the patches here to largely invalidate any historical performance benchmarks for a comparison perspective. Even on a CPU with PCID capabilities there is definitely a hit on certain types of operations. We're also very much in the "patch to make it secure" phase, with an optimisation phase still to come, and it's not certain what Intel, AMD, and/or ARM are going to push out by way of updates that may mitigate the impact. There's going to be a lot of focus on improving the performance situation over the next while, so the odds are reasonable that full historical benchmarks will need to be run repeatedly, I guess? Paul On January 11, 2018 12:36:16 AM UTC, Nick Coghlan <ncoghlan(a)gmail.com> wrote: Hi folks, Reading https://medium.com/implodinggradients/meltdown-c24a9d5e254e <https://medium.com/implodinggradients/meltdown-c24a9d5e254e> prompts me to ask: are speed.python.org benchmark results produced now actually going to be comparable with those executed last year? Or will the old results need to be backfilled again with the new baseline OS performance? Cheers, Nick.

2 years, 10 months

Impact of Meltdown/Spectre OS patches on benchmark results?

by Nick Coghlan

Hi folks, Reading https://medium.com/implodinggradients/meltdown-c24a9d5e254e prompts me to ask: are speed.python.org benchmark results produced now actually going to be comparable with those executed last year? Or will the old results need to be backfilled again with the new baseline OS performance? Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

2 years, 10 months

Re: [Speed] Update Django from 1.11 to 2.0? What about Python 2.7 and PyPy2?

by Nick Coghlan

On 10 January 2018 at 02:57, Victor Stinner <victor.stinner(a)gmail.com> wrote: > I would prefer to not have to install Django 1.11 *and* Django 2.0 in > the same virtual environment (I'm not sure that it's technically > possible). It means that the Django 1.11 benchmark would be specific > to Python 2, whereas the Django 2.0 benchmark would be specific to > Python 3. So with an hypothetical future performance version, it would > become impossible to compare django template benchmark between Python > 2 and Python 3. I'm not sure that the Python 2 benchmark would be > useful. The point of the benchmark is to test non-trivial string manipulation, not to test the latest version of Django's template engine. Just lock the dependency at Django 1.11, and explain why. Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

2 years, 10 months

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

Speed January 2018