Speed November 2016

speed@python.org

10 participants
28 discussions

Re: [Speed] Latest enhancements of perf 0.8.1 and performance 0.3.1

by Armin Rigo

Hi Victor, On 2 November 2016 at 16:53, Victor Stinner <victor.stinner(a)gmail.com> wrote: > 2016-11-02 15:20 GMT+01:00 Armin Rigo <armin.rigo(a)gmail.com>: >> Is that really the kind of examples you want to put forward? > > I am not a big fan of timeit, but we must use it sometimes to > micro-optimizations in CPython to check if an optimize really makes > CPython faster or not. I am only trying to enhance timeit. > Understanding results require to understand how the statements are > executed. Don't get me wrong, I understand the point of the following usage of timeit: python2 -m perf timeit '[1,2]*1000' --duplicate=1000 What I'm criticizing here is this instead: python2 -m perf timeit '[1,2]*1000' --duplicate=1000 --compare-to=pypy because you're very unlikely to get any relevant information from such a comparison. I stand by my original remark: I would say it should be an error or at least a big fat warning to use --duplicate and PyPy in the same invocation. This is as opposed to silently ignoring --duplicate for PyPy, which is just adding more confusion imho. A bientôt, Armin.

3 years, 9 months

Re: [Speed] Latest enhancements of perf 0.8.1 and performance 0.3.1

by Victor Stinner

2016-11-02 15:20 GMT+01:00 Armin Rigo <armin.rigo(a)gmail.com>: > Is that really the kind of examples you want to put forward? I am not a big fan of timeit, but we must use it sometimes to micro-optimizations in CPython to check if an optimize really makes CPython faster or not. I am only trying to enhance timeit. Understanding results require to understand how the statements are executed. > This example means "compare CPython where the data cache gets extra pressure from reading a strangely large code object, I wrote --duplicate option to benchmark "x+y" with "x=1; y=2". I know, it's an extreme and stupid benchmark, but many people spend a lot of time on trying to optimize this in Python/ceval.c: https://bugs.python.org/issue21955 I tried multiple values of --duplicate when benchmarking x+y, and x+y seems "faster" when using a larger --duplicate value. I understand that the cost of the outer loop is higher than the cost of "reading a strangely large code object". I provide a tool and I try to document how to use it. But it's hard to prevent users to use it for stupid things. For example, recently I spent time trying to optimize bytes%args in Python 3 after reading an article, but then I realized that the Python 2 benchmark was meaningless: https://atleastfornow.net/blog/not-all-bytes/ def bytes_plus(): b"hi" + b" " + b"there" ... benchmark(bytes_plus) ... bytes_plus() is optimized by the _compiler_, so the benchmark measure the cost of LOAD_CONST :-) The issue was not the tool but the usage of the tool :-D Victor

3 years, 9 months

Re: [Speed] Latest enhancements of perf 0.8.1 and performance 0.3.1

by Armin Rigo

Hi Victor, On 2 November 2016 at 13:00, Victor Stinner <victor.stinner(a)gmail.com> wrote: > On CPython, --duplicate makes sense (no?). So for example, the > following command should use duplicate on CPython but not on PyPy: > > python2 -m perf timeit '[1,2]*1000' --duplicate=1000 --compare-to=pypy This example means "compare CPython where the data cache gets extra pressure from reading a strangely large code object, and PyPy where the multiplication might be entirely removed for all I know". Is that really the kind of examples you want to put forward? A bientôt, Armin.

3 years, 9 months

Re: [Speed] Latest enhancements of perf 0.8.1 and performance 0.3.1

by Victor Stinner

Hum, so for an usability point of view, I think that the best to do is to ignore the option if Python has a JIT. On CPython, --duplicate makes sense (no?). So for example, the following command should use duplicate on CPython but not on PyPy: python2 -m perf timeit '[1,2]*1000' --duplicate=1000 --compare-to=pypy Victor

3 years, 9 months

Re: [Speed] Latest enhancements of perf 0.8.1 and performance 0.3.1

by Armin Rigo

Hi Victor, On 2 November 2016 at 11:50, Victor Stinner <victor.stinner(a)gmail.com> wrote: > 2016-11-02 11:04 GMT+01:00 Armin Rigo <armin.rigo(a)gmail.com>: >> On 19 October 2016 at 18:55, Victor Stinner <victor.stinner(a)gmail.com> wrote: >>> 3) new --duplication option to perf timeit >> >> This is never a good idea on top of PyPy, so I wouldn't mind if using >> this option on top of PyPy threw an error. > > Can you please elaborate? Yes, exactly :-) Consider a benchmark written like that: for i in range(lots): z = a + b z = a + b z = a + b z = a + b z = a + b What you are really measuring by running PyPy on this is completely different from what you *think* you are measuring---in this case, mostly everything is optimized away. If you try to make it actually do something so that it's not optimized away, then the problem of duplicating lines becomes of making the tracing JIT compiler not happy at all. If you duplicate the lines too many times, the loop body becomes too long for the JIT compiler to swallow---never duplicates 1000 times, that's always too much! But even if you duplicate only 10 times, then the more subtle problem is: assume that each line can follow *two* control flow paths (even internally, e.g. because of some condition done in RPython). (It is likely the case, if you try to do something non-trivially-optimizable-away.) Then if you duplicate the line 10 times, there are suddenly 2**10 control flow paths. That means the JIT will never be able to warm up completely. Suddenly you are measuring the JIT compiler's performance and not at all your code's. The --duplication option on PyPy is thus either useless or limited to use cases where you definitely know there is only one code path ever followed, and don't duplicate too much, and know for sure that multiple repetitions of the same line won't cause cross-line optimizations. That's not possible to explain without going very technical. A bientôt, Armin.

3 years, 9 months

Re: [Speed] Latest enhancements of perf 0.8.1 and performance 0.3.1

by Victor Stinner

2016-11-02 11:04 GMT+01:00 Armin Rigo <armin.rigo(a)gmail.com>: > On 19 October 2016 at 18:55, Victor Stinner <victor.stinner(a)gmail.com> wrote: >> 3) new --duplication option to perf timeit > > This is never a good idea on top of PyPy, so I wouldn't mind if using > this option on top of PyPy threw an error. Can you please elaborate? Victor

3 years, 9 months

Re: [Speed] Latest enhancements of perf 0.8.1 and performance 0.3.1

by Armin Rigo

Hi Victor, On 19 October 2016 at 18:55, Victor Stinner <victor.stinner(a)gmail.com> wrote: > 3) new --duplication option to perf timeit This is never a good idea on top of PyPy, so I wouldn't mind if using this option on top of PyPy threw an error. A bientôt, Armin.

3 years, 9 months

Re: [Speed] [Python-Dev] Benchmarking Python and micro-optimizations

by Ludovic Gasc

Hi, Thanks first for that, it's very interesting. About to enrich benchmark suite, I might have a suggestion: We might add REST/JSON scenarios, because a lot of people use Python for that. It isn't certainly not the best REST/JSON scenarios, because they have a small payload, but better than nothing: https://www.techempower.com/benchmarks/#section=code&hw=peak&test=fortune Moreover, we already have several implementations for the most populars Web frameworks: https://github.com/TechEmpower/FrameworkBenchmarks/tree/master/frameworks/P… The drawback is that a lot of tests need a database. I can help if you're interested in. Have a nice week.

3 years, 9 months

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

Speed November 2016