Mailman 3 November 2016 - Speed

Re: [Speed] Latest enhancements of perf 0.8.1 and performance 0.3.1
by Armin Rigo Nov. 2, 2016

Nov. 2, 2016

Hi Victor, On 2 November 2016 at 16:53, Victor Stinner <victor.stinner(a)gmail.com> wrote: > 2016-11-02 15:20 GMT+01:00 Armin Rigo <armin.rigo(a)gmail.com>: >> Is that really the kind of examples you want to put forward? > > I am not a big fan of timeit, but we must use it sometimes to > micro-optimizations in CPython to check if an optimize really makes > CPython faster or not. I am only trying to enhance timeit. > Understanding results require to understand how the statements are > executed. Don't get me wrong, I understand the point of the following usage of timeit: python2 -m perf timeit '[1,2]*1000' --duplicate=1000 What I'm criticizing here is this instead: python2 -m perf timeit '[1,2]*1000' --duplicate=1000 --compare-to=pypy because you're very unlikely to get any relevant information from such a comparison. I stand by my original remark: I would say it should be an error or at least a big fat warning to use --duplicate and PyPy in the same invocation. This is as opposed to silently ignoring --duplicate for PyPy, which is just adding more confusion imho. A bientôt, Armin.

1 0

Re: [Speed] Latest enhancements of perf 0.8.1 and performance 0.3.1
by Victor Stinner Nov. 2, 2016

Nov. 2, 2016

2016-11-02 15:20 GMT+01:00 Armin Rigo <armin.rigo(a)gmail.com>: > Is that really the kind of examples you want to put forward? I am not a big fan of timeit, but we must use it sometimes to micro-optimizations in CPython to check if an optimize really makes CPython faster or not. I am only trying to enhance timeit. Understanding results require to understand how the statements are executed. > This example means "compare CPython where the data cache gets extra pressure from reading a strangely large code object, I wrote --duplicate option to benchmark "x+y" with "x=1; y=2". I know, it's an extreme and stupid benchmark, but many people spend a lot of time on trying to optimize this in Python/ceval.c: https://bugs.python.org/issue21955 I tried multiple values of --duplicate when benchmarking x+y, and x+y seems "faster" when using a larger --duplicate value. I understand that the cost of the outer loop is higher than the cost of "reading a strangely large code object". I provide a tool and I try to document how to use it. But it's hard to prevent users to use it for stupid things. For example, recently I spent time trying to optimize bytes%args in Python 3 after reading an article, but then I realized that the Python 2 benchmark was meaningless: https://atleastfornow.net/blog/not-all-bytes/ def bytes_plus(): b"hi" + b" " + b"there" ... benchmark(bytes_plus) ... bytes_plus() is optimized by the _compiler_, so the benchmark measure the cost of LOAD_CONST :-) The issue was not the tool but the usage of the tool :-D Victor

1 0

Re: [Speed] Latest enhancements of perf 0.8.1 and performance 0.3.1
by Armin Rigo Nov. 2, 2016

Nov. 2, 2016

Hi Victor, On 2 November 2016 at 13:00, Victor Stinner <victor.stinner(a)gmail.com> wrote: > On CPython, --duplicate makes sense (no?). So for example, the > following command should use duplicate on CPython but not on PyPy: > > python2 -m perf timeit '[1,2]*1000' --duplicate=1000 --compare-to=pypy This example means "compare CPython where the data cache gets extra pressure from reading a strangely large code object, and PyPy where the multiplication might be entirely removed for all I know". Is that really the kind of examples you want to put forward? A bientôt, Armin.

1 0

Re: [Speed] Latest enhancements of perf 0.8.1 and performance 0.3.1
by Victor Stinner Nov. 2, 2016

Nov. 2, 2016

Hum, so for an usability point of view, I think that the best to do is to ignore the option if Python has a JIT. On CPython, --duplicate makes sense (no?). So for example, the following command should use duplicate on CPython but not on PyPy: python2 -m perf timeit '[1,2]*1000' --duplicate=1000 --compare-to=pypy Victor

1 0

Re: [Speed] Latest enhancements of perf 0.8.1 and performance 0.3.1
by Armin Rigo Nov. 2, 2016

Nov. 2, 2016

Hi Victor, On 2 November 2016 at 11:50, Victor Stinner <victor.stinner(a)gmail.com> wrote: > 2016-11-02 11:04 GMT+01:00 Armin Rigo <armin.rigo(a)gmail.com>: >> On 19 October 2016 at 18:55, Victor Stinner <victor.stinner(a)gmail.com> wrote: >>> 3) new --duplication option to perf timeit >> >> This is never a good idea on top of PyPy, so I wouldn't mind if using >> this option on top of PyPy threw an error. > > Can you please elaborate? Yes, exactly :-) Consider a benchmark written like that: for i in range(lots): z = a + b z = a + b z = a + b z = a + b z = a + b What you are really measuring by running PyPy on this is completely different from what you *think* you are measuring---in this case, mostly everything is optimized away. If you try to make it actually do something so that it's not optimized away, then the problem of duplicating lines becomes of making the tracing JIT compiler not happy at all. If you duplicate the lines too many times, the loop body becomes too long for the JIT compiler to swallow---never duplicates 1000 times, that's always too much! But even if you duplicate only 10 times, then the more subtle problem is: assume that each line can follow *two* control flow paths (even internally, e.g. because of some condition done in RPython). (It is likely the case, if you try to do something non-trivially-optimizable-away.) Then if you duplicate the line 10 times, there are suddenly 2**10 control flow paths. That means the JIT will never be able to warm up completely. Suddenly you are measuring the JIT compiler's performance and not at all your code's. The --duplication option on PyPy is thus either useless or limited to use cases where you definitely know there is only one code path ever followed, and don't duplicate too much, and know for sure that multiple repetitions of the same line won't cause cross-line optimizations. That's not possible to explain without going very technical. A bientôt, Armin.

1 0

Re: [Speed] Latest enhancements of perf 0.8.1 and performance 0.3.1
by Victor Stinner Nov. 2, 2016

Nov. 2, 2016

2016-11-02 11:04 GMT+01:00 Armin Rigo <armin.rigo(a)gmail.com>: > On 19 October 2016 at 18:55, Victor Stinner <victor.stinner(a)gmail.com> wrote: >> 3) new --duplication option to perf timeit > > This is never a good idea on top of PyPy, so I wouldn't mind if using > this option on top of PyPy threw an error. Can you please elaborate? Victor

1 0

Re: [Speed] Latest enhancements of perf 0.8.1 and performance 0.3.1
by Armin Rigo Nov. 2, 2016

Nov. 2, 2016

Hi Victor, On 19 October 2016 at 18:55, Victor Stinner <victor.stinner(a)gmail.com> wrote: > 3) new --duplication option to perf timeit This is never a good idea on top of PyPy, so I wouldn't mind if using this option on top of PyPy threw an error. A bientôt, Armin.

1 0

Re: [Speed] [Python-Dev] Benchmarking Python and micro-optimizations
by Ludovic Gasc Nov. 1, 2016

Nov. 1, 2016

Hi, Thanks first for that, it's very interesting. About to enrich benchmark suite, I might have a suggestion: We might add REST/JSON scenarios, because a lot of people use Python for that. It isn't certainly not the best REST/JSON scenarios, because they have a small payload, but better than nothing: https://www.techempower.com/benchmarks/#section=code&hw=peak&test=fortune Moreover, we already have several implementations for the most populars Web frameworks: https://github.com/TechEmpower/FrameworkBenchmarks/tree/master/frameworks/P… The drawback is that a lot of tests need a database. I can help if you're interested in. Have a nice week.

1 0