Mailman 3 February 2016 - Speed

Re: [Speed] Should we change what benchmarks we have?
by Victor Stinner 12 Feb '16

12 Feb '16

2016-02-11 19:36 GMT+01:00 Brett Cannon <brett(a)python.org>: > Are we happy with the current benchmarks? bm_regex8 looks unstable, but I don't know if it's an issue of the benchmark itself or perf.py (see the other thread "[Speed] Any changes we want to make to perf.py?"). I spent a lot of time (probably too much!) last months trying to micro-optimize some parts of Python, specially operations on Python int. See for example this long issue: https://bugs.python.org/issue21955 At the end, the discussed patched only makes two benchmarks faster: nbody & spectral_norm. I'm disappointed because I don't know if it's worth to take these micro-optimizations only to run two *benchmarks* faster. Are they representative of "regular" Python code and "real-world applications"? Or are they typical maths benchmark? For maths, we all know that pure Python sucks and that maybe better options are available: PyPy, numba, Cython, etc. For example, PyPy is around 10x faster, whereas discussed micro-optimizations are 1.18x faster in the best case (in one very specific micro-benchmark). > Are there some we want to drop? > How about add? Do we want to have explanations as to why each benchmark is > included? A better balance of micro vs. macro benchmarks (and probably > matching groups)? For some kinds of optimizations, I consider that a micro-benchmark is enough. I don't have strict rules. Basically, it's when you know that the change cannot introduce slow-down in other cases, but will only benefit on one specific case. So the best is to write a tiny benchmark just for this case. Victor

1 0

Re: [Speed] Should we change what benchmarks we have?
by Kevin Modzelewski 12 Feb '16

12 Feb '16

We on the Pyston team have created some new benchmarks which I can recommend using; I wouldn't call them "macrobenchmarks" since they don't test entire applications, but we've found them to be better than the existing benchmarks, which tend to be quite microbenchmarky. For example, our django-templating benchmark actually exercises the django templating system, as opposed to bm_django.py which just tests unicode concatenation. You can find them here https://github.com/dropbox/pyston-perf/tree/master/benchmarking/benchmark_s… The current ones we look at are django_template3_10x, sqlalchemy_imperative2_10x, and pyxl_bench_10x. On Thu, Feb 11, 2016 at 10:36 AM, Brett Cannon <brett(a)python.org> wrote: > Are we happy with the current benchmarks? Are there some we want to drop? > How about add? Do we want to have explanations as to why each benchmark is > included? A better balance of micro vs. macro benchmarks (and probably > matching groups)? > > _______________________________________________ > Speed mailing list > Speed(a)python.org > https://mail.python.org/mailman/listinfo/speed > >

1 0

Re: [Speed] Any changes we want to make to perf.py?
by Victor Stinner 12 Feb '16

12 Feb '16

I don't think that using a fixed number of iterations is good to get stable benchmark results. I opened the following issue to discussed that: https://bugs.python.org/issue26275 I proposed to calibrate the number of runs and the number of loops using time. I'm not convinced myself yet that it's a good idea. For "runs" and "loops", I'm talking about something like that: times = [] for run in range(runs): dt = time.perf_counter() for loop in loops: func() # or python instructions timed.append(dt - time.perf_counter()) Victor 2016-02-11 19:31 GMT+01:00 Brett Cannon <brett(a)python.org>: > Some people have brought up the idea of tweaking how perf.py drives the > benchmarks. I personally wonder if we should go from a elapsed time > measurement to # of executions in a set amount of time measurement to get a > more stable number that's easier to measure and will make sense even as > Python and computers get faster (I got this idea from Mozilla's Dromaeo > benchmark suite: https://wiki.mozilla.org/Dromaeo). > > _______________________________________________ > Speed mailing list > Speed(a)python.org > https://mail.python.org/mailman/listinfo/speed >

1 0

Should we change what benchmarks we have?
by Brett Cannon 12 Feb '16

12 Feb '16

Are we happy with the current benchmarks? Are there some we want to drop? How about add? Do we want to have explanations as to why each benchmark is included? A better balance of micro vs. macro benchmarks (and probably matching groups)?

1 0

Do we want to stop vendoring source of third-party libraries with the benchmarks?
by Brett Cannon 12 Feb '16

12 Feb '16

Maybe we should just have a requirements.txt file for Python 2 and another for Python 3 that are pegged to specific versions? We could even install things into a venv for isolation. If we go this route then we could make the benchmark suite a package on PyPI and have people install the benchmark suite and then have instructions to run pip on the requirements files that we embed in the package. This also gets us around any potential licensing issues with embedding third-party libraries.

1 0

Any changes we want to make to perf.py?
by Brett Cannon 12 Feb '16

12 Feb '16

Some people have brought up the idea of tweaking how perf.py drives the benchmarks. I personally wonder if we should go from a elapsed time measurement to # of executions in a set amount of time measurement to get a more stable number that's easier to measure and will make sense even as Python and computers get faster (I got this idea from Mozilla's Dromaeo benchmark suite: https://wiki.mozilla.org/Dromaeo).

1 0

Re: [Speed] [Python-Dev] speed.python.org
by Gregory P. Smith 07 Feb '16

07 Feb '16

Displaying ratios linearly rather than on a log scale axis can be misleading depending on what you are looking for. (feature request: allow a log scale?) major kudos to everyone involved in getting this setup! On Fri, Feb 5, 2016 at 11:06 PM Nick Coghlan <ncoghlan(a)gmail.com> wrote: > On 6 February 2016 at 04:07, Brett Cannon <brett(a)python.org> wrote: > > On Thu, 4 Feb 2016 at 05:46 Nick Coghlan <ncoghlan(a)gmail.com> wrote: > >> Heh, cdecimal utterly demolishing the old pure Python decimal module > >> on the telco benchmark means normalising against CPython 3.5 rather > >> than 2.7 really isn't very readable :) > > > > I find viewing the graphs using the horizontal layout is much easier to > read > > (the bars are a lot thicker and everything zooms in more). > > That comment was based on the horizontal layout - the telco benchmark > runs ~53x faster in Python 3 than it does in Python 2 (without > switching to cdecimal), so you end up with all the other benchmarks > being squashed into the leftmost couple of grid cells. > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia > _______________________________________________ > Python-Dev mailing list > Python-Dev(a)python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/greg%40krypto.org >

1 0

Re: [Speed] [Python-Dev] speed.python.org
by Nick Coghlan 06 Feb '16

06 Feb '16

On 6 February 2016 at 04:07, Brett Cannon <brett(a)python.org> wrote: > On Thu, 4 Feb 2016 at 05:46 Nick Coghlan <ncoghlan(a)gmail.com> wrote: >> Heh, cdecimal utterly demolishing the old pure Python decimal module >> on the telco benchmark means normalising against CPython 3.5 rather >> than 2.7 really isn't very readable :) > > I find viewing the graphs using the horizontal layout is much easier to read > (the bars are a lot thicker and everything zooms in more). That comment was based on the horizontal layout - the telco benchmark runs ~53x faster in Python 3 than it does in Python 2 (without switching to cdecimal), so you end up with all the other benchmarks being squashed into the leftmost couple of grid cells. Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

1 0

Re: [Speed] [Python-Dev] speed.python.org
by Brett Cannon 06 Feb '16

06 Feb '16

To piggyback on Zach's speed.python.org announcement, we will most likely be kicking off a discussion of redoing the benchmark suite, tweaking the test runner, etc. over on the speed@ ML. Those of us who have been doing perf work lately have found some shortcoming we would like to fix in our benchmarks suite, so if you want to participate in that discussion, please join speed@ by next week. On Wed, 3 Feb 2016 at 22:49 Zachary Ware <zachary.ware+pydev(a)gmail.com> wrote: > I'm happy to announce that speed.python.org is finally functional! > There's not much there yet, as each benchmark builder has only sent > one result so far (and one of those involved a bit of cheating on my > part), but it's there. > > There are likely to be rough edges that still need smoothing out. > When you find them, please report them at > https://github.com/zware/codespeed/issues or on the speed(a)python.org > mailing list. > > Many thanks to Intel for funding the work to get it set up and to > Brett Cannon and Benjamin Peterson for their reviews. > > Happy benchmarking, > -- > Zach > _______________________________________________ > Python-Dev mailing list > Python-Dev(a)python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org >

1 0

Re: [Speed] [Python-Dev] speed.python.org
by Brett Cannon 06 Feb '16

06 Feb '16

On Thu, 4 Feb 2016 at 05:46 Nick Coghlan <ncoghlan(a)gmail.com> wrote: > On 4 February 2016 at 16:48, Zachary Ware <zachary.ware+pydev(a)gmail.com> > wrote: > > I'm happy to announce that speed.python.org is finally functional! > > There's not much there yet, as each benchmark builder has only sent > > one result so far (and one of those involved a bit of cheating on my > > part), but it's there. > > > > There are likely to be rough edges that still need smoothing out. > > When you find them, please report them at > > https://github.com/zware/codespeed/issues or on the speed(a)python.org > > mailing list. > > > > Many thanks to Intel for funding the work to get it set up and to > > Brett Cannon and Benjamin Peterson for their reviews. > > Heh, cdecimal utterly demolishing the old pure Python decimal module > on the telco benchmark means normalising against CPython 3.5 rather > than 2.7 really isn't very readable :) > I find viewing the graphs using the horizontal layout is much easier to read (the bars are a lot thicker and everything zooms in more).

1 0