Mailman 3 June 2016 - Speed

Re: [Speed] Performance comparison of regular expression engines
by Serhiy Storchaka June 11, 2016

June 11, 2016

On 06.03.16 11:30, Maciej Fijalkowski wrote: > this is really difficult to read, can you tell me which column am I looking at? The first column is the searched pattern. The second column is the number of found matches (for control, it should be the same with all engines and versions). The third column, under the "re" header is the time in milliseconds. The column under the "str.find" header is the time of searching without using regular expressions. PyPy 2.2 usually is significantly faster than CPython 2.7, except searching plain string with regular expression. But thanks to Flexible String Representation searching plain string with and without regular expression is faster on CPython 3.6.

2 1

perf 0.3 released
by Victor Stinner June 10, 2016

June 10, 2016

Hi, I just released perf 0.3. Major changes: - "python -m perf" CLI now has 3 commands: show, compare and compare_to. Compare commands says if the difference is significant (I copied the code from perf.py) - TextRunner is now able to spawn child processes, parse command arguments and more - If TextRunner detects isolated CPUs, it automatically pins the CPUs of the worker processes to isolated CPUs - Add ``--json-file`` command line option - Add TextRunner.bench_sample_func() method: the sample function is responsible to measure the elapsed time, useful for microbenchmarks - Enhance a lot of the documentation Writing a benchmark now only takes one line: "perf.text_runner.TextRunner().bench_func(func)"! Full example: --- import time import perf.text_runner def func(): time.sleep(0.001) perf.text_runner.TextRunner().bench_func(func) --- I looked at PyPy benchmarks: https://bitbucket.org/pypy/benchmarks Results can also be serialized to JSON, but the serialization is only done at the end: the final result is serialized. It's not possible to save each run in a JSON file. Running multiple processes is not supported neither. With perf, the final JSON contains all data: all runs, all samples even warmup samples. perf now also collects metadata in each worker process. So it is more safer to compare runs since it's possible to manually check when and how the worker executed the benchmark. For example, the CPU affinity is now saved in metadata. For example, "python -m perf.timeit" now saves the setup and statements in metadata. With perf 0.3, TextRunner now also includes a builtin calibration to compute the number of outter loop iteartions: repeat each sample so it takes between 100 ms and 1 sec (min/max are configurable). Victor

1 0

perf 0.2 released, perf fork of CPython benchmark suite
by Victor Stinner June 7, 2016

June 7, 2016

Hi, I completed the API of my small perf module and released a version 0.2: https://perf.readthedocs.io/ It is supposed to provide the basic tools to collect samples, compute the average, display the result, etc. I started to work on JSON serialization to "easily" run multiple processes. The idea is also to be split the code to produce numbers and the code to display results. I expect that we can do better to display results. See for example speed.python.org and speed.pypy.org, it's nicer than perf.py text output ;-) I also started to hack CPython benchmark suite (benchmarks repository) to use my perf module: https://hg.python.org/sandbox/benchmarks_perf I should now stop NIH and see how to merge my work with PyPy fork of benchmarks ;-) FYI I started to write the perf module because I started to write an article about the impact of CPU speed on Python microbenchmarks, and I wanted to have a smart timeit running multiple processes. Since it was cool to work on such project, I started to hack benchmarks, but maybe I gone too far and I should look at PyPy's benchmark instead ;-) Victor

1 0

Re: [Speed] A new perf module: toolkit to write benchmarks
by Antoine Pitrou June 2, 2016

June 2, 2016

On Thu, 2 Jun 2016 03:19:32 +0200 Victor Stinner <victor.stinner(a)gmail.com> wrote: > I'm interested by the very basic perf.py internal text format: one > timing per line, that's all. But it's incomplete, the "loops" > informaiton is not stored. Maybe a binary format is better? I don't > know yet. Just use a simple JSON format. Regards Antoine.

2 1

Re: [Speed] A new perf module: toolkit to write benchmarks
by Armin Rigo June 2, 2016

June 2, 2016

Hi Victor, On 2 June 2016 at 10:58, Victor Stinner <victor.stinner(a)gmail.com> wrote: > Usually, timeit it used to compare two versions of Python. That's not the use case I'm focusing about here: timeit is also used by Mr. Random Programmer to tweak their Python code to improve the performance. (Often, it's the performance of non-representative microbenchmarks, but well, better have a tool that at least get some saner results than the current timeit.) A bientôt, Armin.

1 0

Re: [Speed] A new perf module: toolkit to write benchmarks
by Antoine Pitrou June 2, 2016

June 2, 2016

On Thu, 2 Jun 2016 10:58:35 +0200 Victor Stinner <victor.stinner(a)gmail.com> wrote: > > Usually, timeit it used to compare two versions of Python. timeit is used for many different things, including comparing two versions of Python, but not only. > For the CLI, we can extend timeit CLI to accept the path/name of two > python binaries. That sounds reasonable. Regards Antoine.

1 0

Re: [Speed] A new perf module: toolkit to write benchmarks
by Victor Stinner June 2, 2016

June 2, 2016

2016-06-02 10:38 GMT+02:00 Armin Rigo <arigo(a)tunes.org>: > Looks good. IMHO the important bit is that `timeit` is simple to use, > readily available, and gives just a number, which makes it very > attractive for people. By default min & max are hidden. You can show them using -v option. To make the output even simpler, maybe the standard deviation can be displayed "in english". Something like: * "Average: 250 ns +- 3 ns" => "Average: 250 ns (stable)", or just "Average: 250 ns" * "Average: 250 ns +- 120 ns" => "Average: 250 ns (not reliable, try again on an idle system)" Usually, timeit it used to compare two versions of Python. Maybe we should focus on this use case, and check if the difference is significant, as perf.py does? By default, perf.py does *not* display any number if the difference is not significant. I like this behaviour, even if it can be surprising for the first time. For the CLI, we can extend timeit CLI to accept the path/name of two python binaries. Or we can use something like pybench to store result into files and then load & compare two files. Victor

1 0

Re: [Speed] A new perf module: toolkit to write benchmarks
by Armin Rigo June 2, 2016

June 2, 2016

Hi Victor, On 2 June 2016 at 03:19, Victor Stinner <victor.stinner(a)gmail.com> wrote: > 5 timeit output ("1000000 loops, best of 3: ... per loop"): > > * 0.247 usec > * 0.252 usec > * 0.247 usec > * 0.251 usec > * 0.251 usec > > 5 perf.timeit outputs ("Average: 25 runs x 3 samples x 10^6 loops: ..."): > > * 250 ns +- 3 ns > * 250 ns +- 3 ns > * 251 ns +- 3 ns > * 251 ns +- 4 ns > * 251 ns +- 3 ns Looks good. IMHO the important bit is that `timeit` is simple to use, readily available, and gives just a number, which makes it very attractive for people. Your output would achieve the same result (with the `+-` added, which is fine) assuming that it eventually replaces `timeit` in the standard library. I know there are many good reasons for why getting just a single number is not enough, but I'd say that we still need to achieve the best practical results given that constrain. The results you posted above seem to show that `perf.timeit` is better than `timeit` at doing that, and I believe that it's a great step forward. A bientôt, Armin.

1 0

A new perf module: toolkit to write benchmarks
by Victor Stinner June 2, 2016

June 2, 2016

Hi, I started to write blog posts on stable benchmarks: 1) https://haypo.github.io/journey-to-stable-benchmark-system.html 2) https://haypo.github.io/journey-to-stable-benchmark-deadcode.html 3) https://haypo.github.io/journey-to-stable-benchmark-average.html One important point is that minimum is commonly used in Python benchmarks, whereas it is a bad practice to get a stable benchmark. I started to work on a toolkit to write benchmarks, the new "perf" module: http://perf.readthedocs.io/en/latest/ https://github.com/haypo/perf I used timeit as a concrete use case, since timeit is popular and badly implemented. timeit currently uses 1 process running the microbenchmarks 3 times and take the minimum. timeit is *known* to be unstable, and the common advice is to run it at least 3 times and again take the minimum of the minimum. Example of links about timeit being unstable: * https://mail.python.org/pipermail/python-dev/2012-August/121379.html * https://bugs.python.org/issue23693 * https://bugs.python.org/issue6422 (not directly related) Moreover, the timeit module disables the garbage collector which is also wrong. It's wrong because it's rare to disable the GC in applications. My goal for the perf module is to provide basic features and then reuse it in existing benchmarks: * mean() and stdev() to display result * clock chosen for benchmark * result classes to store numbers * etc. Work in progress: * new implementation of timeit using multiple processes * perf.metadata module: collect various information about Python, the system, etc. * file format to store numbers and metadata I'm interested by the very basic perf.py internal text format: one timing per line, that's all. But it's incomplete, the "loops" informaiton is not stored. Maybe a binary format is better? I don't know yet. It should be possible to cumulate files of multiple processes. I'm also interested to implement a generic "rerun" command to add more samples if a benchmark doesn't look stable enough. perf.timeit looks more stable than timeit, the CLI is basically the same: replace "-m timeit" with "-m perf.timeit". 5 timeit output ("1000000 loops, best of 3: ... per loop"): * 0.247 usec * 0.252 usec * 0.247 usec * 0.251 usec * 0.251 usec It's disturbing to get 3 different "minimums" :-/ 5 perf.timeit outputs ("Average: 25 runs x 3 samples x 10^6 loops: ..."): * 250 ns +- 3 ns * 250 ns +- 3 ns * 251 ns +- 3 ns * 251 ns +- 4 ns * 251 ns +- 3 ns Note: I also got " 258 ns +- 17 ns" when I opened a webpage in Firefox while the benchmark is running. Note: I ran these benchmarks on a regular Linux without any specific tuning. ASLR is enabled, but the system was idle. Victor

1 0