On 06.03.16 11:30, Maciej Fijalkowski wrote:
> this is really difficult to read, can you tell me which column am I looking at?
The first column is the searched pattern. The second column is the
number of found matches (for control, it should be the same with all
engines and versions). The third column, under the "re" header is the
time in milliseconds. The column under the "str.find" header is the time
of searching without using regular expressions.
PyPy 2.2 usually is significantly faster than CPython 2.7, except
searching plain string with regular expression. But thanks to Flexible
String Representation searching plain string with and without regular
expression is faster on CPython 3.6.
Hi,
I just released perf 0.3. Major changes:
- "python -m perf" CLI now has 3 commands: show, compare and
compare_to. Compare commands says if the difference is significant (I
copied the code from perf.py)
- TextRunner is now able to spawn child processes, parse command
arguments and more
- If TextRunner detects isolated CPUs, it automatically pins the CPUs
of the worker processes to isolated CPUs
- Add ``--json-file`` command line option
- Add TextRunner.bench_sample_func() method: the sample function is
responsible to measure the elapsed time, useful for microbenchmarks
- Enhance a lot of the documentation
Writing a benchmark now only takes one line:
"perf.text_runner.TextRunner().bench_func(func)"! Full example:
---
import time
import perf.text_runner
def func():
time.sleep(0.001)
perf.text_runner.TextRunner().bench_func(func)
---
I looked at PyPy benchmarks:
https://bitbucket.org/pypy/benchmarks
Results can also be serialized to JSON, but the serialization is only
done at the end: the final result is serialized. It's not possible to
save each run in a JSON file.
Running multiple processes is not supported neither.
With perf, the final JSON contains all data: all runs, all samples
even warmup samples.
perf now also collects metadata in each worker process. So it is more
safer to compare runs since it's possible to manually check when and
how the worker executed the benchmark. For example, the CPU affinity
is now saved in metadata.
For example, "python -m perf.timeit" now saves the setup and
statements in metadata.
With perf 0.3, TextRunner now also includes a builtin calibration to
compute the number of outter loop iteartions: repeat each sample so it
takes between 100 ms and 1 sec (min/max are configurable).
Victor
Hi,
I completed the API of my small perf module and released a version 0.2:
https://perf.readthedocs.io/
It is supposed to provide the basic tools to collect samples, compute
the average, display the result, etc. I started to work on JSON
serialization to "easily" run multiple processes. The idea is also to
be split the code to produce numbers and the code to display results.
I expect that we can do better to display results. See for example
speed.python.org and speed.pypy.org, it's nicer than perf.py text
output ;-)
I also started to hack CPython benchmark suite (benchmarks repository)
to use my perf module:
https://hg.python.org/sandbox/benchmarks_perf
I should now stop NIH and see how to merge my work with PyPy fork of
benchmarks ;-)
FYI I started to write the perf module because I started to write an
article about the impact of CPU speed on Python microbenchmarks, and I
wanted to have a smart timeit running multiple processes. Since it was
cool to work on such project, I started to hack benchmarks, but maybe
I gone too far and I should look at PyPy's benchmark instead ;-)
Victor
On Thu, 2 Jun 2016 03:19:32 +0200
Victor Stinner <victor.stinner(a)gmail.com>
wrote:
> I'm interested by the very basic perf.py internal text format: one
> timing per line, that's all. But it's incomplete, the "loops"
> informaiton is not stored. Maybe a binary format is better? I don't
> know yet.
Just use a simple JSON format.
Regards
Antoine.
Hi Victor,
On 2 June 2016 at 10:58, Victor Stinner <victor.stinner(a)gmail.com> wrote:
> Usually, timeit it used to compare two versions of Python.
That's not the use case I'm focusing about here: timeit is also used by Mr.
Random Programmer to tweak their Python code to improve the
performance. (Often, it's the performance of non-representative
microbenchmarks, but well, better have a tool that at least get some
saner results than the current timeit.)
A bientôt,
Armin.
On Thu, 2 Jun 2016 10:58:35 +0200
Victor Stinner <victor.stinner(a)gmail.com>
wrote:
>
> Usually, timeit it used to compare two versions of Python.
timeit is used for many different things, including comparing two
versions of Python, but not only.
> For the CLI, we can extend timeit CLI to accept the path/name of two
> python binaries.
That sounds reasonable.
Regards
Antoine.
2016-06-02 10:38 GMT+02:00 Armin Rigo <arigo(a)tunes.org>:
> Looks good. IMHO the important bit is that `timeit` is simple to use,
> readily available, and gives just a number, which makes it very
> attractive for people.
By default min & max are hidden. You can show them using -v option.
To make the output even simpler, maybe the standard deviation can be
displayed "in english". Something like:
* "Average: 250 ns +- 3 ns" => "Average: 250 ns (stable)", or just
"Average: 250 ns"
* "Average: 250 ns +- 120 ns" => "Average: 250 ns (not reliable, try
again on an idle system)"
Usually, timeit it used to compare two versions of Python. Maybe we
should focus on this use case, and check if the difference is
significant, as perf.py does? By default, perf.py does *not* display
any number if the difference is not significant. I like this
behaviour, even if it can be surprising for the first time.
For the CLI, we can extend timeit CLI to accept the path/name of two
python binaries. Or we can use something like pybench to store result
into files and then load & compare two files.
Victor
Hi Victor,
On 2 June 2016 at 03:19, Victor Stinner <victor.stinner(a)gmail.com> wrote:
> 5 timeit output ("1000000 loops, best of 3: ... per loop"):
>
> * 0.247 usec
> * 0.252 usec
> * 0.247 usec
> * 0.251 usec
> * 0.251 usec
>
> 5 perf.timeit outputs ("Average: 25 runs x 3 samples x 10^6 loops: ..."):
>
> * 250 ns +- 3 ns
> * 250 ns +- 3 ns
> * 251 ns +- 3 ns
> * 251 ns +- 4 ns
> * 251 ns +- 3 ns
Looks good. IMHO the important bit is that `timeit` is simple to use,
readily available, and gives just a number, which makes it very
attractive for people. Your output would achieve the same result
(with the `+-` added, which is fine) assuming that it eventually
replaces `timeit` in the standard library.
I know there are many good reasons for why getting just a single
number is not enough, but I'd say that we still need to achieve the
best practical results given that constrain. The results you posted
above seem to show that `perf.timeit` is better than `timeit` at doing
that, and I believe that it's a great step forward.
A bientôt,
Armin.
Hi,
I started to write blog posts on stable benchmarks:
1) https://haypo.github.io/journey-to-stable-benchmark-system.html
2) https://haypo.github.io/journey-to-stable-benchmark-deadcode.html
3) https://haypo.github.io/journey-to-stable-benchmark-average.html
One important point is that minimum is commonly used in Python
benchmarks, whereas it is a bad practice to get a stable benchmark.
I started to work on a toolkit to write benchmarks, the new "perf" module:
http://perf.readthedocs.io/en/latest/https://github.com/haypo/perf
I used timeit as a concrete use case, since timeit is popular and
badly implemented. timeit currently uses 1 process running the
microbenchmarks 3 times and take the minimum. timeit is *known* to be
unstable, and the common advice is to run it at least 3 times and
again take the minimum of the minimum.
Example of links about timeit being unstable:
* https://mail.python.org/pipermail/python-dev/2012-August/121379.html
* https://bugs.python.org/issue23693
* https://bugs.python.org/issue6422 (not directly related)
Moreover, the timeit module disables the garbage collector which is
also wrong. It's wrong because it's rare to disable the GC in
applications.
My goal for the perf module is to provide basic features and then
reuse it in existing benchmarks:
* mean() and stdev() to display result
* clock chosen for benchmark
* result classes to store numbers
* etc.
Work in progress:
* new implementation of timeit using multiple processes
* perf.metadata module: collect various information about Python, the
system, etc.
* file format to store numbers and metadata
I'm interested by the very basic perf.py internal text format: one
timing per line, that's all. But it's incomplete, the "loops"
informaiton is not stored. Maybe a binary format is better? I don't
know yet.
It should be possible to cumulate files of multiple processes. I'm
also interested to implement a generic "rerun" command to add more
samples if a benchmark doesn't look stable enough.
perf.timeit looks more stable than timeit, the CLI is basically the
same: replace "-m timeit" with "-m perf.timeit".
5 timeit output ("1000000 loops, best of 3: ... per loop"):
* 0.247 usec
* 0.252 usec
* 0.247 usec
* 0.251 usec
* 0.251 usec
It's disturbing to get 3 different "minimums" :-/
5 perf.timeit outputs ("Average: 25 runs x 3 samples x 10^6 loops: ..."):
* 250 ns +- 3 ns
* 250 ns +- 3 ns
* 251 ns +- 3 ns
* 251 ns +- 4 ns
* 251 ns +- 3 ns
Note: I also got " 258 ns +- 17 ns" when I opened a webpage in Firefox
while the benchmark is running.
Note: I ran these benchmarks on a regular Linux without any specific
tuning. ASLR is enabled, but the system was idle.
Victor
