Hi all, Long time no see! Apologies if this is not the right place to ask this question. I'm trying to run the benchmarks from speed.pypy.org on my system (just with cpython for now -- there is a chance that Linaro will be doing some work on the performance of Python on ARM at some point), but when I check out https://bitbucket.org/pypy/benchmarks/overview and run "./runner.py --fast" I get this: Running hexiom2... INFO:root:Running /usr/bin/python /home/mwhudson/src/benchmarks/own/hexiom2.py -n 5 INFO:root:Running /usr/bin/python /home/mwhudson/src/benchmarks/own/hexiom2.py -n 5 Traceback (most recent call last): File "./runner.py", line 302, in <module> main(sys.argv[1:]) File "./runner.py", line 283, in main full_store=full_store, branch=branch) File "./runner.py", line 41, in run_and_store results = perf.main(opts, funcs) File "/home/mwhudson/src/benchmarks/unladen_swallow/perf.py", line 1617, in main bench_result = func(base_cmd_prefix, changed_cmd_prefix, options) File "/home/mwhudson/src/benchmarks/benchmarks.py", line 17, in BM return SimpleBenchmark(Measure, *args, **kwds) File "/home/mwhudson/src/benchmarks/unladen_swallow/perf.py", line 462, in SimpleBenchmark return CompareBenchmarkData(base_data, changed_data, options) File "/home/mwhudson/src/benchmarks/unladen_swallow/perf.py", line 736, in CompareBenchmarkData return CompareMultipleRuns(base_times, changed_times, options) File "/home/mwhudson/src/benchmarks/unladen_swallow/perf.py", line 704, in CompareMultipleRuns significant, t_score = IsSignificant(base_times, changed_times) File "/home/mwhudson/src/benchmarks/unladen_swallow/perf.py", line 189, in IsSignificant t_score = TScore(sample1, sample2) File "/home/mwhudson/src/benchmarks/unladen_swallow/perf.py", line 170, in TScore return (avg(sample1) - avg(sample2)) / math.sqrt(error * 2) ZeroDivisionError: float division by zero is this just an artifact of running --fast? It seems the code could probably do with being a bit more robust? Cheers, mwh
Hi Michael, On Wed, May 1, 2013 at 3:54 AM, Michael Hudson-Doyle <micahel@gmail.com> wrote:
File "/home/mwhudson/src/benchmarks/unladen_swallow/perf.py", line 170, in TScore return (avg(sample1) - avg(sample2)) / math.sqrt(error * 2) ZeroDivisionError: float division by zero
The code could be more robust indeed :-( It's because hexiom2 runs only once in --fast mode, so it ends up with error == exactly 0.0. Fixed in f7abffc04667. A bientôt, Armin.
On Wed, May 1, 2013 at 11:16 AM, Armin Rigo <arigo@tunes.org> wrote:
Hi Michael,
On Wed, May 1, 2013 at 3:54 AM, Michael Hudson-Doyle <micahel@gmail.com> wrote:
File "/home/mwhudson/src/benchmarks/unladen_swallow/perf.py", line 170, in TScore return (avg(sample1) - avg(sample2)) / math.sqrt(error * 2) ZeroDivisionError: float division by zero
The code could be more robust indeed :-( It's because hexiom2 runs only once in --fast mode, so it ends up with error == exactly 0.0. Fixed in f7abffc04667.
A bientôt,
Armin. _______________________________________________ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
Btw, this is a benchmark run (without --fast) summary: http://paste.pound-python.org/show/32751/ It seems CPython on ARM is kinda bad (despite the fact that our assembler is bad too)
On 1 May 2013 21:48, Maciej Fijalkowski <fijall@gmail.com> wrote:
On Wed, May 1, 2013 at 11:16 AM, Armin Rigo <arigo@tunes.org> wrote:
Hi Michael,
On Wed, May 1, 2013 at 3:54 AM, Michael Hudson-Doyle <micahel@gmail.com> wrote:
File "/home/mwhudson/src/benchmarks/unladen_swallow/perf.py", line 170, in TScore return (avg(sample1) - avg(sample2)) / math.sqrt(error * 2) ZeroDivisionError: float division by zero
The code could be more robust indeed :-( It's because hexiom2 runs only once in --fast mode, so it ends up with error == exactly 0.0. Fixed in f7abffc04667.
A bientôt,
Armin. _______________________________________________ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
Btw, this is a benchmark run (without --fast) summary:
Thanks for that.
It seems CPython on ARM is kinda bad (despite the fact that our assembler is bad too)
Yeah. Maybe that's something we (Linaro) can do something about... Cheers, mwh
[apologies for the off-list mail] Thanks. For another kind of robustness, is it really necessary to throw away all the benchmark results when one (translate) fails? My run without --fast just fell over after well over a day of running :( Cheers, mwh On 1 May 2013 21:16, Armin Rigo <arigo@tunes.org> wrote:
Hi Michael,
On Wed, May 1, 2013 at 3:54 AM, Michael Hudson-Doyle <micahel@gmail.com> wrote:
File "/home/mwhudson/src/benchmarks/unladen_swallow/perf.py", line 170, in TScore return (avg(sample1) - avg(sample2)) / math.sqrt(error * 2) ZeroDivisionError: float division by zero
The code could be more robust indeed :-( It's because hexiom2 runs only once in --fast mode, so it ends up with error == exactly 0.0. Fixed in f7abffc04667.
A bientôt,
Armin.
participants (3)
-
Armin Rigo -
Maciej Fijalkowski -
Michael Hudson-Doyle