[pypy-dev] Clang benchmarks

Fri Dec 9 05:25:09 CET 2011

1. I had just run 'pypy ./runner.py' not knowing what I was actually
supposed to do<http://static.tumblr.com/tmrizjj/TS8lu94i7/1320363661170.jpg>.
I have since re-run the benchmarks, after discussing how they should
properly be run, and came up with the results below.

2. These are "normal" JiTting PyPys, however one is using the asmgcc gc
where as the other is using the shadowstack gc. I thought it might be
interesting to see if any significant info can be gathered from this
benchmark (something.

I compared my llvm/clang build to the prebuilt 64bit Linux binary provided
on the PyPy downloads page.
pypy ./runner.py --baseline=/usr/bin/pypy -p /home/chaos/pypy-1.7/bin/pypy

The baseline being the clang build of pypy.

Here are the results from stdout:

Wrote
/home/chaos/Desktop/benchmarks/lib/pypy/lib_pypy/ctypes_config_cache/_pyexpat_x86_64_.py.
Wrote
/home/chaos/Desktop/benchmarks/lib/pypy/lib_pypy/ctypes_config_cache/_locale_x86_64_.py.
Wrote
/home/chaos/Desktop/benchmarks/lib/pypy/lib_pypy/ctypes_config_cache/_resource_x86_64_.py.
Wrote
/home/chaos/Desktop/benchmarks/lib/pypy/lib_pypy/ctypes_config_cache/_syslog_x86_64_.py.
Running ai...
Running bm_chameleon...
Running bm_mako...
Running chaos...
Running crypto_pyaes...
Running django...
Running fannkuch...
Running float...
Running go...
Running html5lib...
Running json_bench...
Running meteor-contest...
Running nbody_modified...
Running pyflate-fast...
Running raytrace-simple...
Running richards...
Running rietveld...
Running slowspitfire...
Running spambayes...
Running spectral-norm...
Running spitfire...
Running spitfire_cstringio...
Running sympy_expand...
Running sympy_integrate...
Running sympy_str...
Running sympy_sum...
Running telco...
Running translate...
Running twisted_iteration...
Running twisted_names...
Running twisted_pb...
Running twisted_tcp...

Report on Linux infinity 3.1.1-gentoo #1 SMP PREEMPT Sun Nov 20 03:57:03
PST 2011 x86_64 AMD Turion(tm) II Dual-Core Mobile M520
Total CPU cores: 2

### ai ###
Min: 0.139957 -> 0.134342: 1.0418x faster
Avg: 0.156932 -> 0.151293: 1.0373x faster
Not significant
Stddev: 0.01776 -> 0.01588: 1.1185x smaller

### bm_chameleon ###
Min: 0.051917 -> 0.050041: 1.0375x faster
Avg: 0.064268 -> 0.060951: 1.0544x faster
Not significant
Stddev: 0.01868 -> 0.01755: 1.0643x smaller

### bm_mako ###
Min: 0.146432 -> 0.136968: 1.0691x faster
Avg: 0.164050 -> 0.153924: 1.0658x faster
Not significant
Stddev: 0.02840 -> 0.02575: 1.1028x smaller

### chaos ###
Min: 0.024366 -> 0.025635: 1.0521x slower
Avg: 0.037429 -> 0.038286: 1.0229x slower
Not significant
Stddev: 0.07200 -> 0.07000: 1.0286x smaller

### crypto_pyaes ###
Min: 0.121551 -> 0.133157: 1.0955x slower
Avg: 0.136914 -> 0.147496: 1.0773x slower
Not significant
Stddev: 0.05749 -> 0.05621: 1.0229x smaller

### django ###
Min: 0.101975 -> 0.095751: 1.0650x faster
Avg: 0.110838 -> 0.104973: 1.0559x faster
Significant (t=3.146148, a=0.95)
Stddev: 0.00960 -> 0.00904: 1.0616x smaller

### fannkuch ###
Min: 0.511440 -> 0.446432: 1.1456x faster
Avg: 0.521307 -> 0.451537: 1.1545x faster
Significant (t=23.373959, a=0.95)
Stddev: 0.01503 -> 0.01482: 1.0138x smaller

### float ###
Min: 0.083709 -> 0.082221: 1.0181x faster
Avg: 0.106045 -> 0.102772: 1.0318x faster
Not significant
Stddev: 0.01608 -> 0.01566: 1.0271x smaller

### go ###
Min: 0.269653 -> 0.263512: 1.0233x faster
Avg: 0.481460 -> 0.468852: 1.0269x faster
Not significant
Stddev: 0.19745 -> 0.19127: 1.0323x smaller

### html5lib ###
Min: 5.790741 -> 5.692144: 1.0173x faster
Avg: 7.893078 -> 7.687743: 1.0267x faster
Not significant
Stddev: 2.83904 -> 2.67228: 1.0624x smaller

### json_bench ###
Min: 3.498221 -> 3.506862: 1.0025x slower
Avg: 3.541327 -> 3.546107: 1.0013x slower
Not significant
Stddev: 0.14921 -> 0.13440: 1.1102x smaller

### meteor-contest ###
Min: 0.338499 -> 0.313130: 1.0810x faster
Avg: 0.346351 -> 0.320863: 1.0794x faster
Significant (t=8.045312, a=0.95)
Stddev: 0.01531 -> 0.01635: 1.0676x larger

### nbody_modified ###
Min: 0.084227 -> 0.083003: 1.0147x faster
Avg: 0.086098 -> 0.085613: 1.0057x faster
Not significant
Stddev: 0.00588 -> 0.00667: 1.1353x larger

### pyflate-fast ###
Min: 1.049249 -> 1.025093: 1.0236x faster
Avg: 1.099029 -> 1.069142: 1.0280x faster
Significant (t=5.351335, a=0.95)
Stddev: 0.02828 -> 0.02756: 1.0262x smaller

### raytrace-simple ###
Min: 0.067093 -> 0.068943: 1.0276x slower
Avg: 0.081748 -> 0.082917: 1.0143x slower
Not significant
Stddev: 0.02475 -> 0.02978: 1.2031x larger

### richards ###
Min: 0.008263 -> 0.008141: 1.0150x faster
Avg: 0.009398 -> 0.009126: 1.0298x faster
Not significant
Stddev: 0.00345 -> 0.00286: 1.2085x smaller

### rietveld ###
Min: 0.243964 -> 0.241945: 1.0083x faster
Avg: 0.612855 -> 0.594768: 1.0304x faster
Not significant
Stddev: 0.58823 -> 0.56197: 1.0467x smaller

### slowspitfire ###
Min: 0.643957 -> 0.624110: 1.0318x faster
Avg: 0.671090 -> 0.654331: 1.0256x faster
Significant (t=3.642100, a=0.95)
Stddev: 0.02335 -> 0.02265: 1.0310x smaller

### spambayes ###
Min: 0.153035 -> 0.147505: 1.0375x faster
Avg: 0.294454 -> 0.281956: 1.0443x faster
Not significant
Stddev: 0.12360 -> 0.11756: 1.0514x smaller

### spectral-norm ###
Min: 0.028271 -> 0.029005: 1.0260x slower
Avg: 0.032468 -> 0.033037: 1.0175x slower
Not significant
Stddev: 0.01273 -> 0.01247: 1.0204x smaller

### spitfire ###
Min: 9.720000 -> 9.230000: 1.0531x faster
Avg: 9.817600 -> 9.385200: 1.0461x faster
Significant (t=16.544288, a=0.95)
Stddev: 0.10499 -> 0.15209: 1.4486x larger

### spitfire_cstringio ###
Min: 4.840000 -> 4.710000: 1.0276x faster
Avg: 4.863400 -> 4.748400: 1.0242x faster
Significant (t=8.894143, a=0.95)
Stddev: 0.05294 -> 0.07454: 1.4081x larger

### sympy_expand ###
Min: 2.006796 -> 1.881002: 1.0669x faster
Avg: 2.503000 -> 2.402378: 1.0419x faster
Not significant
Stddev: 0.97556 -> 0.93094: 1.0479x smaller

### sympy_integrate ###
Min: 4.551186 -> 4.328435: 1.0515x faster
Avg: 6.496911 -> 6.308380: 1.0299x faster
Not significant
Stddev: 2.48643 -> 2.42822: 1.0240x smaller

### sympy_str ###
Min: 0.896834 -> 1.034409: 1.1534x slower
Avg: 1.944165 -> 1.933199: 1.0057x faster
Not significant
Stddev: 1.08961 -> 1.02921: 1.0587x smaller

### sympy_sum ###
Min: 1.292385 -> 1.256345: 1.0287x faster
Avg: 1.729903 -> 1.671316: 1.0351x faster
Not significant
Stddev: 0.53698 -> 0.50304: 1.0675x smaller

### telco ###
Min: 0.098984 -> 0.093985: 1.0532x faster
Avg: 0.119122 -> 0.113263: 1.0517x faster
Not significant
Stddev: 0.05782 -> 0.05571: 1.0379x smaller

### trans_annotate ###
Raw results: [1377.2] None

### trans_rtype ###
Raw results: [955.9] None

### trans_backendopt ###
Raw results: [421.4] None

### trans_database ###
Raw results: [568.0] None

### trans_source ###
Raw results: [615.2] None

### twisted_iteration ###
Min: 0.014090 -> 0.014660: 1.0404x slower
Avg: 0.014315 -> 0.014838: 1.0366x slower
Significant (t=-18.280629, a=0.95)
Stddev: 0.00015 -> 0.00014: 1.0274x smaller

### twisted_names ###
Min: 0.007479 -> 0.007457: 1.0030x faster
Avg: 0.007976 -> 0.007883: 1.0118x faster
Significant (t=2.202431, a=0.95)
Stddev: 0.00023 -> 0.00018: 1.2883x smaller

### twisted_pb ###
Min: 0.037453 -> 0.039920: 1.0659x slower
Avg: 0.039451 -> 0.041383: 1.0490x slower
Significant (t=-10.352263, a=0.95)
Stddev: 0.00103 -> 0.00082: 1.2537x smaller

### twisted_tcp ###
Min: 1.113818 -> 1.192250: 1.0704x slower
Avg: 1.138590 -> 1.222393: 1.0736x slower
Significant (t=-16.106458, a=0.95)
Stddev: 0.02450 -> 0.02745: 1.1207x larger

On Thu, Dec 8, 2011 at 01:16, Armin Rigo <arigo at tunes.org> wrote:

> Hi Justin,
>
> On Thu, Dec 8, 2011 at 09:09, Justin Noah <justinnoah at gmail.com> wrote:
> > Here are the llvm/clang build using the shadowstack gc. What do you
> think?
>
> 1. What is the baseline you are comparing it with?
>
> 2. I assume these are normal JITting PyPys.  In that case, then
> for benchmarks like these, it doesn't really matter what C compiler we
> used, because most of the time is spent inside JIT-generated assembler
> anyway.
>
>
> A bientôt,
>
> Armin.
>

-- 
- Justin Noah
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20111208/e49b33dc/attachment-0001.html>