[pypy-issue] [issue766] Shadowstack root finder should be competitive with asmgcc in speed

[Dan Villiom Podlaski Christiansen]

New submission from Dan Villiom Podlaski Christiansen <danchr at gmail.com>:

I just ran a benchmark comparison between two 32-bit builds of PyPy; one using 
asmgcc and the other using shadowstack. The shadowstack root finder is slower 
than asmgcc by 6-20% on the four benchmarks with significant differences. On the 
remainder, it's either slower, but insignificant, or approximately the same in 
speed.

The asmgcc root finder seems like a dead end on Mac OS X: It only works with the 
ancient Apple GCC 4.0, a compiler which probably won't be available on the 
upcoming release of Mac OS X. For this reason, it would be nice to make 
shadowstack the default on Mac OS X.

I've included the summary below, and attached an indented copy of JSON results.

(For some reason or other, many of the benchmarks didn't work. I'll investigate 
later and report back.)


Report on Darwin dookie.local 10.7.0 Darwin Kernel Version 10.7.0: Sat Jan 29 
15:17:16 PST 2011; root:xnu-1504.9.37~1/RELEASE_I386 i386 i386
Total CPU cores: 2

### ai ###
Min: 0.419934 -> 0.489262: 1.1651x slower
Avg: 0.443812 -> 0.510724: 1.1508x slower
Significant (t=-11.569626, a=0.95)
Stddev: 0.02938 -> 0.02845: 1.0325x smaller


### chaos ###
Min: 0.031458 -> 0.031927: 1.0149x slower
Avg: 0.046057 -> 0.045682: 1.0082x faster
Not significant
Stddev: 0.06498 -> 0.06411: 1.0135x smaller


### crypto_pyaes ###
Min: 0.174995 -> 0.181307: 1.0361x slower
Avg: 0.209730 -> 0.205039: 1.0229x faster
Not significant
Stddev: 0.10432 -> 0.11878: 1.1386x larger


### fannkuch ###
Min: 0.608829 -> 0.624988: 1.0265x slower
Avg: 1.833948 -> 1.976551: 1.0778x slower
Not significant
Stddev: 8.57532 -> 9.48609: 1.1062x larger


### float ###
Min: 0.104113 -> 0.102358: 1.0171x faster
Avg: 0.115848 -> 0.111435: 1.0396x faster
Not significant
Stddev: 0.02712 -> 0.01561: 1.7378x smaller


### go ###
Min: 0.129704 -> 0.133338: 1.0280x slower
Avg: 0.197544 -> 0.204464: 1.0350x slower
Not significant
Stddev: 0.06363 -> 0.06910: 1.0860x larger


### meteor-contest ###
Min: 0.342629 -> 0.411106: 1.1999x slower
Avg: 0.350464 -> 0.417532: 1.1914x slower
Significant (t=-24.259370, a=0.95)
Stddev: 0.01634 -> 0.01073: 1.5232x smaller


### nbody_modified ###
Min: 0.102595 -> 0.100291: 1.0230x faster
Avg: 0.104505 -> 0.102977: 1.0148x faster
Not significant
Stddev: 0.00611 -> 0.01208: 1.9769x larger


### pyflate-fast ###
Min: 1.816023 -> 1.924431: 1.0597x slower
Avg: 1.863659 -> 2.016530: 1.0820x slower
Significant (t=-14.342805, a=0.95)
Stddev: 0.03321 -> 0.06765: 2.0369x larger


### raytrace-simple ###
Min: 0.101490 -> 0.102709: 1.0120x slower
Avg: 0.133066 -> 0.154666: 1.1623x slower
Not significant
Stddev: 0.03908 -> 0.12808: 3.2777x larger


### richards ###
Min: 0.009366 -> 0.009467: 1.0108x slower
Avg: 0.012987 -> 0.013818: 1.0640x slower
Not significant
Stddev: 0.00578 -> 0.00697: 1.2061x larger


### slowspitfire ###
Min: 1.585539 -> 1.706415: 1.0762x slower
Avg: 1.607377 -> 1.736389: 1.0803x slower
Significant (t=-23.094884, a=0.95)
Stddev: 0.02139 -> 0.03321: 1.5522x larger


### spectral-norm ###
Min: 0.039061 -> 0.039828: 1.0196x slower
Avg: 0.043881 -> 0.043506: 1.0086x faster
Not significant
Stddev: 0.02074 -> 0.01282: 1.6182x smaller


### telco ###
Min: 0.207924 -> 0.207921: 1.0000x faster
Avg: 0.223767 -> 0.223237: 1.0024x faster
Not significant
Stddev: 0.02803 -> 0.02889: 1.0307x larger


### waf ###
Min: 13.688543 -> 13.493718: 1.0144x faster
Avg: 13.806294 -> 15.223909: 1.1027x slower
Not significant
Stddev: 0.11723 -> 3.58803: 30.6075x larger

----------
files: result-indented.json
messages: 2685
nosy: danchr, pypy-issue
priority: wish
status: unread
title: Shadowstack root finder should be competitive with asmgcc in speed

________________________________________
PyPy bug tracker <tracker at bugs.pypy.org>
<https://bugs.pypy.org/issue766>
________________________________________