On Wed, Nov 4, 2009 at 5:54 PM, Collin Winter <collinw@gmail.com> wrote:
Do note that the --track_memory option to perf.py imposes some overhead that interferes with the performance figures.
Thanks for the notice, without -m/--track_memory the deviation in results is indeed much smaller.
I'd recommend running the benchmarks again without --track_memory.
Done: $ python unladen-tests/perf.py -r --benchmarks=-2to3,all py261/python wpy/python Report on Linux zeus 2.6.31-14-generic #48-Ubuntu SMP Fri Oct 16 14:05:01 UTC 2009 x86_64 Total CPU cores: 2 ai: Min: 0.629343 -> 0.576259: 9.21% faster Avg: 0.634689 -> 0.581551: 9.14% faster Significant (t=39.404870, a=0.95) Stddev: 0.01259 -> 0.00484: 160.04% smaller call_simple: Min: 1.796710 -> 1.700046: 5.69% faster Avg: 1.801533 -> 1.716367: 4.96% faster Significant (t=137.452069, a=0.95) Stddev: 0.00522 -> 0.00333: 56.64% smaller django: Min: 1.280840 -> 1.275350: 0.43% faster Avg: 1.287179 -> 1.287233: 0.00% slower Not significant Stddev: 0.01055 -> 0.00581: 81.60% smaller iterative_count: Min: 0.211744 -> 0.123271: 71.77% faster Avg: 0.213148 -> 0.128596: 65.75% faster Significant (t=88.510311, a=0.95) Stddev: 0.00233 -> 0.00926: 74.80% larger normal_startup: Min: 0.520829 -> 0.516412: 0.86% faster Avg: 0.559170 -> 0.554678: 0.81% faster Not significant Stddev: 0.02031 -> 0.02093: 2.98% larger pickle: Min: 1.988127 -> 1.926643: 3.19% faster Avg: 2.000676 -> 1.936185: 3.33% faster Significant (t=36.712505, a=0.95) Stddev: 0.01650 -> 0.00603: 173.67% smaller pickle_dict: Min: 1.681116 -> 1.619192: 3.82% faster Avg: 1.701952 -> 1.629548: 4.44% faster Significant (t=34.513963, a=0.95) Stddev: 0.01721 -> 0.01200: 43.46% smaller pickle_list: Min: 0.918128 -> 0.884967: 3.75% faster Avg: 0.925534 -> 0.891200: 3.85% faster Significant (t=60.451407, a=0.95) Stddev: 0.00496 -> 0.00276: 80.00% smaller pybench: Min: 58692 -> 51128: 14.79% faster Avg: 59914 -> 52316: 14.52% faster regex_compile: Min: 0.894190 -> 0.816447: 9.52% faster Avg: 0.900353 -> 0.826003: 9.00% faster Significant (t=24.974080, a=0.95) Stddev: 0.00448 -> 0.02943: 84.78% larger regex_effbot: Min: 0.124442 -> 0.123750: 0.56% faster Avg: 0.134908 -> 0.126137: 6.95% faster Significant (t=5.496357, a=0.95) Stddev: 0.01581 -> 0.00218: 625.68% smaller regex_v8: Min: 0.132730 -> 0.143494: 7.50% slower Avg: 0.134287 -> 0.147387: 8.89% slower Significant (t=-40.654627, a=0.95) Stddev: 0.00108 -> 0.00304: 64.34% larger rietveld: Min: 0.754050 -> 0.737335: 2.27% faster Avg: 0.770227 -> 0.754642: 2.07% faster Significant (t=7.547765, a=0.95) Stddev: 0.01434 -> 0.01486: 3.49% larger slowpickle: Min: 0.858494 -> 0.795162: 7.96% faster Avg: 0.862350 -> 0.799479: 7.86% faster Significant (t=133.690989, a=0.95) Stddev: 0.00394 -> 0.00257: 52.92% smaller slowspitfire: Min: 0.955587 -> 0.909843: 5.03% faster Avg: 0.965960 -> 0.925845: 4.33% faster Significant (t=16.351067, a=0.95) Stddev: 0.01237 -> 0.02119: 41.63% larger slowunpickle: Min: 0.409312 -> 0.346982: 17.96% faster Avg: 0.412381 -> 0.349148: 18.11% faster Significant (t=242.889869, a=0.95) Stddev: 0.00198 -> 0.00169: 17.61% smaller startup_nosite: Min: 0.195620 -> 0.194328: 0.66% faster Avg: 0.230811 -> 0.238523: 3.23% slower Significant (t=-3.869944, a=0.95) Stddev: 0.01932 -> 0.02052: 5.87% larger threaded_count: Min: 0.222133 -> 0.133764: 66.06% faster Avg: 0.236670 -> 0.147750: 60.18% faster Significant (t=57.472693, a=0.95) Stddev: 0.01317 -> 0.00813: 61.98% smaller unpack_sequence: Min: 0.000129 -> 0.000119: 8.43% faster Avg: 0.000132 -> 0.000123: 7.22% faster Significant (t=24.614061, a=0.95) Stddev: 0.00003 -> 0.00011: 77.02% larger unpickle: Min: 1.191255 -> 1.149132: 3.67% faster Avg: 1.218023 -> 1.162351: 4.79% faster Significant (t=21.222711, a=0.95) Stddev: 0.02242 -> 0.01362: 64.54% smaller unpickle_list: Min: 0.880991 -> 0.965611: 8.76% slower Avg: 0.898949 -> 0.985231: 8.76% slower Significant (t=-17.387537, a=0.95) Stddev: 0.04838 -> 0.01103: 338.79% smaller