[pypy-dev] benchmarking input

[Carl Friedrich]

Franck Pommereau wrote:
>> I need some input for the benchmarking infrastructure. I'm nearly
>> at the point where I need to have some place to run it before
>> continuing (i.e. I need to try and use it, not just speculate).
> 
> This is not an answer to your question but might interest you.
> 
> We it comes to benchmarking, the following paper is certainly worth
> reading: http://www-plan.cs.colorado.edu/diwan/asplos09.pdf
> It explains how many benchmarks are biased and result in faulty measures
> (in important proportions), and how this can be compensated.

Seconded. This is an excellent paper. Btw, issues like this might be a 
reason to really reuse parts of the unladen-swallow benchmark runner, 
since they seem to have put work into doing the right thing from a 
statistics point of view.

Cheers,

Carl Friedrich