
Franck Pommereau wrote:
I need some input for the benchmarking infrastructure. I'm nearly at the point where I need to have some place to run it before continuing (i.e. I need to try and use it, not just speculate).
This is not an answer to your question but might interest you.
We it comes to benchmarking, the following paper is certainly worth reading: http://www-plan.cs.colorado.edu/diwan/asplos09.pdf It explains how many benchmarks are biased and result in faulty measures (in important proportions), and how this can be compensated.
Seconded. This is an excellent paper. Btw, issues like this might be a reason to really reuse parts of the unladen-swallow benchmark runner, since they seem to have put work into doing the right thing from a statistics point of view. Cheers, Carl Friedrich