On Sun, Nov 24, 2013 at 5:32 PM, Yaroslav Halchenko <lists@onerussian.com> wrote:
On Tue, 15 Oct 2013, Nathaniel Smith wrote:
What do you have to lose?
btw -- fresh results are here http://yarikoptic.github.io/numpy-vbench/ .
I have tuned benchmarking so it now reflects the best performance across multiple executions of the whole battery, thus eliminating spurious variance if estimate is provided from a single point in time. Eventually I expect many of those curves to become even "cleaner".
On another note, what do you think of moving the vbench benchmarks into the main numpy tree? We already require everyone who submits a bug fix to add a test; there are a bunch of speed enhancements coming in these days and it would be nice if we had some way to ask people to submit a benchmark along with each one so that we know that the enhancement stays enhanced...
On this positive note (it is boring to start a new thread, isn't it?) -- would you be interested in me transfering numpy-vbench over to github.com/numpy ?
If you mean just moving the existing git repo under the numpy organization, like github.com/numpy/numpy-vbench, then I'm not sure how much difference it would make really. What seems like it'd be really useful though would be if the code could move into the main numpy tree, so that people could submit both benchmarks and optimizations within a single PR.
as of today, plots on http://yarikoptic.github.io/numpy-vbench should be updating 24x7 (just a loop, thus no time guarantee after you submit new changes).
Besides benchmarking new benchmarks (your PRs would still be very welcome, so far it was just me and Julian T) and revisions, that process also goes through a random sample of existing previously benchmarked revisions and re-runs the benchmarks thus improving upon the ultimate 'min' timing performance. So you can see already that many plots became much 'cleaner', although now there might be a bit of bias in estimates for recent revisions since they hadn't accumulated yet as many of 'independent runs' as older revisions.
Cool! -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org