[Numpy-discussion] RFC: is it worth giving a lightning talk at PyCon 2014 on numpy vbench-marking?

[Yaroslav Halchenko]

On Tue, 15 Oct 2013, Nathaniel Smith wrote:
> What do you have to lose?

> > btw -- fresh results are here http://yarikoptic.github.io/numpy-vbench/ .

> > I have tuned benchmarking so it now reflects the best performance across
> > multiple executions of the whole battery, thus eliminating spurious
> > variance if estimate is provided from a single point in time.  Eventually I
> > expect many of those curves to become even "cleaner".

> On another note, what do you think of moving the vbench benchmarks
> into the main numpy tree? We already require everyone who submits a
> bug fix to add a test; there are a bunch of speed enhancements coming
> in these days and it would be nice if we had some way to ask people to
> submit a benchmark along with each one so that we know that the
> enhancement stays enhanced...

On this positive note (it is boring to start a new thread, isn't it?) --
would you be interested in me transfering numpy-vbench over to
github.com/numpy ? 

as of today, plots on http://yarikoptic.github.io/numpy-vbench  should
be updating 24x7 (just a loop, thus no time guarantee after you submit
new changes).  

Besides benchmarking new benchmarks (your PRs  would still be very
welcome,  so far it was just me and Julian T) and revisions, that
process also goes through a random sample of existing previously
benchmarked revisions and re-runs the benchmarks thus improving upon the
ultimate 'min' timing performance.  So you can see already that many
plots became much 'cleaner', although now there might be a bit of bias
in estimates for recent revisions since they hadn't accumulated yet as
many of 'independent runs' as older revisions.

-- 
Yaroslav O. Halchenko, Ph.D.
http://neuro.debian.net http://www.pymvpa.org http://www.fail2ban.org
Senior Research Associate,     Psychological and Brain Sciences Dept.
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834                       Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik