[Numpy-discussion] RFC: is it worth giving a lightning talk at PyCon 2014 on numpy vbench-marking?

[Yaroslav Halchenko]

On Tue, 15 Oct 2013, Nathaniel Smith wrote:
> > and I think it might be worth letting people know about my little project.  I
> > would really appreciate your sincere feedback (e.g. "not worth it" would be
> > valuable too).  Here is the title/abstract

> >     numpy-vbench -- speed benchmarks for NumPy

> >     http://yarikoptic.github.io/numpy-vbench provides collection of speed
> >     performance benchmarks for NumPy.  Benchmarking of multiple
> >     maintenance and current development branches allows not only to timely
> >     react to new performance regressions, but also to compare NumPy
> >     performance across releases.  Your contributions would help to
> >     guarantee that your code does not become slower with a new NumPy
> >     release.

> What do you have to lose?

time? ;)

> > btw -- fresh results are here http://yarikoptic.github.io/numpy-vbench/ .

> > I have tuned benchmarking so it now reflects the best performance across
> > multiple executions of the whole battery, thus eliminating spurious
> > variance if estimate is provided from a single point in time.  Eventually I
> > expect many of those curves to become even "cleaner".

> On another note, what do you think of moving the vbench benchmarks
> into the main numpy tree? We already require everyone who submits a
> bug fix to add a test; there are a bunch of speed enhancements coming
> in these days and it would be nice if we had some way to ask people to
> submit a benchmark along with each one so that we know that the
> enhancement stays enhanced...

I would be thrilled to do so, but

#1 I would need time I currently do not have to harmonize my setup for
   such inclusion

#2 I am still not sure if my tune ups on top of vbench should get merged
   as is https://github.com/pydata/vbench/pull/33

#3 vbench itself might go through a considerable RF if someone finds some
   spare time: https://github.com/pydata/vbench/issues/34
  
If merge for the PR #2 would happen, and test_perf.py from pandas would
get integrated within vbench -- then it might be logical to think about
#1 regardless of the ultimate resolution to #3 ;)  Whenever I have
some time for it I would try to push in that direction, and I do not
think it is unfeasible to see all of that done actually by PyCon 2014
(so the talk will have a different url ;) ), I just can't promise ATM.

-- 
Yaroslav O. Halchenko, Ph.D.
http://neuro.debian.net http://www.pymvpa.org http://www.fail2ban.org
Senior Research Associate,     Psychological and Brain Sciences Dept.
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834                       Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik