[Numpy-discussion] Really cruel draft of vbench setup for NumPy (.add.reduce benchmarks since 2011)

Yaroslav Halchenko lists at onerussian.com
Tue Jul 16 23:53:48 EDT 2013


and to put so far reported findings into some kind of automated form,
please welcome

http://www.onerussian.com/tmp/numpy-vbench/#benchmarks-performance-analysis

This is based on a simple 1-way anova of last 10 commits and some point
in the past where 10 other commits had smallest timing and were significantly
different from the last 10 commits.

"Possible recent" is probably too noisy and not sure if useful -- it should
point to a closest in time (to the latest commits) diff where a
significant excursion from current performance was detected.  So per se it has
nothing to do with the initial detected performance hit, but in some cases
seems still to reasonably locate commits hitting on performance.

Enjoy,

On Tue, 09 Jul 2013, Yaroslav Halchenko wrote:

> Julian Taylor contributed some benchmarks he was "concerned" about, so
> now the collection is even better.

> I will keep updating tests on the same url:
> http://www.onerussian.com/tmp/numpy-vbench/
> [it is now running and later I will upload with more commits for higher temporal fidelity]

> of particular interest for you might be:
> some minor consistent recent losses in
> http://www.onerussian.com/tmp/numpy-vbench/vb_vb_io.html#strided-assign-float64
> http://www.onerussian.com/tmp/numpy-vbench/vb_vb_io.html#strided-assign-float32
> http://www.onerussian.com/tmp/numpy-vbench/vb_vb_io.html#strided-assign-int16
> http://www.onerussian.com/tmp/numpy-vbench/vb_vb_io.html#strided-assign-int8

> seems have lost more than 25% of performance throughout the timeline
> http://www.onerussian.com/tmp/numpy-vbench/vb_vb_io.html#memcpy-int8 

> "fast" calls to all/any seemed to be hurt twice in their life time now running
> *3 times slower* than in 2011 -- inflection points correspond to regressions
> and/or their fixes in those functions to bring back performance on "slow"
> cases (when array traversal is needed, e.g. on arrays of zeros for any)

> http://www.onerussian.com/tmp/numpy-vbench/vb_vb_reduce.html#numpy-all-fast
> http://www.onerussian.com/tmp/numpy-vbench/vb_vb_reduce.html#numpy-any-fast

> Enjoy

> On Mon, 01 Jul 2013, Yaroslav Halchenko wrote:

> > FWIW -- updated plots with contribution from Julian Taylor
> > http://www.onerussian.com/tmp/numpy-vbench-20130701/vb_vb_indexing.html#mmap-slicing
> > ;-)

> > On Mon, 01 Jul 2013, Yaroslav Halchenko wrote:

> > > Hi Guys,

> > > not quite the recommendations you expressed,  but here is my ugly
> > > attempt to improve benchmarks coverage:

> > > http://www.onerussian.com/tmp/numpy-vbench-20130701/index.html

> > > initially I also ran those ufunc benchmarks per each dtype separately,
> > > but then resulting webpage is loong which brings my laptop on its knees
> > > by firefox.  So I commented those out for now, and left only "summary"
> > > ones across multiple datatypes.

> > > There is a bug in sphinx which forbids embedding some figures for
> > > vb_random "as is", so pardon that for now...

> > > I have not set cpu affinity of the process (but ran it at nice -10), so  may be
> > > that also contributed to variance of benchmark estimates.  And there probably
> > > could be more of goodies (e.g. gc control etc) to borrow from
> > > https://github.com/pydata/pandas/blob/master/vb_suite/test_perf.py which I have
> > > just discovered to minimize variance.

> > > nothing really interesting was pin-pointed so far, besides that 

> > > - svd became a bit faster since few months back ;-)

> > > http://www.onerussian.com/tmp/numpy-vbench-20130701/vb_vb_linalg.html

> > > - isnan (and isinf, isfinite) got improved

> > > http://www.onerussian.com/tmp/numpy-vbench-20130701/vb_vb_ufunc.html#numpy-isnan-a-10types

> > > - right_shift got a miniscule slowdown from what it used to be?

> > > http://www.onerussian.com/tmp/numpy-vbench-20130701/vb_vb_ufunc.html#numpy-right-shift-a-a-3types

> > > As before -- current code of those benchmarks collection is available
> > > at http://github.com/yarikoptic/numpy-vbench/pull/new/master

> > > if you have specific snippets you would like to benchmark -- just state them
> > > here or send a PR -- I will add them in.

> > > Cheers,

> > > On Tue, 07 May 2013, Daπid wrote:

> > > > On 7 May 2013 13:47, Sebastian Berg <sebastian at sipsolutions.net> wrote:
> > > > > Indexing/assignment was the first thing I thought of too (also because
> > > > > fancy indexing/assignment really could use some speedups...). Other then
> > > > > that maybe some timings for small arrays/scalar math, but that might be
> > > > > nice for that GSoC project.

> > > > Why not going bigger? Ufunc operations on big arrays, CPU and memory bound.

> > > > Also, what about interfacing with other packages? It may increase the
> > > > compiling overhead, but I would like to see Cython in action (say,
> > > > only last version, maybe it can be fixed).
> > > > _______________________________________________
> > > > NumPy-Discussion mailing list
> > > > NumPy-Discussion at scipy.org
> > > > http://mail.scipy.org/mailman/listinfo/numpy-discussion
-- 
Yaroslav O. Halchenko, Ph.D.
http://neuro.debian.net http://www.pymvpa.org http://www.fail2ban.org
Senior Research Associate,     Psychological and Brain Sciences Dept.
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834                       Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik        



More information about the NumPy-Discussion mailing list