Re: [Numpy-discussion] Really cruel draft of vbench setup for NumPy (.add.reduce benchmarks since 2011)

16 Jul 2013

      and to put so far reported findings into some kind of automated form,
please welcome

http://www.onerussian.com/tmp/numpy-vbench/#benchmarks-performance-analysis

This is based on a simple 1-way anova of last 10 commits and some point
in the past where 10 other commits had smallest timing and were significantly
different from the last 10 commits.

"Possible recent" is probably too noisy and not sure if useful -- it should
point to a closest in time (to the latest commits) diff where a
significant excursion from current performance was detected.  So per se it has
nothing to do with the initial detected performance hit, but in some cases
seems still to reasonably locate commits hitting on performance.

Enjoy,

On Tue, 09 Jul 2013, Yaroslav Halchenko wrote:
...
Julian Taylor contributed some benchmarks he was "concerned" about, so
now the collection is even better.
...
I will keep updating tests on the same url:
http://www.onerussian.com/tmp/numpy-vbench/
[it is now running and later I will upload with more commits for higher temporal fidelity]
...
of particular interest for you might be:
some minor consistent recent losses in
http://www.onerussian.com/tmp/numpy-vbench/vb_vb_io.html#strided-assign-floa...
http://www.onerussian.com/tmp/numpy-vbench/vb_vb_io.html#strided-assign-floa...
http://www.onerussian.com/tmp/numpy-vbench/vb_vb_io.html#strided-assign-int1...
http://www.onerussian.com/tmp/numpy-vbench/vb_vb_io.html#strided-assign-int8
...
seems have lost more than 25% of performance throughout the timeline
http://www.onerussian.com/tmp/numpy-vbench/vb_vb_io.html#memcpy-int8
...
"fast" calls to all/any seemed to be hurt twice in their life time now running
*3 times slower* than in 2011 -- inflection points correspond to regressions
and/or their fixes in those functions to bring back performance on "slow"
cases (when array traversal is needed, e.g. on arrays of zeros for any)
...
http://www.onerussian.com/tmp/numpy-vbench/vb_vb_reduce.html#numpy-all-fast
http://www.onerussian.com/tmp/numpy-vbench/vb_vb_reduce.html#numpy-any-fast
...
Enjoy
...
On Mon, 01 Jul 2013, Yaroslav Halchenko wrote:
...
...
FWIW -- updated plots with contribution from Julian Taylor
http://www.onerussian.com/tmp/numpy-vbench-20130701/vb_vb_indexing.html#mmap...
;-)
...
...
On Mon, 01 Jul 2013, Yaroslav Halchenko wrote:
...
...
...
Hi Guys,
...
...
...
not quite the recommendations you expressed,  but here is my ugly
attempt to improve benchmarks coverage:
...
...
...
http://www.onerussian.com/tmp/numpy-vbench-20130701/index.html
...
...
...
initially I also ran those ufunc benchmarks per each dtype separately,
but then resulting webpage is loong which brings my laptop on its knees
by firefox.  So I commented those out for now, and left only "summary"
ones across multiple datatypes.
...
...
...
There is a bug in sphinx which forbids embedding some figures for
vb_random "as is", so pardon that for now...
...
...
...
I have not set cpu affinity of the process (but ran it at nice -10), so  may be
that also contributed to variance of benchmark estimates.  And there probably
could be more of goodies (e.g. gc control etc) to borrow from
https://github.com/pydata/pandas/blob/master/vb_suite/test_perf.py which I have
just discovered to minimize variance.
...
...
...
nothing really interesting was pin-pointed so far, besides that
...
...
...
- svd became a bit faster since few months back ;-)
...
...
...
http://www.onerussian.com/tmp/numpy-vbench-20130701/vb_vb_linalg.html
...
...
...
- isnan (and isinf, isfinite) got improved
...
...
...
http://www.onerussian.com/tmp/numpy-vbench-20130701/vb_vb_ufunc.html#numpy-i...
...
...
...
- right_shift got a miniscule slowdown from what it used to be?
...
...
...
http://www.onerussian.com/tmp/numpy-vbench-20130701/vb_vb_ufunc.html#numpy-r...
...
...
...
As before -- current code of those benchmarks collection is available
at http://github.com/yarikoptic/numpy-vbench/pull/new/master
...
...
...
if you have specific snippets you would like to benchmark -- just state them
here or send a PR -- I will add them in.
...
...
...
Cheers,
...
...
...
On Tue, 07 May 2013, Daπid wrote:
...
...
...
...
On 7 May 2013 13:47, Sebastian Berg  wrote:
...
Indexing/assignment was the first thing I thought of too (also because
fancy indexing/assignment really could use some speedups...). Other then
that maybe some timings for small arrays/scalar math, but that might be
nice for that GSoC project.
...
...
...
...
Why not going bigger? Ufunc operations on big arrays, CPU and memory bound.
...
...
...
...
Also, what about interfacing with other packages? It may increase the
compiling overhead, but I would like to see Cython in action (say,
only last version, maybe it can be fixed).
_______________________________________________
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
-- 
Yaroslav O. Halchenko, Ph.D.
http://neuro.debian.net http://www.pymvpa.org http://www.fail2ban.org
Senior Research Associate,     Psychological and Brain Sciences Dept.
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834                       Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik