RFC: is it worth giving a lightning talk at PyCon 2014 on numpy vbench-marking?
![](https://secure.gravatar.com/avatar/634276db962a81d2e8eb008236e8760d.jpg?s=120&d=mm&r=g)
Hi Guys, PyCon 2014 will be just around the corner from where I am, so I decided to attend. Being lazy (or busy) I haven't submitted any big talk but thinking to submit few lightning talks (just 5 min and 400 characters abstract limit), and I think it might be worth letting people know about my little project. I would really appreciate your sincere feedback (e.g. "not worth it" would be valuable too). Here is the title/abstract numpy-vbench -- speed benchmarks for NumPy http://yarikoptic.github.io/numpy-vbench provides collection of speed performance benchmarks for NumPy. Benchmarking of multiple maintenance and current development branches allows not only to timely react to new performance regressions, but also to compare NumPy performance across releases. Your contributions would help to guarantee that your code does not become slower with a new NumPy release. btw -- fresh results are here http://yarikoptic.github.io/numpy-vbench/ . I have tuned benchmarking so it now reflects the best performance across multiple executions of the whole battery, thus eliminating spurious variance if estimate is provided from a single point in time. Eventually I expect many of those curves to become even "cleaner". -- Yaroslav O. Halchenko, Ph.D. http://neuro.debian.net http://www.pymvpa.org http://www.fail2ban.org Senior Research Associate, Psychological and Brain Sciences Dept. Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755 Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419 WWW: http://www.linkedin.com/in/yarik
![](https://secure.gravatar.com/avatar/97c543aca1ac7bbcfb5279d0300c8330.jpg?s=120&d=mm&r=g)
On Tue, Oct 15, 2013 at 5:13 PM, Yaroslav Halchenko <lists@onerussian.com> wrote:
What do you have to lose?
On another note, what do you think of moving the vbench benchmarks into the main numpy tree? We already require everyone who submits a bug fix to add a test; there are a bunch of speed enhancements coming in these days and it would be nice if we had some way to ask people to submit a benchmark along with each one so that we know that the enhancement stays enhanced... -n
![](https://secure.gravatar.com/avatar/634276db962a81d2e8eb008236e8760d.jpg?s=120&d=mm&r=g)
On Tue, 15 Oct 2013, Nathaniel Smith wrote:
numpy-vbench -- speed benchmarks for NumPy
What do you have to lose?
time? ;)
btw -- fresh results are here http://yarikoptic.github.io/numpy-vbench/ .
I would be thrilled to do so, but #1 I would need time I currently do not have to harmonize my setup for such inclusion #2 I am still not sure if my tune ups on top of vbench should get merged as is https://github.com/pydata/vbench/pull/33 #3 vbench itself might go through a considerable RF if someone finds some spare time: https://github.com/pydata/vbench/issues/34 If merge for the PR #2 would happen, and test_perf.py from pandas would get integrated within vbench -- then it might be logical to think about #1 regardless of the ultimate resolution to #3 ;) Whenever I have some time for it I would try to push in that direction, and I do not think it is unfeasible to see all of that done actually by PyCon 2014 (so the talk will have a different url ;) ), I just can't promise ATM. -- Yaroslav O. Halchenko, Ph.D. http://neuro.debian.net http://www.pymvpa.org http://www.fail2ban.org Senior Research Associate, Psychological and Brain Sciences Dept. Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755 Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419 WWW: http://www.linkedin.com/in/yarik
![](https://secure.gravatar.com/avatar/634276db962a81d2e8eb008236e8760d.jpg?s=120&d=mm&r=g)
On Tue, 15 Oct 2013, Nathaniel Smith wrote:
What do you have to lose?
btw -- fresh results are here http://yarikoptic.github.io/numpy-vbench/ .
On this positive note (it is boring to start a new thread, isn't it?) -- would you be interested in me transfering numpy-vbench over to github.com/numpy ? as of today, plots on http://yarikoptic.github.io/numpy-vbench should be updating 24x7 (just a loop, thus no time guarantee after you submit new changes). Besides benchmarking new benchmarks (your PRs would still be very welcome, so far it was just me and Julian T) and revisions, that process also goes through a random sample of existing previously benchmarked revisions and re-runs the benchmarks thus improving upon the ultimate 'min' timing performance. So you can see already that many plots became much 'cleaner', although now there might be a bit of bias in estimates for recent revisions since they hadn't accumulated yet as many of 'independent runs' as older revisions. -- Yaroslav O. Halchenko, Ph.D. http://neuro.debian.net http://www.pymvpa.org http://www.fail2ban.org Senior Research Associate, Psychological and Brain Sciences Dept. Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755 Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419 WWW: http://www.linkedin.com/in/yarik
![](https://secure.gravatar.com/avatar/97c543aca1ac7bbcfb5279d0300c8330.jpg?s=120&d=mm&r=g)
On Sun, Nov 24, 2013 at 5:32 PM, Yaroslav Halchenko <lists@onerussian.com> wrote:
If you mean just moving the existing git repo under the numpy organization, like github.com/numpy/numpy-vbench, then I'm not sure how much difference it would make really. What seems like it'd be really useful though would be if the code could move into the main numpy tree, so that people could submit both benchmarks and optimizations within a single PR.
Cool! -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org
![](https://secure.gravatar.com/avatar/634276db962a81d2e8eb008236e8760d.jpg?s=120&d=mm&r=g)
On Sun, 24 Nov 2013, Nathaniel Smith wrote:
I just thought about better visibility for this little project which is now under some shmooptic/ on github ;)
concur! ;) we would need to finally look into - RFing vb_suite/test_perf.py from within pandas over into vbench - making our changes into stock vbench - moving numpy-vbench under numpy tree -- Yaroslav O. Halchenko, Ph.D. http://neuro.debian.net http://www.pymvpa.org http://www.fail2ban.org Senior Research Associate, Psychological and Brain Sciences Dept. Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755 Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419 WWW: http://www.linkedin.com/in/yarik
![](https://secure.gravatar.com/avatar/c0da24f75f763b6bac90b519064f30b3.jpg?s=120&d=mm&r=g)
On 25.11.2013 02:32, Yaroslav Halchenko wrote:
using the vbench I created a comparison of gcc and clang with different options. Cliffnotes: * gcc -O2 performs 5-10% better than -O3 in most benchmarks, except in a few select cases where the vectorizer does its magic * gcc and clang are very close in performance, but the cases where a compiler wins by a large margin its mostly gcc that wins I have collected some interesting plots on this notebook: http://nbviewer.ipython.org/7646615
![](https://secure.gravatar.com/avatar/18a30ce6d84de6ce5c11ce006d10f616.jpg?s=120&d=mm&r=g)
Have you tried on an Intel CPU? I have both a i5 quad core and an i7 octo core where I could run it over the weekend. One may expect some compiler magic taking advantage of the advanced features, specially the i7. /David On Nov 25, 2013 8:16 PM, "Julian Taylor" <jtaylor.debian@googlemail.com> wrote:
![](https://secure.gravatar.com/avatar/c0da24f75f763b6bac90b519064f30b3.jpg?s=120&d=mm&r=g)
there isn't that much code in numpy that profits from modern x86 instruction sets, even the simple arithmetic loops are strided and thus unvectorizable by the compiler. They have been vectorized manually in 1.8 using sse2 and it is on my todo list to add runtime detected avx support. On 26.11.2013 09:57, Daπid wrote:
![](https://secure.gravatar.com/avatar/634276db962a81d2e8eb008236e8760d.jpg?s=120&d=mm&r=g)
ok -- since no negative feedback received -- submitted as is. I will let you know when it gets rejected or accepted. cheers, On Tue, 15 Oct 2013, Yaroslav Halchenko wrote:
Hi Guys,
numpy-vbench -- speed benchmarks for NumPy
btw -- fresh results are here http://yarikoptic.github.io/numpy-vbench/ .
![](https://secure.gravatar.com/avatar/95198572b00e5fbcd97fb5315215bf7a.jpg?s=120&d=mm&r=g)
On Tue, Oct 15, 2013 at 10:36 AM, Yaroslav Halchenko <lists@onerussian.com>wrote:
ok -- since no negative feedback received -- submitted as is. I will let you know when it gets rejected or accepted.
Let me know if it's accepted: I'll be keynoting at PyCon'14, and since my focus will obviously be scientific computing, I'd be happy to mention a few talks from our community in my slides. Cheers, f -- Fernando Perez (@fperez_org; http://fperez.org) fperez.net-at-gmail: mailing lists only (I ignore this when swamped!) fernando.perez-at-berkeley: contact me here for any direct mail
![](https://secure.gravatar.com/avatar/634276db962a81d2e8eb008236e8760d.jpg?s=120&d=mm&r=g)
On Mon, 25 Nov 2013, Fernando Perez wrote:
thank you Fernando! So far I have heard nothing from PyCon people. Cheers! -- Yaroslav O. Halchenko, Ph.D. http://neuro.debian.net http://www.pymvpa.org http://www.fail2ban.org Senior Research Associate, Psychological and Brain Sciences Dept. Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755 Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419 WWW: http://www.linkedin.com/in/yarik
![](https://secure.gravatar.com/avatar/97c543aca1ac7bbcfb5279d0300c8330.jpg?s=120&d=mm&r=g)
On Tue, Oct 15, 2013 at 5:13 PM, Yaroslav Halchenko <lists@onerussian.com> wrote:
What do you have to lose?
On another note, what do you think of moving the vbench benchmarks into the main numpy tree? We already require everyone who submits a bug fix to add a test; there are a bunch of speed enhancements coming in these days and it would be nice if we had some way to ask people to submit a benchmark along with each one so that we know that the enhancement stays enhanced... -n
![](https://secure.gravatar.com/avatar/634276db962a81d2e8eb008236e8760d.jpg?s=120&d=mm&r=g)
On Tue, 15 Oct 2013, Nathaniel Smith wrote:
numpy-vbench -- speed benchmarks for NumPy
What do you have to lose?
time? ;)
btw -- fresh results are here http://yarikoptic.github.io/numpy-vbench/ .
I would be thrilled to do so, but #1 I would need time I currently do not have to harmonize my setup for such inclusion #2 I am still not sure if my tune ups on top of vbench should get merged as is https://github.com/pydata/vbench/pull/33 #3 vbench itself might go through a considerable RF if someone finds some spare time: https://github.com/pydata/vbench/issues/34 If merge for the PR #2 would happen, and test_perf.py from pandas would get integrated within vbench -- then it might be logical to think about #1 regardless of the ultimate resolution to #3 ;) Whenever I have some time for it I would try to push in that direction, and I do not think it is unfeasible to see all of that done actually by PyCon 2014 (so the talk will have a different url ;) ), I just can't promise ATM. -- Yaroslav O. Halchenko, Ph.D. http://neuro.debian.net http://www.pymvpa.org http://www.fail2ban.org Senior Research Associate, Psychological and Brain Sciences Dept. Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755 Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419 WWW: http://www.linkedin.com/in/yarik
![](https://secure.gravatar.com/avatar/634276db962a81d2e8eb008236e8760d.jpg?s=120&d=mm&r=g)
On Tue, 15 Oct 2013, Nathaniel Smith wrote:
What do you have to lose?
btw -- fresh results are here http://yarikoptic.github.io/numpy-vbench/ .
On this positive note (it is boring to start a new thread, isn't it?) -- would you be interested in me transfering numpy-vbench over to github.com/numpy ? as of today, plots on http://yarikoptic.github.io/numpy-vbench should be updating 24x7 (just a loop, thus no time guarantee after you submit new changes). Besides benchmarking new benchmarks (your PRs would still be very welcome, so far it was just me and Julian T) and revisions, that process also goes through a random sample of existing previously benchmarked revisions and re-runs the benchmarks thus improving upon the ultimate 'min' timing performance. So you can see already that many plots became much 'cleaner', although now there might be a bit of bias in estimates for recent revisions since they hadn't accumulated yet as many of 'independent runs' as older revisions. -- Yaroslav O. Halchenko, Ph.D. http://neuro.debian.net http://www.pymvpa.org http://www.fail2ban.org Senior Research Associate, Psychological and Brain Sciences Dept. Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755 Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419 WWW: http://www.linkedin.com/in/yarik
![](https://secure.gravatar.com/avatar/97c543aca1ac7bbcfb5279d0300c8330.jpg?s=120&d=mm&r=g)
On Sun, Nov 24, 2013 at 5:32 PM, Yaroslav Halchenko <lists@onerussian.com> wrote:
If you mean just moving the existing git repo under the numpy organization, like github.com/numpy/numpy-vbench, then I'm not sure how much difference it would make really. What seems like it'd be really useful though would be if the code could move into the main numpy tree, so that people could submit both benchmarks and optimizations within a single PR.
Cool! -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org
![](https://secure.gravatar.com/avatar/634276db962a81d2e8eb008236e8760d.jpg?s=120&d=mm&r=g)
On Sun, 24 Nov 2013, Nathaniel Smith wrote:
I just thought about better visibility for this little project which is now under some shmooptic/ on github ;)
concur! ;) we would need to finally look into - RFing vb_suite/test_perf.py from within pandas over into vbench - making our changes into stock vbench - moving numpy-vbench under numpy tree -- Yaroslav O. Halchenko, Ph.D. http://neuro.debian.net http://www.pymvpa.org http://www.fail2ban.org Senior Research Associate, Psychological and Brain Sciences Dept. Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755 Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419 WWW: http://www.linkedin.com/in/yarik
![](https://secure.gravatar.com/avatar/c0da24f75f763b6bac90b519064f30b3.jpg?s=120&d=mm&r=g)
On 25.11.2013 02:32, Yaroslav Halchenko wrote:
using the vbench I created a comparison of gcc and clang with different options. Cliffnotes: * gcc -O2 performs 5-10% better than -O3 in most benchmarks, except in a few select cases where the vectorizer does its magic * gcc and clang are very close in performance, but the cases where a compiler wins by a large margin its mostly gcc that wins I have collected some interesting plots on this notebook: http://nbviewer.ipython.org/7646615
![](https://secure.gravatar.com/avatar/18a30ce6d84de6ce5c11ce006d10f616.jpg?s=120&d=mm&r=g)
Have you tried on an Intel CPU? I have both a i5 quad core and an i7 octo core where I could run it over the weekend. One may expect some compiler magic taking advantage of the advanced features, specially the i7. /David On Nov 25, 2013 8:16 PM, "Julian Taylor" <jtaylor.debian@googlemail.com> wrote:
![](https://secure.gravatar.com/avatar/c0da24f75f763b6bac90b519064f30b3.jpg?s=120&d=mm&r=g)
there isn't that much code in numpy that profits from modern x86 instruction sets, even the simple arithmetic loops are strided and thus unvectorizable by the compiler. They have been vectorized manually in 1.8 using sse2 and it is on my todo list to add runtime detected avx support. On 26.11.2013 09:57, Daπid wrote:
![](https://secure.gravatar.com/avatar/634276db962a81d2e8eb008236e8760d.jpg?s=120&d=mm&r=g)
ok -- since no negative feedback received -- submitted as is. I will let you know when it gets rejected or accepted. cheers, On Tue, 15 Oct 2013, Yaroslav Halchenko wrote:
Hi Guys,
numpy-vbench -- speed benchmarks for NumPy
btw -- fresh results are here http://yarikoptic.github.io/numpy-vbench/ .
![](https://secure.gravatar.com/avatar/95198572b00e5fbcd97fb5315215bf7a.jpg?s=120&d=mm&r=g)
On Tue, Oct 15, 2013 at 10:36 AM, Yaroslav Halchenko <lists@onerussian.com>wrote:
ok -- since no negative feedback received -- submitted as is. I will let you know when it gets rejected or accepted.
Let me know if it's accepted: I'll be keynoting at PyCon'14, and since my focus will obviously be scientific computing, I'd be happy to mention a few talks from our community in my slides. Cheers, f -- Fernando Perez (@fperez_org; http://fperez.org) fperez.net-at-gmail: mailing lists only (I ignore this when swamped!) fernando.perez-at-berkeley: contact me here for any direct mail
![](https://secure.gravatar.com/avatar/634276db962a81d2e8eb008236e8760d.jpg?s=120&d=mm&r=g)
On Mon, 25 Nov 2013, Fernando Perez wrote:
thank you Fernando! So far I have heard nothing from PyCon people. Cheers! -- Yaroslav O. Halchenko, Ph.D. http://neuro.debian.net http://www.pymvpa.org http://www.fail2ban.org Senior Research Associate, Psychological and Brain Sciences Dept. Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755 Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419 WWW: http://www.linkedin.com/in/yarik
participants (6)
-
Aron Ahmadia
-
Daπid
-
Fernando Perez
-
Julian Taylor
-
Nathaniel Smith
-
Yaroslav Halchenko