I was spending some time trying to track down how to speed up an algorithm that gets called a bunch of times during an optimization. I was startled when I finally figured out that most of the time was wasted by using the built-in pyhton min function. It turns out that in my case, using array.min() (i.e. the method of the Numpy array) is 300-500 times faster than the built-in python min function (i.e. min(array)). So, thank you Travis and everyone who has put so much time into thinking through Numpy and making it fast (as well as making sure it is correct). And to the rest of us, use the Numpy array methods whenever you can. Thanks, Ryan
Ryan Krauss wrote:
I was spending some time trying to track down how to speed up an algorithm that gets called a bunch of times during an optimization. I was startled when I finally figured out that most of the time was wasted by using the built-in pyhton min function. It turns out that in my case, using array.min() (i.e. the method of the Numpy array) is 300-500 times faster than the built-in python min function (i.e. min(array)).
So, thank you Travis and everyone who has put so much time into thinking through Numpy and making it fast (as well as making sure it is correct).
The builtin min function is a bit confusing because it usually does work on NumPy arrays. But, as you've noticed it is always slower because it uses the "generic sequence interface" that NumPy arrays expose. So, it's basically not much faster than a Python loop. In this case you are also being hit by the fact that scalarmath is not yet implemented (it's getting close though...) so the returned array scalars are being compared using the bulky ufunc machinery on each element separately. In Python 2.5 we are going to have the same issues with the new any() and all() functions of Python. -Travis
Travis Oliphant wrote:
In Python 2.5 we are going to have the same issues with the new any() and all() functions of Python.
"Namespaces are one honking great idea -- let's do more of those!" Yet another reason to deprecate import * ! -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
On Wed, 26 Apr 2006, Christopher Barker wrote:
Travis Oliphant wrote:
In Python 2.5 we are going to have the same issues with the new any() and all() functions of Python.
"Namespaces are one honking great idea -- let's do more of those!"
Yet another reason to deprecate import * !
Yep! But it would not work for `min` as there is no such function in numpy. (would we need one?...) Best, Arnd
Moin, On Wed, 26 Apr 2006, Travis Oliphant wrote:
Ryan Krauss wrote:
I was spending some time trying to track down how to speed up an algorithm that gets called a bunch of times during an optimization. I was startled when I finally figured out that most of the time was wasted by using the built-in pyhton min function. It turns out that in my case, using array.min() (i.e. the method of the Numpy array) is 300-500 times faster than the built-in python min function (i.e. min(array)).
So, thank you Travis and everyone who has put so much time into thinking through Numpy and making it fast (as well as making sure it is correct).
The builtin min function is a bit confusing because it usually does work on NumPy arrays. But, as you've noticed it is always slower because it uses the "generic sequence interface" that NumPy arrays expose. So, it's basically not much faster than a Python loop. In this case you are also being hit by the fact that scalarmath is not yet implemented (it's getting close though...) so the returned array scalars are being compared using the bulky ufunc machinery on each element separately.
In Python 2.5 we are going to have the same issues with the new any() and all() functions of Python.
I am just preparing a small text to collect such cases for the wiki. However, I am not sure about a good name for such a page: http://www.scipy.org/Cookbook/Speed http://www.scipy.org/Cookbook/SpeedProblems http://www.scipy.org/Cookbook/Performance ? (As usual, it is easy to start a page, than to properly maintain it. OTOH things like this get lost very quickly, in particular with this nice amount of traffic here). In addition this also relates to - profiling (For example I would like to add the contents of http://mail.enthought.com/pipermail/enthought-dev/2006-January/001075.html to the wiki at some point) - psyco - pyrex - f2py - weave - numexpr - ... Presently much of this is listed in the Cookbook under "Using NumPy With Other Languages (Advanced)", and therefore the above "Python only" issues don't quite fit. Any suggestions? Best, Arnd
Hi Arnd, You could call it PerformanceTips and include some search terms like "speed" in the page so search engines pick them up. Gary R. Arnd Baecker wrote:
I am just preparing a small text to collect such cases for the wiki.
However, I am not sure about a good name for such a page: http://www.scipy.org/Cookbook/Speed http://www.scipy.org/Cookbook/SpeedProblems http://www.scipy.org/Cookbook/Performance ?
I think this is a great idea. We get a lot of these kinds of questions on the list, and the collective wisdom of people here who have really dug into this is really impressive. But, that wisdom does need to be a little easier to find. Speaking of which, I don't always feel like I get trustworthy results out of the profiler, so when I really want to know what is going on I find myself doing this alot: t1=time.time() [block of code here] t2=time.time() [more code] t3=time.time() and then comparing t3-t2 and t2-t1 to narrow down where the code is spending its time. Does anyone have good tips on how to do good profiling? Or is this question so vague and counter-intuitive that I seem silly and I had better come back with a believable example? Thanks, Ryan On 4/27/06, Gary Ruben <gruben@bigpond.net.au> wrote:
Hi Arnd,
You could call it PerformanceTips and include some search terms like "speed" in the page so search engines pick them up.
Gary R.
Arnd Baecker wrote:
I am just preparing a small text to collect such cases for the wiki.
However, I am not sure about a good name for such a page: http://www.scipy.org/Cookbook/Speed http://www.scipy.org/Cookbook/SpeedProblems http://www.scipy.org/Cookbook/Performance ?
------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 _______________________________________________ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
On Thu, 27 Apr 2006, Ryan Krauss wrote:
I think this is a great idea. We get a lot of these kinds of questions on the list, and the collective wisdom of people here who have really dug into this is really impressive. But, that wisdom does need to be a little easier to find.
Speaking of which, I don't always feel like I get trustworthy results out of the profiler, so when I really want to know what is going on I find myself doing this alot:
t1=time.time() [block of code here] t2=time.time() [more code] t3=time.time()
and then comparing t3-t2 and t2-t1 to narrow down where the code is spending its time.
Does anyone have good tips on how to do good profiling? Or is this question so vague and counter-intuitive that I seem silly and I had better come back with a believable example?
Maybe this one is of interest then: http://www.physik.tu-dresden.de/~baecker/comp_talks.html and goto "Python and Co - some recent developments" Quite late in the talk there is an example on Profiling (sorry, it seems that no direct linking is possible) The corresponding files are at http://www.physik.tu-dresden.de/~baecker/talks/pyco/BenchExamples/ Essentially it is an example of using kcachegrind to display the results of hotshot (see also: http://mail.enthought.com/pipermail/enthought-dev/2006-January/001075.html ) Best, Arnd
Thanks,
Ryan
On 4/27/06, Gary Ruben <gruben@bigpond.net.au> wrote:
Hi Arnd,
You could call it PerformanceTips and include some search terms like "speed" in the page so search engines pick them up.
Gary R.
Arnd Baecker wrote:
I am just preparing a small text to collect such cases for the wiki.
However, I am not sure about a good name for such a page: http://www.scipy.org/Cookbook/Speed http://www.scipy.org/Cookbook/SpeedProblems http://www.scipy.org/Cookbook/Performance ?
------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 _______________________________________________ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd_________________________________________... Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
A Dijous 27 Abril 2006 15:40, Ryan Krauss va escriure:
I think this is a great idea. We get a lot of these kinds of questions on the list, and the collective wisdom of people here who have really dug into this is really impressive. But, that wisdom does need to be a little easier to find.
Speaking of which, I don't always feel like I get trustworthy results out of the profiler, so when I really want to know what is going on I find myself doing this alot:
t1=time.time() [block of code here] t2=time.time() [more code] t3=time.time()
and then comparing t3-t2 and t2-t1 to narrow down where the code is spending its time.
Does anyone have good tips on how to do good profiling? Or is this question so vague and counter-intuitive that I seem silly and I had better come back with a believable example?
Well, if you are on Linux, and want to time C extension, then oprofile is a *very* good option. Another profiling tool is Cachegrind, part of Valgrind. It uses the processor emulation of Valgrind to run the executable, and catches all memory accesses for the trace. In addition, you can combine the output of oprofile with Cachegrind. In [3] one can see more info about these and more tools. [1] http://oprofile.sourceforge.net [2] http://kcachegrind.sourceforge.net/ [3] https://uimon.cern.ch/twiki/bin/view/Atlas/OptimisingCode Cheers, --
0,0< Francesc Altet http://www.carabos.com/ V V Cárabos Coop. V. Enjoy Data "-"
On Thu, 27 Apr 2006, Gary Ruben wrote:
Hi Arnd,
You could call it PerformanceTips and include some search terms like "speed" in the page so search engines pick them up.
Alright, I put all I know on this (which is not that much ;-) at http://www.scipy.org/PerformanceTips The pointers to weave/f2py/pyrex/ (ah - psyco is missing) will have to be added. Also the profiling/benchmarking aspect, which is important (actually more important even before thinking about PerformanceTips) needs to be put somewhere, maybe even separately under http://www.scipy.org/BenchmarkingAndProfiling Best, Arnd
Gary R.
Arnd Baecker wrote:
I am just preparing a small text to collect such cases for the wiki.
However, I am not sure about a good name for such a page: http://www.scipy.org/Cookbook/Speed http://www.scipy.org/Cookbook/SpeedProblems http://www.scipy.org/Cookbook/Performance ?
------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 _______________________________________________ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
participants (6)
-
Arnd Baecker -
Christopher Barker -
Francesc Altet -
Gary Ruben -
Ryan Krauss -
Travis Oliphant