Hello NG, in one script, I generated a matrix "dx" which has a (200000, 10) shape. Well, everything is working fine (matrix multiplication, sum, exponentiation etc...), but I'm having problems in squaring all elements of this matrix. Suppose I have: import Numeric import random dx = Numeric.ones((200000,10), Numeric.Float0, savespace=1) dx[:,:] = random.random() td1 = dx*2.0 ---> This takes about 0.2 seconds td2 = dx**2.0 ---> This takes about 6 seconds Why this difference? And why it is so slow the exponentiation? I have tried the same operation in Matlab and it was almost instantaneous. I don't want to stick with Matlab, noting the great job the Numeric developers have done. Is there something that I am missing? Sorry if it is a basic question. Andrea.
andrea_gavana@tin.it writes:
Hello NG,
in one script, I generated a matrix "dx" which has a (200000, 10) shape. Well, everything is working fine (matrix multiplication, sum, exponentiation etc...), but I'm having problems in squaring all elements of this matrix.
Suppose I have:
import Numeric import random
dx = Numeric.ones((200000,10), Numeric.Float0, savespace=1) dx[:,:] = random.random()
It's faster to do import RandomArray dx = RandomArray.random((200000,10)).astype(Numeric.Float0) dx.savespace(1) (it won't use the same RNG as random, though)
td1 = dx*2.0 ---> This takes about 0.2 seconds td2 = dx**2.0 ---> This takes about 6 seconds
Why this difference? And why it is so slow the exponentiation? I have tried the same operation in Matlab and it was almost instantaneous. I don't want to stick with Matlab, noting the great job the Numeric developers have done.
Hmm, I don't see this, with either 23.8 or 24.0b2. What version of Numeric are you using, and on what platform? dx**2 uses the C 'pow' function on each element (so using 2 or 2.0 here doesn't make a difference), so may be your platform pow() is slow. dx*dx should be faster (it takes 80% of the time that dx**2 does on my machine). -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm@physics.mcmaster.ca
Hello David
It's faster to do
import RandomArray
dx = RandomArray.random((200000,10)).astype(Numeric.Float0) dx.savespace(1)
(it won't use the same RNG as random, though)
thanks for the tip.
td1 = dx*2.0 ---> This takes about 0.2 seconds td2 = dx**2.0 ---> This takes about 6 seconds
Hmm, I don't see this, with either 23.8 or 24.0b2. What version of Numeric are you using, and on what platform?
I have currently Numeric 23.8, with numarray 1.3.1. I am on a Windows XP machine, with 1 GB RAM and 3.0 GHz. My Python version is 2.3.4.
dx**2 uses the C 'pow' function on each element (so using 2 or 2.0 here doesn't make a difference), so may be your platform pow() is slow.
Probably that is the reason... but why? Is Windows the problem?
dx*dx should be faster (it takes 80% of the time that dx**2 does on my machine).
On my machine, dx*dx is istantaneous (about 0.1 secons). This are my timings: import time; start = time.time(); td = dx*dx; print time.time() - start 0.121000051498 import time; start = time.time(); td = dx**2; print time.time() - start 4.367000103 !!!!!!!! Does anyone have an explanation for this?!?!? Thank you for your suggestions. Andrea.
andrea_gavana@tin.it writes:
Hello David
Hmm, I don't see this, with either 23.8 or 24.0b2. What version of Numeric are you using, and on what platform?
I have currently Numeric 23.8, with numarray 1.3.1. I am on a Windows XP machine, with 1 GB RAM and 3.0 GHz. My Python version is 2.3.4.
dx**2 uses the C 'pow' function on each element (so using 2 or 2.0 here doesn't make a difference), so may be your platform pow() is slow.
Probably that is the reason... but why? Is Windows the problem?
dx*dx should be faster (it takes 80% of the time that dx**2 does on my machine).
On my machine, dx*dx is istantaneous (about 0.1 secons). This are my timings:
import time; start = time.time(); td = dx*dx; print time.time() - start 0.121000051498
import time; start = time.time(); td = dx**2; print time.time() - start 4.367000103
!!!!!!!!
Could you check the speed of td = exp(2*log(dx)) If it's comparable to dx**2, then your pow() library function isn't any good :-( -- it's not special-casing integer powers. -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm@physics.mcmaster.ca
Hello David,
Could you check the speed of
td = exp(2*log(dx))
If it's comparable to dx**2, then your pow() library function isn't any good :-( -- it's not special-casing integer powers.
I get: start = time.time(); td = Numeric.exp(2*Numeric.log(dx)); print time.time() - start 0.530999898911 Does this make any sense to anyone? I am a bit puzzled... Thanks however for your kind help/suggestions. Andrea.
On Wed, 2005-05-11 at 16:57 -0400, David M. Cooke wrote:
td = exp(2*log(dx))
If it's comparable to dx**2, then your pow() library function isn't any good :-( -- it's not special-casing integer powers.
I believe that this is the case. IIRC, MinGW uses a wrapper around msvcrt.dll's pow() that handles this special case for this very reason. HTH, -Jonathan
Jonathan Brandmeyer <jbrandmeyer@earthlink.net> writes:
On Wed, 2005-05-11 at 16:57 -0400, David M. Cooke wrote:
td = exp(2*log(dx))
If it's comparable to dx**2, then your pow() library function isn't any good :-( -- it's not special-casing integer powers.
I believe that this is the case. IIRC, MinGW uses a wrapper around msvcrt.dll's pow() that handles this special case for this very reason.
Is this is a big problem for others? This is the first report I've seen of the problem on Windows. It may be worth special-casing the power ufunc to handle some non-array powers (x**2, x**3, x**0.5, for instance). -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm@physics.mcmaster.ca
Hello David, >Is this is a big problem for others? This is the first report I've >seen of the problem on Windows. It may be worth special-casing the >power ufunc to handle some non-array powers (x**2, x**3, x**0.5, for >instance). It's not very clear to me what you wrote, but I have the same behavior on a Windows 2000 machine I have at work (here is WinXP), with Numeric 23.7 (here is 23.8), with Python 2.3.5 (here is 2.3.4). These 2 machines do not have anything in common, as far as I know. Regarding what you said (if I have correctly understood), I get: >>>start = time.time(); td = dx*dx; print time.time() - start 0.269999980927 >>> start = time.time(); td = dx**2; print time.time() - start 4.5560002327 >>> start = time.time(); td = dx**3; print time.time() - start 4.5759999752 >>> start = time.time(); td = dx**0.5; print time.time() - start 4.19600009918 >>> start = time.time(); td = sqrt(dx); print time.time() - start 0.301000118256 >>> start = time.time(); td = exp(log(dx)); print time.time() - start 0.792000055313 >>> start = time.time(); td = dx**RandomArray.random((1,)); print time.time() - start 4.14599990845 Whatever power I use, the timing is almost the same. Curious however, how sqrt(dx) and dx**0.5 behave really differently. Thanks for your help. Andrea.
What compiler are you using to build Numeric and Python? Tony andrea_gavana@tin.it said: > Hello David, > >>Is this is a big problem for others? This is the first report I've >>seen of the problem on Windows. It may be worth special-casing the >>power ufunc to handle some non-array powers (x**2, x**3, x**0.5, for >>instance). > > It's not very clear to me what you wrote, but I have the same behavior on > a Windows 2000 machine I have at work (here is WinXP), with Numeric 23.7 > (here is 23.8), with Python 2.3.5 (here is 2.3.4). These 2 machines do not > have anything in common, as far as I know. > Regarding what you said (if I have correctly understood), I get: > >>>>start = time.time(); td = dx*dx; print time.time() - start > 0.269999980927 > >>>> start = time.time(); td = dx**2; print time.time() - start > 4.5560002327 > >>>> start = time.time(); td = dx**3; print time.time() - start > 4.5759999752 > >>>> start = time.time(); td = dx**0.5; print time.time() - start > 4.19600009918 > >>>> start = time.time(); td = sqrt(dx); print time.time() - start > 0.301000118256 > >>>> start = time.time(); td = exp(log(dx)); print time.time() - start > 0.792000055313 > >>>> start = time.time(); td = dx**RandomArray.random((1,)); print >>>> time.time() > - start > 4.14599990845 > > Whatever power I use, the timing is almost the same. Curious however, how > sqrt(dx) and dx**0.5 behave really differently. > > Thanks for your help. > > Andrea. > > > > ------------------------------------------------------- > This SF.Net email is sponsored by Oracle Space Sweepstakes > Want to be the first software developer in space? > Enter now for the Oracle Space Sweepstakes! > http://ads.osdn.com/?ad_ids93&alloc_id281&opÌk > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > >
Hello Tony,
What compiler are you using to build Numeric and Python?
No compiler for Python or Numeric. For Python, I use the binaries that you can find at www.python.org. I don't have any idea which compiler they have used. For Numeric, I downloaded the binaries that you can find at SourceForge. Python 2.3.4/2.3.5, Numeric 23.8/23.7, Windows XP/2000 Andrea.
What is the value of sys.version? For me running the attached script yields: Fedora 3: Python version: 2.3.4 (#1, Feb 2 2005, 12:11:53) [GCC 3.4.2 20041017 (Red Hat 3.4.2-6.fc3)] *: 0.381051 **2: 1.31382 **3: 1.198 **0.5: 1.67896 sqrt(): 0.357029 exp(log): 1.2737 exp(2*log): 2.17751 exp(.5*log): 2.19792 Win2k on vmware 5 under Fedora 3 Python version: 2.3.3 (#51, Feb 16 2004, 04:07:52) [MSC v.1200 32 bit (Intel)] *: 0.391 **2: 1.468 **3: 1.547 **0.5: 1.469 sqrt(): 0.437 exp(log): 1.672 exp(2*log): 2.718 exp(.5*log): 2.734 Odd. andrea_gavana@tin.it said:
Hello Tony,
What compiler are you using to build Numeric and Python?
No compiler for Python or Numeric. For Python, I use the binaries that you can find at www.python.org. I don't have any idea which compiler they have used. For Numeric, I downloaded the binaries that you can find at SourceForge.
Python 2.3.4/2.3.5, Numeric 23.8/23.7, Windows XP/2000
Andrea.
-- ajs: anthony.seward@ieee.org
More data on FC3 Python version: 2.3.4 (#1, Feb 2 2005, 12:11:53) [GCC 3.4.2 20041017 (Red Hat 3.4.2-6.fc3)] ========== Numeric ========== *: 0.379871 **2: 1.17747 **3: 1.1788 **0.5: 1.66172 sqrt(): 0.362143 exp(log): 1.2656 exp(2*log): 2.16179 exp(.5*log): 2.22712 ========== numarray ========== *: 0.0414059 **2: 0.914946 **3: 1.14649 **0.5: 1.43333 sqrt(): 0.10821 exp(log): 1.1466 exp(2*log): 1.17661 exp(.5*log): 1.24503
Hello Tony,
What is the value of sys.version?
D:\MyProjects>python Python 2.3.4 (#53, May 25 2004, 21:17:02) [MSC v.1200 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information.
import sys sys.version '2.3.4 (#53, May 25 2004, 21:17:02) [MSC v.1200 32 bit (Intel)]'
Fedora 3: *: 0.381051 **2: 1.31382 **3: 1.198 **0.5: 1.67896 sqrt(): 0.357029 exp(log): 1.2737 exp(2*log): 2.17751 exp(.5*log): 2.19792
Win2k on vmware 5 under Fedora 3 Python version: 2.3.3 (#51, Feb 16 2004, 04:07:52) [MSC v.1200 32 bit (Intel)] *: 0.391 **2: 1.468 **3: 1.547 **0.5: 1.469 sqrt(): 0.437 exp(log): 1.672 exp(2*log): 2.718 exp(.5*log): 2.734
Win XP (with Numeric 23.8): D:\MyProjects>numeric-stats.py Python version: 2.3.4 (#53, May 25 2004, 21:17:02) [MSC v.1200 32 bit (Intel)] *: 0.12 **2: 5.037 **3: 5.168 **0.5: 5.558 sqrt(): 0.14 exp(log): 0.46 exp(2*log): 1.012 exp(.5*log): 1.021 Win XP (with numarray 1.3.1): D:\MyProjects>numeric-stats.py Python version: 2.3.4 (#53, May 25 2004, 21:17:02) [MSC v.1200 32 bit (Intel)] *: 0.321 **2: 1.502 **3: 1.502 **0.5: 1.492 sqrt(): 1.953 exp(log): 1.993 exp(2*log): 2.043 exp(.5*log): 2.033 I get the same results (scaled on RAM) with my Windows 2000 machine at work. Is there some kind of explanation? A possible solution? Does anyone else get similar differences in timings as I do?
Odd.
*Really* Odd. Thank you for your suggestions. Andrea
On Thu, 2005-05-12 at 14:05 -0400, David M. Cooke wrote:
Jonathan Brandmeyer <jbrandmeyer@earthlink.net> writes:
On Wed, 2005-05-11 at 16:57 -0400, David M. Cooke wrote:
td = exp(2*log(dx))
If it's comparable to dx**2, then your pow() library function isn't any good :-( -- it's not special-casing integer powers.
I believe that this is the case. IIRC, MinGW uses a wrapper around msvcrt.dll's pow() that handles this special case for this very reason.
Is this is a big problem for others?
Its never bugged me enough to do anything about it ;)
This is the first report I've seen of the problem on Windows. It may be worth special-casing the power ufunc to handle some non-array powers (x**2, x**3, x**0.5, for instance).
If you go this route, I suggest you just use the implementation in mingw-runtime - its in the public domain and has had some precision issues debugged out of it already. -Jonathan
andrea_gavana@tin.it wrote:
Hello NG,
in one script, I generated a matrix "dx" which has a (200000, 10) shape. Well, everything is working fine (matrix multiplication, sum, exponentiation etc...), but I'm having problems in squaring all elements of this matrix.
Suppose I have:
import Numeric import random
dx = Numeric.ones((200000,10), Numeric.Float0, savespace=1) dx[:,:] = random.random()
td1 = dx*2.0 ---> This takes about 0.2 seconds td2 = dx**2.0 ---> This takes about 6 seconds
Why this difference? And why it is so slow the exponentiation? I have tried the same operation in Matlab and it was almost instantaneous. I don't want to stick with Matlab, noting the great job the Numeric developers have done.
Is there something that I am missing?
I'm not sure what the problem may be, but I certainly don't see your results: In [22]: dx=RandomArray.random((200000, 10)).astype(Numeric.Float0) In [23]: time td1 = dx*2.0 CPU times: user 0.20 s, sys: 0.06 s, total: 0.26 s Wall time: 0.26 In [24]: time td2 = dx**2.0 CPU times: user 0.30 s, sys: 0.06 s, total: 0.36 s Wall time: 0.36 An increase like this (50% in cpu time) is reasonable for computing a power vs. a plain multiplication. This is using Numeric 23.7 on a Fedora3 box (2.6 GHZ P4). Do you know for a fact that you have enough RAM, and that the 6 secs. don't come from paging (the arrays aren't that large, but you might have other stuff in memory forcing paging). Cheers, f
Hello Fernando,
I'm not sure what the problem may be, but I certainly don't see your results:
In [22]: dx=RandomArray.random((200000, 10)).astype(Numeric.Float0)
In [23]: time td1 = dx*2.0 CPU times: user 0.20 s, sys: 0.06 s, total: 0.26 s Wall time: 0.26
In [24]: time td2 = dx**2.0 CPU times: user 0.30 s, sys: 0.06 s, total: 0.36 s Wall time: 0.36
This is what I get: start = time.time(); td2 = dx**2; print time.time() - start 4.367000103 start = time.time(); td2 = dx*dx; print time.time() - start 0.121000051498 This is on a Windows XP, 1 GB RAM, 3.0 GHz, Python 2.3.4, Numeric 23.8.
This is using Numeric 23.7 on a Fedora3 box (2.6 GHZ P4). Do you know for a fact that you have enough RAM, and that the 6 secs. don't come from paging (the arrays aren't that large, but you might have other stuff in memory forcing paging).
No, the only thing I have opened is PythonWin (the editor), and I am using currently 100 MB of the RAM. I don't know why this happens... Thanks for your help. Andrea.
participants (5)
-
andrea_gavana＠tin.it -
Anthony Joseph Seward -
cookedm＠physics.mcmaster.ca -
Fernando Perez -
Jonathan Brandmeyer