Mailman 3 February 2006 - NumPy-Discussion

mixing arrays and matrices: squeeze yes, flattened no?
by Sven Schreiber Feb. 22, 2006

Feb. 22, 2006

Hi, sometimes I'm still struggling with peculiarities of numpy-arrays vs. numpy-matrices; my latest story goes like this: I first slice out a column of a 2d-numpy-array (a = somearray[:,1]). I can just manage to understand the resulting shape ( == (112,) ). Then I slice a column from a numpy-matrix b = somematrix[:,1] and get the expected (112,1) shape. Then I do what I thought was the easiest thing in the world, I subtract the two vectors: c = a - b I was very surprised by the bug that showed up due to the fact that c.shape == (112,112) !! First conclusion: broadcasting is nice and everything, but here I somehow think that it shouldn't be like this. I like numpy, but this is frustrating. Next, I try to workaround by b.squeeze(). That seems to work, but why is b.squeeze().shape == (1, 112) instead of (112,)? Then I thought maybe b.flattened() does the job, but then I get an error (matrix has no attr flattened). Again, I'm baffled. Could someone please explain? I already own the numpy-book, otherwise I wouldn't even have thought of using those methods, but here it hasn't enlightened me. Second (preliminary) conclusion: I will paranoically use even more asmatrix()-conversions in my code to avoid dealing with those array-beasts ;-) and get column vectors I can trust... Is there a better general advice than to say: "numpy-matrices and numpy-arrays are best kept in separated worlds" ? Thanks for any insights, Sven

3 7

Method to shift elements in an array?
by Zachary Pincus Feb. 22, 2006

Feb. 22, 2006

Hello folks, Does numpy have an built-in mechanism to shift elements along some axis in an array? (e.g. to "roll" [0,1,2,3] by some offset, here 2, to make [2,3,0,1]) If not, what would be the fastest way to implement this in python? Using take? Using slicing and concatenation? Zach

4 4

Re: [Numpy-discussion] Trouble installing Numpy on AIX 5.2.
by mfmorss＠aep.com Feb. 22, 2006

Feb. 22, 2006

This problem was solved by adding "#include <fenv.h>" to ...numpy-0.9.5 /numpy/core/src/umathmodule.c.src Mark F. Morss Principal Analyst, Market Risk American Electric Power mfmorss(a)aep.com Sent by: numpy-discussion- To admin(a)lists.sourc numpy-discussion eforge.net <numpy-discussion(a)lists.sourceforge .net> cc 02/22/2006 09:06 AM Subject [Numpy-discussion] Trouble installing Numpy on AIX 5.2. I built Python successfully on our AIX 5.2 server using "./configure --without-cxx --disable-ipv6". (This uses the native IBM C compiler, invoking it as "cc_r". We have no C++ compiler.) But I have been unable to install Numpy-0.9.5 using the same compiler. After "python setup.py install," the relevant section of the output was: compile options: '-Ibuild/src/numpy/core/src -Inumpy/core/include -Ibuild/src/numpy/core -Inumpy/core/src -Inumpy/core/include -I/pydirectory/include/python2.4 -c' cc_r: build/src/numpy/core/src/umathmodule.c "build/src/numpy/core/src/umathmodule.c", line 2566.25: 1506-045 (S) Undeclared identifier FE_OVERFLOW. "build/src/numpy/core/src/umathmodule.c", line 2584.25: 1506-045 (S) Undeclared identifier FE_OVERFLOW. "build/src/numpy/core/src/umathmodule.c", line 2602.25: 1506-045 (S) Undeclared identifier FE_OVERFLOW. "build/src/numpy/core/src/umathmodule.c", line 2620.25: 1506-045 (S) Undeclared identifier FE_OVERFLOW. "build/src/numpy/core/src/umathmodule.c", line 2638.25: 1506-045 (S) Undeclared identifier FE_OVERFLOW. "build/src/numpy/core/src/umathmodule.c", line 2654.25: 1506-045 (S) Undeclared identifier FE_OVERFLOW. "build/src/numpy/core/src/umathmodule.c", line 2674.25: 1506-045 (S) Undeclared identifier FE_OVERFLOW. "build/src/numpy/core/src/umathmodule.c", line 2694.25: 1506-045 (S) Undeclared identifier FE_OVERFLOW. "build/src/numpy/core/src/umathmodule.c", line 2714.25: 1506-045 (S) Undeclared identifier FE_OVERFLOW. "build/src/numpy/core/src/umathmodule.c", line 2734.25: 1506-045 (S) Undeclared identifier FE_OVERFLOW. "build/src/numpy/core/src/umathmodule.c", line 9307.32: 1506-280 (W) Function argument assignment between types "long double*" and "double*" is not allowed. "build/src/numpy/core/src/umathmodule.c", line 2566.25: 1506-045 (S) Undeclared identifier FE_OVERFLOW. "build/src/numpy/core/src/umathmodule.c", line 2584.25: 1506-045 (S) Undeclared identifier FE_OVERFLOW. "build/src/numpy/core/src/umathmodule.c", line 2602.25: 1506-045 (S) Undeclared identifier FE_OVERFLOW. "build/src/numpy/core/src/umathmodule.c", line 2620.25: 1506-045 (S) Undeclared identifier FE_OVERFLOW. "build/src/numpy/core/src/umathmodule.c", line 2638.25: 1506-045 (S) Undeclared identifier FE_OVERFLOW. "build/src/numpy/core/src/umathmodule.c", line 2654.25: 1506-045 (S) Undeclared identifier FE_OVERFLOW. "build/src/numpy/core/src/umathmodule.c", line 2674.25: 1506-045 (S) Undeclared identifier FE_OVERFLOW. "build/src/numpy/core/src/umathmodule.c", line 2694.25: 1506-045 (S) Undeclared identifier FE_OVERFLOW. "build/src/numpy/core/src/umathmodule.c", line 2714.25: 1506-045 (S) Undeclared identifier FE_OVERFLOW. "build/src/numpy/core/src/umathmodule.c", line 2734.25: 1506-045 (S) Undeclared identifier FE_OVERFLOW. "build/src/numpy/core/src/umathmodule.c", line 9307.32: 1506-280 (W) Function argument assignment between types "long double*" and "double*" is not allowed. error: Command "cc_r -DNDEBUG -O -Ibuild/src/numpy/core/src -Inumpy/core/include -Ibuild/src/numpy/core -Inumpy/core/src -Inumpy/core/include -I/app/sandbox/s625662/installed/include/python2.4 -c build/src/numpy/core/src/umathmodule.c -o build/temp.aix-5.2-2.4 /build/src/numpy/core/src/umathmodule.o" failed with exit status 1 A closely related question is, how can I modify the Numpy setup.py and/or distutils files to enable me to control the options with which cc_r is invoked? I inspected these files, but not being very expert in Python, I could not figure this out. Mark F. Morss Principal Analyst, Market Risk American Electric Power ------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Do you grep through log files for problems? Stop! Download the new AJAX search engine that makes searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642 _______________________________________________ Numpy-discussion mailing list Numpy-discussion(a)lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion

2 2

Trouble installing Numpy on AIX 5.2.
by mfmorss＠aep.com Feb. 22, 2006

Feb. 22, 2006

I built Python successfully on our AIX 5.2 server using "./configure --without-cxx --disable-ipv6". (This uses the native IBM C compiler, invoking it as "cc_r". We have no C++ compiler.) But I have been unable to install Numpy-0.9.5 using the same compiler. After "python setup.py install," the relevant section of the output was: compile options: '-Ibuild/src/numpy/core/src -Inumpy/core/include -Ibuild/src/numpy/core -Inumpy/core/src -Inumpy/core/include -I/pydirectory/include/python2.4 -c' cc_r: build/src/numpy/core/src/umathmodule.c "build/src/numpy/core/src/umathmodule.c", line 2566.25: 1506-045 (S) Undeclared identifier FE_OVERFLOW. "build/src/numpy/core/src/umathmodule.c", line 2584.25: 1506-045 (S) Undeclared identifier FE_OVERFLOW. "build/src/numpy/core/src/umathmodule.c", line 2602.25: 1506-045 (S) Undeclared identifier FE_OVERFLOW. "build/src/numpy/core/src/umathmodule.c", line 2620.25: 1506-045 (S) Undeclared identifier FE_OVERFLOW. "build/src/numpy/core/src/umathmodule.c", line 2638.25: 1506-045 (S) Undeclared identifier FE_OVERFLOW. "build/src/numpy/core/src/umathmodule.c", line 2654.25: 1506-045 (S) Undeclared identifier FE_OVERFLOW. "build/src/numpy/core/src/umathmodule.c", line 2674.25: 1506-045 (S) Undeclared identifier FE_OVERFLOW. "build/src/numpy/core/src/umathmodule.c", line 2694.25: 1506-045 (S) Undeclared identifier FE_OVERFLOW. "build/src/numpy/core/src/umathmodule.c", line 2714.25: 1506-045 (S) Undeclared identifier FE_OVERFLOW. "build/src/numpy/core/src/umathmodule.c", line 2734.25: 1506-045 (S) Undeclared identifier FE_OVERFLOW. "build/src/numpy/core/src/umathmodule.c", line 9307.32: 1506-280 (W) Function argument assignment between types "long double*" and "double*" is not allowed. "build/src/numpy/core/src/umathmodule.c", line 2566.25: 1506-045 (S) Undeclared identifier FE_OVERFLOW. "build/src/numpy/core/src/umathmodule.c", line 2584.25: 1506-045 (S) Undeclared identifier FE_OVERFLOW. "build/src/numpy/core/src/umathmodule.c", line 2602.25: 1506-045 (S) Undeclared identifier FE_OVERFLOW. "build/src/numpy/core/src/umathmodule.c", line 2620.25: 1506-045 (S) Undeclared identifier FE_OVERFLOW. "build/src/numpy/core/src/umathmodule.c", line 2638.25: 1506-045 (S) Undeclared identifier FE_OVERFLOW. "build/src/numpy/core/src/umathmodule.c", line 2654.25: 1506-045 (S) Undeclared identifier FE_OVERFLOW. "build/src/numpy/core/src/umathmodule.c", line 2674.25: 1506-045 (S) Undeclared identifier FE_OVERFLOW. "build/src/numpy/core/src/umathmodule.c", line 2694.25: 1506-045 (S) Undeclared identifier FE_OVERFLOW. "build/src/numpy/core/src/umathmodule.c", line 2714.25: 1506-045 (S) Undeclared identifier FE_OVERFLOW. "build/src/numpy/core/src/umathmodule.c", line 2734.25: 1506-045 (S) Undeclared identifier FE_OVERFLOW. "build/src/numpy/core/src/umathmodule.c", line 9307.32: 1506-280 (W) Function argument assignment between types "long double*" and "double*" is not allowed. error: Command "cc_r -DNDEBUG -O -Ibuild/src/numpy/core/src -Inumpy/core/include -Ibuild/src/numpy/core -Inumpy/core/src -Inumpy/core/include -I/app/sandbox/s625662/installed/include/python2.4 -c build/src/numpy/core/src/umathmodule.c -o build/temp.aix-5.2-2.4 /build/src/numpy/core/src/umathmodule.o" failed with exit status 1 A closely related question is, how can I modify the Numpy setup.py and/or distutils files to enable me to control the options with which cc_r is invoked? I inspected these files, but not being very expert in Python, I could not figure this out. Mark F. Morss Principal Analyst, Market Risk American Electric Power

2 1

Re: [Numpy-discussion] Rookie problems - Why is C-code much faster?
by Tim Hochberg Feb. 22, 2006

Feb. 22, 2006

Mads Ipsen wrote: >On Tue, 21 Feb 2006, Tim Hochberg wrote: > > > >>This all makes perfect sense, but what happended to box? In your >>original code there was a step where you did some mumbo jumbo and box >>and rint. Namely: >> >> > >It's a minor detail, but the reason for this is the following > >Suppose you have a line with length of box = 10 with periodic boundary >conditions (basically this is a circle). Now consider two points x0 = 1 >and x1 = 9 on this line. The shortest distance dx between the points x0 >and x1 is dx = -2 and not 8. The calculation > > dx = x1 - x0 ( = +8) > dx -= box*rint(dx/box) ( = -2) > >will give you the desired result, namely dx = -2. Hope this makes better >sense. Note that fmod() won't work since > > fmod(dx,box) = 8 > > I think you could use some variation like "fmod(dx+box/2, box) - box/2" but rint seems better. >Part of my original post was concerned with the fact, that I initially was >using around() from numpy for this step. This was terribly slow, so I made >some custom changes and added rint() from the C-math library to the numpy >module, giving a speedup factor about 4 for this particular line in the >code. > >Best regards // Mads > > > OK, that all makes sense. You might want to try the following, which factors out all the divisions and half the multiplies by box and produces several fewer temporaries. Note I replaced x**2 with x*x, which for the moment is much faster (I don't know if you've been following the endless yacking about optimizing x**n, but x**2 will get fast eventually). Depending on what you're doing with r2, you may be able to avoid the last multiple by box as well. # Loop over all particles xbox = x/box ybox = y/box for i in range(n-1): dx = xbox[i+1:] - xbox[i] dy = ybox[i+1:] - ybox[i] dx -= rint(dx) dy -= rint(dy) r2 = (dx*dx + dy*dy) r2 *= box Regards, -tim

3 2

$B!!$*BT$A$7$F$*$j$^$9(B
by ラブステーション Feb. 22, 2006

Feb. 22, 2006

厳選したセレブ女性との付き合いはどうですか？濡れに濡れまくる女性を何時でも何処でも完全無料での紹介致します。ご近所検索・写メール検索など、メンバーだけの豊富なサービスがたっぷり！納得いくまでじっくりとお相手探し。スチュワーデス・看護婦・モデル・OL・主婦など大量在籍しております。じっくりと貴方の理想のお相手をゲットして熱い夜を過してください。 http://www.covcov.net?num=112 ただ今、直メ配信サービス実施中！無料登録だけで女性より直接お返事が貴方のメールBOXに届きます♪ 拒否：refuse(a)www.covcov.net

1 0

Problems building numpy w/ ATLAS on Solaris 8
by skip＠pobox.com Feb. 21, 2006

Feb. 21, 2006

After a brief hiatus I'm back to trying to build numpy. Last time I checked in (on the scipy list) I had successfully built ATLAS and created this simple site.cfg file in .../numpy/distutils/site.cfg: [atlas] library_dirs = /home/titan/skipm/src/ATLAS/lib/SunOS_Babe include_dirs = /home/titan/skipm/src/ATLAS/include/SunOS_Babe # for overriding the names of the atlas libraries atlas_libs = lapack, f77blas, cblas, atlas I svn up'd (now at rev 2138), zapped my build directory, then executed "python setup.py build". Just in case it matters, I'm using Python 2.4.2 built with GCC 3.4.1 on Solaris 8. Here's the output of my build attempt: Running from numpy source directory. No module named __svn_version__ F2PY Version 2_2138 blas_opt_info: blas_mkl_info: /home/ink/skipm/src/numpy/numpy/distutils/system_info.py:531: UserWarning: Library error: libs=['mkl', 'vml', 'guide'] found_libs=[] warnings.warn("Library error: libs=%s found_libs=%s" % \ NOT AVAILABLE atlas_blas_threads_info: Setting PTATLAS=ATLAS /home/ink/skipm/src/numpy/numpy/distutils/system_info.py:531: UserWarning: Library error: libs=['lapack', 'f77blas', 'cblas', 'atlas'] found_libs=[] warnings.warn("Library error: libs=%s found_libs=%s" % \ Setting PTATLAS=ATLAS Setting PTATLAS=ATLAS FOUND: libraries = ['lapack', 'f77blas', 'cblas', 'atlas'] library_dirs = ['/home/titan/skipm/src/ATLAS/lib/SunOS_Babe'] language = c include_dirs = ['/opt/include'] ... See my site.cfg file? Why does it affect library_dirs but not include_dirs? running build_src building extension "atlas_version" sources adding 'build/src/atlas_version_0x33c6fa32.c' to sources. running build_ext customize UnixCCompiler customize UnixCCompiler using build_ext building 'atlas_version' extension compiling C sources gcc options: '-fno-strict-aliasing -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -fPIC' compile options: '-I/opt/include -Inumpy/core/include -I/opt/app/g++lib6/python-2.4/include/python2.4 -c' /opt/lang/gcc-3.4/bin/gcc -shared build/temp.solaris-2.8-i86pc-2.4/build/src/atlas_version_0x33c6fa32.o -L/home/titan/skipm/src/ATLAS/lib/SunOS_Babe -llapack -lf77blas -lcblas -latlas -o build/temp.solaris-2.8-i86pc-2.4/atlas_version.so Text relocation remains referenced against symbol offset in file <unknown> 0x7 /home/titan/skipm/src/ATLAS/lib/SunOS_Babe/libatlas.a(ATL_buildinfo.o) <unknown> 0xc /home/titan/skipm/src/ATLAS/lib/SunOS_Babe/libatlas.a(ATL_buildinfo.o) ... bunch of missing <unknown>s elided ... printf 0x1b /home/titan/skipm/src/ATLAS/lib/SunOS_Babe/libatlas.a(ATL_buildinfo.o) printf 0x2d /home/titan/skipm/src/ATLAS/lib/SunOS_Babe/libatlas.a(ATL_buildinfo.o) printf 0x3f /home/titan/skipm/src/ATLAS/lib/SunOS_Babe/libatlas.a(ATL_buildinfo.o) printf 0x51 /home/titan/skipm/src/ATLAS/lib/SunOS_Babe/libatlas.a(ATL_buildinfo.o) ... what's this? can't find printf??? ... ld: fatal: relocations remain against allocatable but non-writable sections collect2: ld returned 1 exit status Text relocation remains referenced against symbol offset in file <unknown> 0x7 /home/titan/skipm/src/ATLAS/lib/SunOS_Babe/libatlas.a(ATL_buildinfo.o) ... more eliding ... printf 0x108 /home/titan/skipm/src/ATLAS/lib/SunOS_Babe/libatlas.a(ATL_buildinfo.o) ld: fatal: relocations remain against allocatable but non-writable sections collect2: ld returned 1 exit status ##### msg: error: Command "/opt/lang/gcc-3.4/bin/gcc -shared build/temp.solaris-2.8-i86pc-2.4/build/src/atlas_version_0x33c6fa32.o -L/home/titan/skipm/src/ATLAS/lib/SunOS_Babe -llapack -lf77blas -lcblas -latlas -o build/temp.solaris-2.8-i86pc-2.4/atlas_version.so" failed with exit status 1 error: Command "/opt/lang/gcc-3.4/bin/gcc -shared build/temp.solaris-2.8-i86pc-2.4/build/src/atlas_version_0x33c6fa32.o -L/home/titan/skipm/src/ATLAS/lib/SunOS_Babe -llapack -lf77blas -lcblas -latlas -o build/temp.solaris-2.8-i86pc-2.4/atlas_version.so" failed with exit status 1 FOUND: libraries = ['lapack', 'f77blas', 'cblas', 'atlas'] library_dirs = ['/home/titan/skipm/src/ATLAS/lib/SunOS_Babe'] language = c define_macros = [('NO_ATLAS_INFO', 2)] include_dirs = ['/opt/include'] Warning: distutils distribution has been initialized, it may be too late to add an extension _dotblas ... How can I initialize things earlier? Does it matter? Traceback (most recent call last): File "setup.py", line 76, in ? setup_package() File "setup.py", line 63, in setup_package config.add_subpackage('numpy') File "/home/ink/skipm/src/numpy/numpy/distutils/misc_util.py", line 592, in add_subpackage config_list = self.get_subpackage(subpackage_name,subpackage_path) File "/home/ink/skipm/src/numpy/numpy/distutils/misc_util.py", line 582, in get_subpackage subpackage_path) File "/home/ink/skipm/src/numpy/numpy/distutils/misc_util.py", line 539, in _get_configuration_from_setup_py config = setup_module.configuration(*args) File "/home/ink/skipm/src/numpy/numpy/setup.py", line 10, in configuration config.add_subpackage('core') File "/home/ink/skipm/src/numpy/numpy/distutils/misc_util.py", line 592, in add_subpackage config_list = self.get_subpackage(subpackage_name,subpackage_path) File "/home/ink/skipm/src/numpy/numpy/distutils/misc_util.py", line 582, in get_subpackage subpackage_path) File "/home/ink/skipm/src/numpy/numpy/distutils/misc_util.py", line 539, in _get_configuration_from_setup_py config = setup_module.configuration(*args) File "numpy/core/setup.py", line 215, in configuration config.add_data_dir('tests') File "/home/ink/skipm/src/numpy/numpy/distutils/misc_util.py", line 636, in add_data_dir self.add_data_files((ds,filenames)) File "/home/ink/skipm/src/numpy/numpy/distutils/misc_util.py", line 702, in add_data_files dist.data_files.extend(data_dict.items()) AttributeError: 'NoneType' object has no attribute 'extend' And finally, a traceback. What's up with that? In parallel with trying to build with ATLAS I'm also trying Travis's suggestion of explicitly setting PTATLAS, ATLAS and BLAS to "None". Numpy builds when I do that. -- Skip Montanaro - skip(a)pobox.com "The values to which people cling most stubbornly under inappropriate conditions are those values that were previously the source of their greatest triumphs over adversity." -- Jared Diamond in "Collapse"

2 4

Shouldn't singular_value_decomposition respect full_matrices?
by Zachary Pincus Feb. 21, 2006

Feb. 21, 2006

numpy.linglg.singular_value_decomposition is defined as follows: def singular_value_decomposition(A, full_matrices=0): return svd(A, 0) Shouldn't that last line be return svd(A, full_matrices) Zach

1 0

Rookie problems - Why is C-code much faster?
by Mads Ipsen Feb. 21, 2006

Feb. 21, 2006

Hey, I am relatively new to python and Numeric, and am currently involved in a project of developing molecular dynamics code written in python and using Numeric for number crunching. During this, I've run into some problems, that I hope I can get some assistance with here. My main issues here are: 1. around() appears to be slow 2. C code appears to be much faster 1. One of the bottle necks in MD is the calculation of the distances between all particle pairs. In MD simulations with periodic boundary conditions, you have to estimate the shortest distance between all the particle pairs in your system. Fx. on a line of length box = 10, the distance dx between two points x0 = 1 and x1 = 9, will be dx = -2 (and NOT dx = 9). One way to do this in numpy is dx = x1 - x0 dx -= around(dx/box) My first observation here, is that around() seems to be very slow. So I looked in umathmodule.c and implemented rint() from the C math library and made my own custom Numeric module. This gives a speed-up by a factor of app. 4 compared to around(). I suggest that rint() is added as a ufunc or is there any concerns here that I am not aware of? 2. Here is the main loop for finding all possible pair distances, which corresponds to a loop over the upper triangular part of a square matrix # Loop over all particles for i in range(n-1): dx = x[i+1:] - x[i] dy = y[i+1:] - y[i] dx -= box*rint(dx/box) dy -= box*rint(dy/box) r2 = dx**2 + dy**2 # square of dist. between points where x and y contain the positions of the particles. A naive implementation in C is // loop over all particles for (int i=0; i<n-1; i++) { for (int j=i+1; j<n; j++) { dx = x[j] - x[i]; dy = y[j] - y[i]; dx -= box*rint(dx/box); dy -= box*rint(dy/box); r2 = dx*dx + dy*dy; } } For n = 2500 particles, i.e. 3123750 particle pairs, the C loop is app. 10 times faster than the Python/Numeric counterpart. This is of course not satisfactory. Are there any things I am doing completely wrong here, basic approaches completely misunderstood, misuses etc? Any suggestions, guidelines, hints are most welcome. Best regards, Mads Ipsen +---------------------------------+-------------------------+ | Mads Ipsen | | | Dept. of Chemistry | phone: +45-35320220 | | H.C.Ørsted Institute | fax: +45-35320322 | | Universitetsparken 5 | | | DK-2100 Copenhagen Ø, Denmark | mpi(a)osc.kiku.dk | +---------------------------------+-------------------------+

8 12

wiki page for record arrays
by Stefan van der Walt Feb. 21, 2006

Feb. 21, 2006

I wrote a short tutorial on using record arrays, which can be found at http://www.scipy.org/ArrayRecords The page is named ArrayRecords instead of RecordArrays, so I'd be glad if someone with priviledges could rename it. Also, please fix any mistakes I might have made. Regards Stéfan

3 5