Mailman 3 October 2008 - NumPy-Discussion

installation problem on Red Hat
by Christoph Göbl 01 Nov '08

01 Nov '08

dear members, I'm very sorry to bother you with a (hopefully) simple problem... I need pyhton and the numerical package to run another program. I installed Python, it works fine. But I can't install the numpy package. To install the oder Numeric package was no problem, but I need the newer numpy... after "python setup.py install" I get an error message after sime while: ... compiling C sources C compiler: gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC compile options: '-DNO_ATLAS_INFO=1 -Inumpy/core/include -Ibuild/src.linux-x86_64-2.5/numpy/core/include/numpy -Inumpy/core/src -Inumpy/core/include -I/usr/local/include/python2.5 -c' /usr/local/bin/g77 -g -Wall -g -Wall -shared build/temp.linux-x86_64-2.5/numpy/linalg/lapack_litemodule.o build/temp.linux-x86_64-2.5/numpy/linalg/python_xerbla.o -L/usr/lib -llapack -lblas -lg2c -o build/lib.linux-x86_64-2.5/numpy/linalg/lapack_lite.so /usr/bin/ld: skipping incompatible /usr/lib/liblapack.so when searching for -llapack /usr/bin/ld: skipping incompatible /usr/lib/liblapack.a when searching for -llapack /usr/bin/ld: skipping incompatible /usr/lib/libblas.so when searching for -lblas /usr/bin/ld: skipping incompatible /usr/lib/libblas.a when searching for -lblas /usr/bin/ld: skipping incompatible /usr/lib/libg2c.so when searching for -lg2c /usr/bin/ld: skipping incompatible /usr/lib/libg2c.a when searching for -lg2c /usr/bin/ld: skipping incompatible /usr/lib/libfrtbegin.a when searching for -lfrtbegin /usr/bin/ld: skipping incompatible /usr/lib/libg2c.so when searching for -lg2c /usr/bin/ld: skipping incompatible /usr/lib/libg2c.a when searching for -lg2c /usr/bin/ld: cannot find -lgcc_s collect2: ld returned 1 exit status /usr/bin/ld: skipping incompatible /usr/lib/liblapack.so when searching for -llapack /usr/bin/ld: skipping incompatible /usr/lib/liblapack.a when searching for -llapack /usr/bin/ld: skipping incompatible /usr/lib/libblas.so when searching for -lblas /usr/bin/ld: skipping incompatible /usr/lib/libblas.a when searching for -lblas /usr/bin/ld: skipping incompatible /usr/lib/libg2c.so when searching for -lg2c /usr/bin/ld: skipping incompatible /usr/lib/libg2c.a when searching for -lg2c /usr/bin/ld: skipping incompatible /usr/lib/libfrtbegin.a when searching for -lfrtbegin /usr/bin/ld: skipping incompatible /usr/lib/libg2c.so when searching for -lg2c /usr/bin/ld: skipping incompatible /usr/lib/libg2c.a when searching for -lg2c /usr/bin/ld: cannot find -lgcc_s collect2: ld returned 1 exit status error: Command "/usr/local/bin/g77 -g -Wall -g -Wall -shared build/temp.linux-x86_64-2.5/numpy/linalg/lapack_litemodule.o build/temp.linux-x86_64-2.5/numpy/linalg/python_xerbla.o -L/usr/lib -llapack -lblas -lg2c -o build/lib.linux-x86_64-2.5/numpy/linalg/lapack_lite.so" failed with exit status 1 ... I don't know, what I can do... On a Suse 10.2 it ran easily, but on the other computer, Red Hat 3.4.3_9, X86_64, gcc 3.4.3 there is always this error message. I read in another forum, that a person solved a similar problem using unsetenv ldflags But - sorry I'm a Newbie in Linux and Python - there the installation was on another platform I think. Anyway, maybe it's a linking problem? thank you very much for any thoughts you may waste on my problems... best regards, Christoph _________________________________________________________________ Hol dir 30 kostenlose Emoticons für deinen Windows Live Messenger http://www.livemessenger-emoticons.com/funfamily/de-at/

7 8

Numpy matrix multiplication slow even though ATLAS linked
by Jan-Willem van de Meent 31 Oct '08

31 Oct '08

Dear all, This is my first post to this list. I am having perfomance issues with with numpy/atlas. Doing dot(a,a) for a 2000x2000 matrix takes about 1m40s, even though numpy is appears to link to my atlas libraries: I did a quick benchmark by running the following script: #! /usr/bin/env python import numpy import time try: import numpy.core._dotblas print 'Using ATLAS:' except ImportError: print 'No ATLAS:' t = time.time() x = numpy.random.random((1000,1000)) y = numpy.random.random((1000,1000)) z = numpy.dot(x, y) print time.time()-t My laptop is a Dell D620 Core Duo T2300 1.66 Ghz, running Archlinux with GCC 4.3.2, atlas 3.8.2, python 2.5.2 and numpy 1.2.1. Output of the script above is: Using ATLAS: 7.99549412727 A department desktop PC, Pentium D 3.00 Ghz, running Scientific Linux, with GCC 4.1.2, atlas 3.7.30, python 2.5.1 and numpy 1.1.0, runs this test 24 times faster: Using ATLAS: 0.337520122528 So even though _dotblas.so exists, matrix multiplication appears to run at pretty much the same speed as if atlas were not available. Running ldd on _dotblas.so suggests that numpy is indeed linking to the atlas libs: ldd /usr/lib/python2.5/site-packages/numpy/core/_dotblas.so linux-gate.so.1 => (0xb7fcf000) libatlas.so => /usr/lib/libatlas.so (0xb7cb5000) liblapack.so => /usr/lib/liblapack.so (0xb77ab000) libcblas.so => /usr/lib/libcblas.so (0xb778b000) libf77blas.so => /usr/lib/libf77blas.so (0xb776f000) libpython2.5.so.1.0 => /usr/lib/libpython2.5.so.1.0 (0xb7630000) libpthread.so.0 => /lib/libpthread.so.0 (0xb7618000) libc.so.6 => /lib/libc.so.6 (0xb74d6000) libm.so.6 => /lib/libm.so.6 (0xb74b0000) libgfortran.so.3 => /usr/lib/libgfortran.so.3 (0xb73ff000) libdl.so.2 => /lib/libdl.so.2 (0xb73fb000) libutil.so.1 => /lib/libutil.so.1 (0xb73f6000) /lib/ld-linux.so.2 (0xb7fd0000) Something appears to be going wrong in the compilation process. In particular, the compile process appears unable to determine the version of the atlas libs installed (I get NO_ATLAS_INFO). I've pasted some snippets from the build output which may be relevant below. Can anyone help me figure this out? Many thanks in advance, Jan-Willem -- Jan-Willem van de Meent Research Student Goldstein Lab, DAMTP University of Cambridge -- Snippets from setup.py output ... atlas_threads_info: Setting PTATLAS=ATLAS libraries lapack_atlas not found in /usr/lib numpy.distutils.system_info.atlas_threads_info Setting PTATLAS=ATLAS Setting PTATLAS=ATLAS FOUND: libraries = ['lapack', 'atlas', 'lapack', 'cblas', 'f77blas', 'ptcblas', 'ptf77blas'] library_dirs = ['/usr/lib'] language = f77 include_dirs = ['/usr/include'] ... customize Gnu95FCompiler using config compiling '_configtest.c': /* This file is generated from numpy/distutils/system_info.py */ void ATL_buildinfo(void); int main(void) { ATL_buildinfo(); return 0; } C compiler: gcc -pthread -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -march=i686 -mtune=generic -O2 -pipe -fPIC compile options: '-c' gcc: _configtest.c gcc -pthread _configtest.o -L/usr/lib -llapack -latlas -llapack -lcblas -lf77blas -lptcblas -lptf77blas -o _configtest /usr/bin/ld: _configtest: hidden symbol `__powidf2' in /usr/lib/gcc/i686-pc-linux-gnu/4.3.2/libgcc.a(_powidf2.o) is referenced by DSO /usr/bin/ld: final link failed: Nonrepresentable section on output collect2: ld returned 1 exit status /usr/bin/ld: _configtest: hidden symbol `__powidf2' in /usr/lib/gcc/i686-pc-linux-gnu/4.3.2/libgcc.a(_powidf2.o) is referenced by DSO /usr/bin/ld: final link failed: Nonrepresentable section on output collect2: ld returned 1 exit status failure. removing: _configtest.c _configtest.o Status: 255 Output: FOUND: libraries = ['lapack', 'atlas', 'lapack', 'cblas', 'f77blas', 'ptcblas', 'ptf77blas'] library_dirs = ['/usr/lib'] language = f77 define_macros = [('NO_ATLAS_INFO', 2)] include_dirs = ['/usr/include']

4 7

newaxis and contiguity
by David Cournapeau 31 Oct '08

31 Oct '08

Hi, I was wondering about the following behavior: a = np.random.randn(10) b = a[:. np.newaxis] print b.flags # F_CONTIGUOUS and C_CONTIGUOUS are both false Is it for simplicity purpose, or is there a more fundamental reason why a rank 2, 'column' vector can't be F_CONTIGUOUS in this case ? Does the operation involving newaxis modifies the data buffer ? thanks, David

2 1

indexing numpy arrays in extensions created with ext_tools
by Søren Nielsen 31 Oct '08

31 Oct '08

Hi, I'm trying to make a weave python extension to use in my program. I already did it in inline, but that doesn't work with py2exe (needs compiler), so I'm creating extensions instead using ext_tools. Is there a way I can use blitz with ext_tools? so that I can refer to numpy arrays like a(x,y) in the c code?? I have alot of 2D arrays that needs indexing in the C code.. right now I can only index one dimension since the first is always zero for some reason.. so if I write a[1,1] in the c code, I still get returned the a[0,1] Any help is appreciated! Thanks, Soren Some silly test code: ----------------------------------------- mod = ext_tools.ext_module('test_ext') a = zeros((2,2)) x = 1 y = 0 test_code = """ py::tuple args(1); if(a[x,y] == 1) { args[0] = 1; } else { args[0] = 0; } args[0] = a[x,y]; return_val = args; """ ravg = ext_tools.ext_function('test', test_code, ['a', 'x', 'y']) mod.add_function(ravg) mod.compile()

1 0

ANN: PyTables 2.1rc1 is ready to be tested
by Francesc Alted 31 Oct '08

31 Oct '08

============================ Announcing PyTables 2.1rc1 ============================ PyTables is a library for managing hierarchical datasets and designed to efficiently cope with extremely large amounts of data with support for full 64-bit file addressing. PyTables runs on top of the HDF5 library and NumPy package for achieving maximum throughput and convenient use. In PyTables 2.1rc1 many new features and a handful of bugs have been addressed. This is a release candidate, so, in addition to the tarball, binaries for Windows are provided too. Also, the API has been frozen and you should only expect bug fixes and documentation improvements for 2.1 final (due to release in a couple of weeks now). This version introduces important improvements, like much faster node opening, creation or navigation, a file-based way to fine-tune the different PyTables parameters (fully documented now in a new appendix of the UG) and support for multidimensional atoms in EArray/CArray objects. Regarding the Pro edition, 3 different kind of indexes have been added so that the user can choose the best for her needs. Also, and due to the introduction of the concept of chunkmaps in OPSI, the responsiveness of complex queries with low selectivity has improved quite a lot. And last but not least, it is possible now to sort completely tables that are ordered by a specific field, with no practical limit in size (up to 2**48 rows, that is, around 281 trillion of rows). More info in: http://www.pytables.org/moin/PyTablesPro#WhatisnewinforthcomingPyTablesPro2… In case you want to know more in detail what has changed in this version, have a look at ``RELEASE_NOTES.txt`` in the tarball. Find the HTML version for this document at: http://www.pytables.org/moin/ReleaseNotes/Release_2.1rc1 You can download a source package of the version 2.1rc1 with generated PDF and HTML docs and binaries for Windows from http://www.pytables.org/download/preliminary Finally, and for the first time, an evaluation version for PyTables Pro has been made available in: http://www.pytables.org/download/evaluation Please read the evaluation license for terms of use of this version: http://www.pytables.org/moin/PyTablesProEvaluationLicense For an on-line version of the manual, visit: http://www.pytables.org/docs/manual-2.1rc1 Resources ========= Go to the PyTables web site for more details: http://www.pytables.org About the HDF5 library: http://hdfgroup.org/HDF5/ About NumPy: http://numpy.scipy.org/ Acknowledgments =============== Thanks to many users who provided feature improvements, patches, bug reports, support and suggestions. See the ``THANKS`` file in the distribution package for a (incomplete) list of contributors. Many thanks also to SourceForge who have helped to make and distribute this package! And last, but not least thanks a lot to the HDF5 and NumPy (and numarray!) makers. Without them PyTables simply would not exist. Share your experience ===================== Let us know of any bugs, suggestions, gripes, kudos, etc. you may have. ---- **Enjoy data!** -- The PyTables Team

1 0

Re: [Numpy-discussion] unable to build atlas and lapack correctly
by Alexandre Lacoste 31 Oct '08

31 Oct '08

Thanks for your answer > You can simply use the sse2 version of atlas: SSE2 is available on any > x86_64 CPU. > Are you sure ? I can't find it... http://packages.ubuntu.com/search?arch=amd64&searchon=names&keywords=atlas%… > > Note that on Ubuntu, you can link to libatals3fg-base (the gfortran, > basic version of ATLAS), and after building numpy, installing atlas for > SSE2 and still get the speed up, thanks to the hwcap capability of the > GNU loader. The hwcap means that when the loaded code (numpy) request a > library (atlas), it will first look for a specially optimized one in > some directory (/usr/lib/sse2, for example), and will use the default > one if the optimized one is not there. > That's what I tried in the first place but I must have done something wrong on that part too ;) > You made a mistake when building lapack and/or atlas. Those are > difficult to build correctly, you should avoid build them by yourself as > much as possible, > I did it a couple of time in the past ... even on 64 bits machines... I guess I was lucky back then. I really need to get it work. I'll give it a couple of other tries. It is only annoying to have to wait half an hour each time. If you can give me a couple of other hints, maybe I'll target the problem faster. Thanks again ªŁ€×

3 3

unable to build atlas and lapack correctly
by Alexandre Lacoste 31 Oct '08

31 Oct '08

Hi I'm trying to install numpy on a x86_64 ubuntu hardy and I'm having a hard time to get atlas to be linked correctly. Installing it through apt-get install with the astraw repository worked on my intel dual core. But now I need to install it in on a x86_64 and the astraw binaries will link against libatlas3gf-base instead of libatlas3gf-3dnow because the 3dnow version simply doesn't exist for x86_64 arch. So I still get slow matrix multiplications ... Then I went through the instructions for building atlas by hand. Then I build numpy. Everything went ok... Now when comes the time to build scipy, I get the following error : from numpy.linalg import lapack_lite ImportError: /usr/local/atlas/lib/liblapack.so: undefined symbol: ztbsv_ $ uname -a Linux graal1 2.6.24-21-generic #1 SMP Mon Aug 25 16:57:51 UTC 2008 x86_64 GNU/Linux $ ldd liblapack.so linux-vdso.so.1 => (0x00007fffe29fe000) libgfortran.so.2 => /usr/lib/libgfortran.so.2 (0x00007febd9ee3000) libm.so.6 => /lib/libm.so.6 (0x00007febd9c62000) libc.so.6 => /lib/libc.so.6 (0x00007febd98ff000) /lib64/ld-linux-x86-64.so.2 (0x00007febda93b000) I'm getting clueless and googling for the error message return noting useful any thoughts ? thanks... ªŁ€×

3 3

[Fwd: Large array using 4 times as much memory as it should]
by Rick Giuly 31 Oct '08

31 Oct '08

Please disregard the last message. I just realised 8*4=32, very sorry about this. Apparently I need some sleep. -Rick -------- Original Message -------- Subject: Large array using 4 times as much memory as it should Date: Thu, 30 Oct 2008 18:41:44 -0700 From: Rick Giuly <rgiuly(a)gmail.com> To: numpy-discussion(a)scipy.org Hello All, I find that python is using about four times as much memory as it should need for arrays. This is problematic as I need to use all available memory for large 3D imaging datasets. Is there a way to get around this problem? Am I making a mistake? Is it a bug? (I'm running windowsXP 32bit with 760M of memory "Available" according to the "Performance" pane of the task manager.) versions: numpy 1.2.0 with python 2.5.2 Any help is appreciated -Rick ************************** Details of my testing: Each test was run from the command line and for each test python was restarted. Testing a 50M array: a = numpy.ones((1024,1024,50), dtype=numpy.uint32) The available memory dropped by 200M Testing a 100M array: a = numpy.ones((1024,1024,100), dtype=numpy.uint32) The available memory dropped by 400M Testing a 200M array: a = numpy.ones((1024,1024,200), dtype=numpy.uint32) The available memory dropped by 750M Testing a 300M array: a = numpy.ones((1024,1024,300), dtype=numpy.uint32) an error occurs: Traceback (most recent call last): File "<stdin>", line 1, in <module> File "o:\software\pythonxy\python\lib\site-packages\numpy\core\numeric.py", li ne 1445, in ones a = empty(shape, dtype, order) MemoryError

2 1

Large array using 4 times as much memory as it should
by Rick Giuly 31 Oct '08

31 Oct '08

Hello All, I find that python is using about four times as much memory as it should need for arrays. This is problematic as I need to use all available memory for large 3D imaging datasets. Is there a way to get around this problem? Am I making a mistake? Is it a bug? (I'm running windowsXP 32bit with 760M of memory "Available" according to the "Performance" pane of the task manager.) versions: numpy 1.2.0 with python 2.5.2 Any help is appreciated -Rick ************************** Details of my testing: Each test was run from the command line and for each test python was restarted. Testing a 50M array: a = numpy.ones((1024,1024,50), dtype=numpy.uint32) The available memory dropped by 200M Testing a 100M array: a = numpy.ones((1024,1024,100), dtype=numpy.uint32) The available memory dropped by 400M Testing a 200M array: a = numpy.ones((1024,1024,200), dtype=numpy.uint32) The available memory dropped by 750M Testing a 300M array: a = numpy.ones((1024,1024,300), dtype=numpy.uint32) an error occurs: Traceback (most recent call last): File "<stdin>", line 1, in <module> File "o:\software\pythonxy\python\lib\site-packages\numpy\core\numeric.py", li ne 1445, in ones a = empty(shape, dtype, order) MemoryError

2 1

Complete LAPACK needed
by Frank Lagor 31 Oct '08

31 Oct '08

Dear all, I need to use functions in scipy which depend on having a complete lapack library. However, I am having a bit of trouble installing numpy and referencing a complete lapack library that I built. I have a few questions that I am hoping someone can help me answer: 1) This machine is a cluster, so do I need to be sure that the lapack library is a .so file? (I think only if I want the processors to access the file at the same time, which is my need...) If so, how do I make a shared library file? 2) If I can tell numpy to use some default version of lapack, will it be a complete version? 3) I was unable to getting numpy installed by referencing the lapack.a library from the site.cfg file. I also tried to set the LAPACK environment variable in my .bashrc file, but this did not seem to work either. The problem is that when I configure numpy, it just does not recognize this file as a lapack library. It ignores it and tries to use an lapack lite library located somewhere else (which doesn't even work anyway). Any ideas? suggestions? Thanks in advance! Frank

1 0