Build questions, atlas, lapack...
Hi all, I'm starting the transition of a large code from Numeric to numpy, so I am now doing a fresh build with a lot more care than before, actually reading all the intermediate messages. I am a bit puzzled and could use some help. This is all on an ubuntu dapper box with the atlas-sse2 packages (and everything else recommended installed). By running as suggested in the scipy readme: python ~/tmp/local/lib/python2.4/site-packages/numpy/distutils/system_info.py I get the following message at some point: ==================================== atlas_info: ( library_dirs = /usr/local/lib:/usr/lib ) ( paths: /usr/lib/atlas,/usr/lib/sse2 ) looking libraries f77blas,cblas,atlas in /usr/local/lib but found None looking libraries f77blas,cblas,atlas in /usr/local/lib but found None looking libraries lapack_atlas in /usr/local/lib but found None looking libraries lapack_atlas in /usr/local/lib but found None looking libraries f77blas,cblas,atlas in /usr/lib/atlas but found None looking libraries f77blas,cblas,atlas in /usr/lib/atlas but found None looking libraries lapack_atlas in /usr/lib/atlas but found None looking libraries lapack_atlas in /usr/lib/atlas but found None ( paths: /usr/lib/sse2/libf77blas.so ) ( paths: /usr/lib/sse2/libcblas.so ) ( paths: /usr/lib/sse2/libatlas.so ) ( paths: /usr/lib/sse2/liblapack_atlas.so ) looking libraries lapack in /usr/lib/sse2 but found None looking libraries lapack in /usr/lib/sse2 but found None looking libraries f77blas,cblas,atlas in /usr/lib but found None looking libraries f77blas,cblas,atlas in /usr/lib but found None looking libraries lapack_atlas in /usr/lib but found None looking libraries lapack_atlas in /usr/lib but found None system_info.atlas_info ( include_dirs = /usr/local/include:/usr/include ) ( paths: /usr/include/atlas_misc.h,/usr/include/atlas_enum.h,/usr/include/atlas_aux.h,/usr/include/atlas_type.h ) /usr/local/installers/src/scipy/numpy/numpy/distutils/system_info.py:870: UserWarning: ********************************************************************* Could not find lapack library within the ATLAS installation. ********************************************************************* warnings.warn(message) ( library_dirs = /usr/local/lib:/usr/lib ) ( paths: /usr/lib/atlas,/usr/lib/sse2 ) FOUND: libraries = ['f77blas', 'cblas', 'atlas'] library_dirs = ['/usr/lib/sse2'] language = c define_macros = [('ATLAS_WITHOUT_LAPACK', None)] ==================================== What I find very puzzling here is that later on, the following goes by: lapack_atlas_info: ( library_dirs = /usr/local/lib:/usr/lib ) ( paths: /usr/lib/atlas,/usr/lib/sse2 ) looking libraries lapack_atlas,f77blas,cblas,atlas in /usr/local/lib but found None looking libraries lapack_atlas,f77blas,cblas,atlas in /usr/local/lib but found None looking libraries lapack_atlas in /usr/local/lib but found None looking libraries lapack_atlas in /usr/local/lib but found None looking libraries lapack_atlas,f77blas,cblas,atlas in /usr/lib/atlas but found None looking libraries lapack_atlas,f77blas,cblas,atlas in /usr/lib/atlas but found None looking libraries lapack_atlas in /usr/lib/atlas but found None looking libraries lapack_atlas in /usr/lib/atlas but found None ( paths: /usr/lib/sse2/liblapack_atlas.so ) ( paths: /usr/lib/sse2/libf77blas.so ) ( paths: /usr/lib/sse2/libcblas.so ) ( paths: /usr/lib/sse2/libatlas.so ) ( paths: /usr/lib/sse2/liblapack_atlas.so ) looking libraries lapack in /usr/lib/sse2 but found None looking libraries lapack in /usr/lib/sse2 but found None looking libraries lapack_atlas,f77blas,cblas,atlas in /usr/lib but found None looking libraries lapack_atlas,f77blas,cblas,atlas in /usr/lib but found None looking libraries lapack_atlas in /usr/lib but found None looking libraries lapack_atlas in /usr/lib but found None system_info.lapack_atlas_info ( include_dirs = /usr/local/include:/usr/include ) ( paths: /usr/include/atlas_misc.h,/usr/include/atlas_enum.h,/usr/include/atlas_aux.h,/usr/include/atlas_type.h ) ( library_dirs = /usr/local/lib:/usr/lib ) ( paths: /usr/lib/atlas,/usr/lib/sse2 ) FOUND: libraries = ['lapack_atlas', 'f77blas', 'cblas', 'atlas'] library_dirs = ['/usr/lib/sse2'] language = c define_macros = [('ATLAS_WITH_LAPACK_ATLAS', None)] ============================================== Does the second mean that it /is/ finding the right libraries? Since the first search in atlas_info is also printing ( paths: /usr/lib/sse2/liblapack_atlas.so ) I don't quite understand why it then reports the warning. For reference, here's the content of the relevant directories on my system: ============================================== longs[sse2]> ls /usr/lib/sse2 libatlas.a libcblas.a libf77blas.a liblapack_atlas.a libatlas.so@ libcblas.so@ libf77blas.so@ liblapack_atlas.so@ libatlas.so.3@ libcblas.so.3@ libf77blas.so.3@ liblapack_atlas.so.3@ libatlas.so.3.0 libcblas.so.3.0 libf77blas.so.3.0 liblapack_atlas.so.3.0 longs[sse2]> ls /usr/lib/atlas/sse2/ libblas.a libblas.so.3@ liblapack.a liblapack.so.3@ libblas.so@ libblas.so.3.0 liblapack.so@ liblapack.so.3.0 ============================================== In summary, I don't really know if this is actually finding what it wants or not, given the two messages. Cheers, f ps - it's worth mentioning that the sequence: python ~/tmp/local/lib/python2.4/site-packages/numpy/distutils/system_info.py gets itself into a nasty recursion where it fires the interactive session 3 times in a row. And in doing so, it splits its own output in a funny way: [...] blas_opt_info: ======================================================================== Starting interactive session ------------------------------------------------------------------------ Tasks: i - Show python/platform/machine information ie - Show environment information c - Show C compilers information c<name> - Set C compiler (current:None) f - Show Fortran compilers information f<name> - Set Fortran compiler (current:None) e - Edit proposed sys.argv[1:]. Task aliases: 0 - Configure 1 - Build 2 - Install 2<prefix> - Install with prefix. 3 - Inplace build 4 - Source distribution 5 - Binary distribution Proposed sys.argv = ['/home/fperez/tmp/local/lib/python2.4/site-packages/numpy/distutils/system_info.py'] Choose a task (^D to quit, Enter to continue with setup): ##### msg: ( library_dirs = /usr/local/lib:/usr/lib ) FOUND: libraries = ['f77blas', 'cblas', 'atlas'] library_dirs = ['/usr/lib/sse2'] language = c define_macros = [('NO_ATLAS_INFO', 2)] ================= I tried to fix it, but the call sequence in that code is convoluted enough that after a few 'import traceback;traceback.print_stack()' tries I sort of gave up. That code is rather (how can I say this nicely) pasta-like :), and thoroughly uncommented, so I'm afraid I won't be able to contribute a cleanup here. I think this tool should run by default in a mode with NO attempt to fire a command-line subsystem of its own, so users can simply run python /path/to/system_info > system_info.log for further analysis.
On Thu, 8 Jun 2006 16:48:27 -0600 "Fernando Perez" <fperez.net@gmail.com> wrote: [snip]
I tried to fix it, but the call sequence in that code is convoluted enough that after a few 'import traceback;traceback.print_stack()' tries I sort of gave up. That code is rather (how can I say this nicely) pasta-like :), and thoroughly uncommented, so I'm afraid I won't be able to contribute a cleanup here.
I think the whole numpy.distutils could use a good cleanup ...
I think this tool should run by default in a mode with NO attempt to fire a command-line subsystem of its own, so users can simply run
python /path/to/system_info > system_info.log
for further analysis.
Agree; I'll look at it. -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm@physics.mcmaster.ca
I don't know if it's related, but I've found on my Debian system that whenever I want to compile something that uses the atlas library, I need to put -L/usr/lib/sse2 on the gcc line, even though everything seems to indicate that the linker has been told to look there already. It could be that Ubuntu has a similar issue, and that it is affecting your build. Dan
On 6/8/06, Dan Christensen <jdc@uwo.ca> wrote:
I don't know if it's related, but I've found on my Debian system that whenever I want to compile something that uses the atlas library, I need to put -L/usr/lib/sse2 on the gcc line, even though everything seems to indicate that the linker has been told to look there already. It could be that Ubuntu has a similar issue, and that it is affecting your build.
mmh, given how green I am in the ubuntu world, you may well be right. But my original question went before any linking happens, since I was just posting the messages from numpy's system_info, which doesn't attempt to link at anything, it just does a static filesystem analysis. So perhaps there is more than one issue here. I'm just trying to clarify, from the given messages (which I found a bit confusing) whether all the atlas/sse2 stuff is actually being picked up or not, at least as far as numpy thinks it is. Cheers, f
On Thu, 8 Jun 2006 16:48:27 -0600 "Fernando Perez" <fperez.net@gmail.com> wrote:
In summary, I don't really know if this is actually finding what it wants or not, given the two messages.
I just went through this on debian sarge which is similar. I put this in site.cgf: [atlas] library_dirs = /usr/lib/atlas/ atlas_libs = lapack, blas Then I needed to set LD_LIBRARY_PATH to point to /usr/lib/atlas/sse2. $ env LD_LIBRARY_PATH=/usr/lib/atlas/sse2 python2.4 Python 2.4.3 (#4, Jun 5 2006, 19:07:06) [GCC 3.4.1 (Debian 3.4.1-5)] on linux2 Type "help", "copyright", "credits" or "license" for more information.
import numpy
[1]+ Stopped env LD_LIBRARY_PATH=/usr/lib/atlas/sse2 python2.4
Look in /proc/PID/maps for the relevant libs: $ ps -a|grep python ... 16953 pts/64 00:00:00 python2.4 $ grep atlas /proc/16953/maps b6fa7000-b750e000 r-xp 00000000 00:0c 1185402 /usr/lib/atlas/sse2/libblas.so.3.0 b750e000-b7513000 rwxp 00567000 00:0c 1185402 /usr/lib/atlas/sse2/libblas.so.3.0 b7513000-b7a58000 r-xp 00000000 00:0c 1185401 /usr/lib/atlas/sse2/liblapack.so.3.0 b7a58000-b7a5b000 rwxp 00545000 00:0c 1185401 /usr/lib/atlas/sse2/liblapack.so.3.0 $ But to really test this is working I ran python under gdb and set a break point on cblas_dgemm. Then a call to numpy.dot should break inside the sse2/liblapack.so.3.0. (also it's a lot faster with the sse2 dgemm) $ env LD_LIBRARY_PATH=/usr/lib/atlas/sse2 gdb python2.4 GNU gdb 6.1-debian Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-linux"...Using host libthread_db library "/lib/tls/libthread_db.so.1". (gdb) break cblas_dgemm Function "cblas_dgemm" not defined. Make breakpoint pending on future shared library load? (y or [n]) y Breakpoint 1 (cblas_dgemm) pending. (gdb) run Starting program: /home/users/simonb/bin/python2.4 [Thread debugging using libthread_db enabled] [New Thread -1210476000 (LWP 17557)] Python 2.4.3 (#4, Jun 5 2006, 19:07:06) [GCC 3.4.1 (Debian 3.4.1-5)] on linux2 Type "help", "copyright", "credits" or "license" for more information. Breakpoint 2 at 0xb7549db0 Pending breakpoint "cblas_dgemm" resolved <------- import numpy is in my pythonstartup
a=numpy.empty((1024,1024),'d') b=numpy.empty((1024,1024),'d') numpy.dot(a,b) [Switching to Thread -1210476000 (LWP 17557)]
Breakpoint 2, 0xb7549db0 in cblas_dgemm () from /usr/lib/atlas/sse2/liblapack.so.3 (gdb) bingo. Simon. -- Simon Burton, B.Sc. Licensed PO Box 8066 ANU Canberra 2601 Australia Ph. 61 02 6249 6940 http://arrowtheory.com
On 6/8/06, Simon Burton <simon@arrowtheory.com> wrote:
On Thu, 8 Jun 2006 16:48:27 -0600 "Fernando Perez" <fperez.net@gmail.com> wrote:
In summary, I don't really know if this is actually finding what it wants or not, given the two messages.
I just went through this on debian sarge which is similar.
I put this in site.cgf:
[atlas] library_dirs = /usr/lib/atlas/ atlas_libs = lapack, blas
Then I needed to set LD_LIBRARY_PATH to point to /usr/lib/atlas/sse2.
[...]
But to really test this is working I ran python under gdb and set a break point on cblas_dgemm. Then a call to numpy.dot should break inside the sse2/liblapack.so.3.0.
(also it's a lot faster with the sse2 dgemm)
$ env LD_LIBRARY_PATH=/usr/lib/atlas/sse2 gdb python2.4
OK, thanks a LOT for that gdb trick: it provides a very nice way to understand what's actually going on. self.note("really, learn better use of gdb") Using that, though, it would then seem as if the build DID successfully find everything without any further action on my part: longs[dist]> gdb python GNU gdb 6.4-debian ... (gdb) break cblas_dgemm Function "cblas_dgemm" not defined. Make breakpoint pending on future shared library load? (y or [n]) y Breakpoint 1 (cblas_dgemm) pending. (gdb) run Starting program: /usr/bin/python ... Python 2.4.3 (#2, Apr 27 2006, 14:43:58) [GCC 4.0.3 (Ubuntu 4.0.3-1ubuntu5)] on linux2 Type "help", "copyright", "credits" or "license" for more information. (no debugging symbols found)
import numpy Breakpoint 2 at 0x40429860 Pending breakpoint "cblas_dgemm" resolved a=numpy.empty((1024,1024),'d') b=numpy.empty((1024,1024),'d') numpy.dot(a,b) [Switching to Thread 1075428416 (LWP 3919)]
Breakpoint 2, 0x40429860 in cblas_dgemm () from /usr/lib/sse2/libcblas.so.3 ====================================================== Note that on my system, LD_LIBRARY_PATH does NOT contain that dir: longs[dist]> env | grep LD_LIB LD_LIBRARY_PATH=/usr/local/lf9560/lib:/usr/local/intel/mkl/8.0.2/lib/32:/usr/local/intel/compiler90/lib:/home/fperez/usr/lib:/home/fperez/usr/local/lib: and I built everything with a plain setup.py install --prefix=~/tmp/local without /any/ tweaks to site.cfg, no LD_LIBRARY_PATH modifications or anything else. I just installed atlas-sse2* and lapack3*, but NOT refblas3*. Basically it seems that the build process does the right thing out of the box, and the warning is spurious. Since I was being extra-careful in this build, I didn't want to let any warning of that kind go unchecked. It might still be worth fixing that warning to prevent others from going on a similar wild goose chase, but I'm not comfortable touching that code (I don't know if anyone besides Pearu is). Thanks for the help! Cheers, f
==================================== atlas_info: ( library_dirs = /usr/local/lib:/usr/lib ) ( paths: /usr/lib/atlas,/usr/lib/sse2 ) looking libraries f77blas,cblas,atlas in /usr/local/lib but found None looking libraries f77blas,cblas,atlas in /usr/local/lib but found None (.. more of these...)
Some of these and similar spurious warnings can be eliminated by replacing the calls to check_libs in system_info.py with calls to check_libs2. Currently these warnings are generated for each file extension that is tested (".so", ".a"...) Alternatively, the warnings could be made more informative. Many of the other warnings could be eliminated by consolidating the various BLAS/LAPACK options. If anyone is manipulating the build system, could he please apply the patch from #114 fixing the Windows build?
I tried to fix it, but the call sequence in that code is convoluted enough that after a few 'import traceback;traceback.print_stack()' tries I sort of gave up. That code is rather (how can I say this nicely) pasta-like :), and thoroughly uncommented, so I'm afraid I won't be able to contribute a cleanup here.
Even if you spent enough time to understand the existing code, you probably wouldn't have a chance to clean up the code because any small change could break some obscure platform/compiler/library combination. Moreover, changes could break the build of scipy and other libraries depending on Numpy-distutils. If you really wanted to rewrite the build code, you'd need to specify a minimum set of supported platform and library combinations, have each of them available for testing and deliberately risk breaking any other platform. Regards, Stephan
participants (5)
-
Dan Christensen -
David M. Cooke -
Fernando Perez -
Simon Burton -
Stephan Tolksdorf