doc bug in numarray 1.1
HI all, I just installed numarray 1.1. All went well with setup.py. Then I decided to try to build it against an atlas lapack. I found, under the Linear Algebra section: """ the setup procedure needs to be modified to force the lapack_lite module to be linked against those rather than the builtin replacement functions. Edit Packages/LinearAlgebra2/setup.py and edit the variables sourcelist, lapack_dirs, and lapack_libs. In sourcelist you should remove all sourcefiles besides .... """ but there is no: Packages/LinearAlgebra2/setup.py However, I did find in: addons.py """ if os.environ.has_key('USE_LAPACK'): BUILTIN_BLAS_LAPACK = 0 else: BUILTIN_BLAS_LAPACK = 1 """ so I tried: export USE_LAPACK=true python setup.py build --gencode Now it appears to be compiling linear_algebra differently, as I now get a linking error, can't find: f90math, fio, or f77math However, if I remove those from lapack_libs in addons.py, I can get it to compile, install, and as far as I can tell, run fine. Why are they in that list? By the way, this is all under Gentoo Linux. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
On Mon, 2004-09-13 at 16:32, Chris Barker wrote:
I just installed numarray 1.1... All went well with setup.py. Then I decided to try to build it against an atlas lapack. I found, under the Linear Algebra section:
""" the setup procedure needs to be modified to force the lapack_lite module to be linked against those rather than the builtin replacement functions.
This does seem like a documentation bug.
addons.py
so I tried:
export USE_LAPACK=true python setup.py build --gencode
Now it appears to be compiling linear_algebra differently, as I now get a linking error, can't find:
f90math, fio, or f77math
Amazingly enough, I just found this problem today myself, and I also confess it is My Fault (tm), as I was unclear in a previous post to thie forum. These libraries are specific to the commercial Absoft Fortran compiler. If you change the lapack_libs assignment in addons.py to lapack_libs = ['lapack', 'f77blas', 'cblas', 'atlas', 'g2c', 'm'] then the command you used above will build against ATLAS with g77. Building against ATLAS is definitely worthwhile. On my little laptop (mobile AMD Athlon XP2800+) the time to solve a 1000x1000 random array went from 10.7 to 0.9 seconds, and that was just using the prebuilt Linux_ATHLON ATLAS tarball from the scipy.com site, not one I compiled myself optimized for my computer. Steve Walton
On Tue, 2004-09-14 at 21:39, Stephen Walton wrote:
These libraries are specific to the commercial Absoft Fortran compiler. If you change the lapack_libs assignment in addons.py to
lapack_libs = ['lapack', 'f77blas', 'cblas', 'atlas', 'g2c', 'm']
And I'm a bit of an idiot. This should be a permanent change; since
numarray itself is written in C, there is no part of it which gets
compiled with Fortran and so no reason to link against vendor Fortran
libraries. I realized this when looking at the Numeric setup.py, which
uses the library list above and which has always built fine on my
Absoft-equipped systems.
--
Stephen Walton
Stephen Walton wrote:
These libraries are specific to the commercial Absoft Fortran compiler. If you change the lapack_libs assignment in addons.py to
lapack_libs = ['lapack', 'f77blas', 'cblas', 'atlas', 'g2c', 'm']
And I'm a bit of an idiot. This should be a permanent change;
great. Do we need to submit a bug report, or is someone going to do this? By the way, if it is found that different library lists are needed for different systems, it would be nice to have a small selection of list of commented out options: if BUILTIN_BLAS_LAPACK: sourcelist = [ os.path.join('Packages/LinearAlgebra2/Src', 'lapack_litemodule.c'), os.path.join('Packages/LinearAlgebra2/Src', 'blas_lite.c'), os.path.join('Packages/LinearAlgebra2/Src', 'f2c_lite.c'), os.path.join('Packages/LinearAlgebra2/Src', 'zlapack_lite.c'), os.path.join('Packages/LinearAlgebra2/Src', 'dlapack_lite.c') ] lapack_libs = [] else: sourcelist = [ os.path.join('Packages/LinearAlgebra2/Src', 'lapack_litemodule.c'), ] # Set to list off libraries to link against. # (only the basenames, e.g. 'lapack') ## for atlas on linux: lapack_libs = ['lapack', 'f77blas', 'cblas', 'atlas', 'm'] ## for absoft on linux: #lapack_libs = ['lapack', 'f77blas', 'cblas', 'atlas', 'm','someotherlib'] ## for whatever on whatever: #lapack_libs = ['a','different','list'] Also: Shouldn't this be inside the "if" above ? # Set to list directories to be searched for BLAS and LAPACK libraries # For absoft on Linux ##lapack_dirs = ['/usr/local/lib/atlas', '/opt/absoft/lib'] # For atlas on Gentoo Linux lapack_dirs = [] Though I suppose it doesn't hurt to search non-exisitant directories. By the way. I set the USE_LAPACK environment variable. Is there a way to pass it in as an option to setup.py instead? That seems a better way of keeping with the spirit of distutils. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
On Wed, 2004-09-15 at 14:25, Chris Barker wrote:
Stephen Walton wrote:
These libraries are specific to the commercial Absoft Fortran compiler. If you change the lapack_libs assignment in addons.py to
lapack_libs = ['lapack', 'f77blas', 'cblas', 'atlas', 'g2c', 'm']
And I'm a bit of an idiot. This should be a permanent change;
great. Do we need to submit a bug report, or is someone going to do this?
I already logged it on SF. I was planning to have two lists of libraries, with Absoft commented out this time around since it is a commercial compiler. The second (active) list would be the one for g77.
By the way, if it is found that different library lists are needed for different systems, it would be nice to have a small selection of list of commented out options:
if BUILTIN_BLAS_LAPACK: sourcelist = [ os.path.join('Packages/LinearAlgebra2/Src', 'lapack_litemodule.c'), os.path.join('Packages/LinearAlgebra2/Src', 'blas_lite.c'), os.path.join('Packages/LinearAlgebra2/Src', 'f2c_lite.c'), os.path.join('Packages/LinearAlgebra2/Src', 'zlapack_lite.c'), os.path.join('Packages/LinearAlgebra2/Src', 'dlapack_lite.c') ] lapack_libs = [] else: sourcelist = [ os.path.join('Packages/LinearAlgebra2/Src', 'lapack_litemodule.c'), ]
# Set to list off libraries to link against. # (only the basenames, e.g. 'lapack') ## for atlas on linux: lapack_libs = ['lapack', 'f77blas', 'cblas', 'atlas', 'm'] ## for absoft on linux: #lapack_libs = ['lapack', 'f77blas', 'cblas', 'atlas', 'm','someotherlib'] ## for whatever on whatever: #lapack_libs = ['a','different','list']
Also: Shouldn't this be inside the "if" above ?
Yes.
# Set to list directories to be searched for BLAS and LAPACK libraries # For absoft on Linux ##lapack_dirs = ['/usr/local/lib/atlas', '/opt/absoft/lib'] # For atlas on Gentoo Linux lapack_dirs = []
Though I suppose it doesn't hurt to search non-exisitant directories.
I think you were right earlier.
By the way. I set the USE_LAPACK environment variable. Is there a way to pass it in as an option to setup.py instead? That seems a better way of keeping with the spirit of distutils.
I added --use_lapack, as in 'python setup.py install --use_lapack'. One thing I noticed was that I didn't appear to need cblas. Comments? We more or less crossed e-mails. I saw your later comments about eliminating Absoft library list altogether but don't think that is necessary; I think hints from other people's installations are useful so unless there's something wrong with the Absoft list, I think we should leave it in, but with g77 as the default. Regards, Todd
note: this may be a second copy, my email system crashed as I was sending it the last time. Stephen Walton wrote:
These libraries are specific to the commercial Absoft Fortran compiler. If you change the lapack_libs assignment in addons.py to
lapack_libs = ['lapack', 'f77blas', 'cblas', 'atlas', 'g2c', 'm']
And I'm a bit of an idiot. This should be a permanent change;
great. Do we need to submit a bug report, or is someone going to do this? By the way, if it is found that different library lists are needed for different systems, it would be nice to have a small selection of list of commented out options: if BUILTIN_BLAS_LAPACK: sourcelist = [ os.path.join('Packages/LinearAlgebra2/Src', 'lapack_litemodule.c'), os.path.join('Packages/LinearAlgebra2/Src', 'blas_lite.c'), os.path.join('Packages/LinearAlgebra2/Src', 'f2c_lite.c'), os.path.join('Packages/LinearAlgebra2/Src', 'zlapack_lite.c'), os.path.join('Packages/LinearAlgebra2/Src', 'dlapack_lite.c') ] lapack_libs = [] else: sourcelist = [ os.path.join('Packages/LinearAlgebra2/Src', 'lapack_litemodule.c'), ] # Set to list off libraries to link against. # (only the basenames, e.g. 'lapack') ## for atlas on linux: lapack_libs = ['lapack', 'f77blas', 'cblas', 'atlas', 'm'] ## for absoft on linux: #lapack_libs = ['lapack', 'f77blas', 'cblas', 'atlas', 'm','someotherlib'] ## for whatever on whatever: #lapack_libs = ['a','different','list'] Also: Shouldn't this be inside the "if" above ? # Set to list directories to be searched for BLAS and LAPACK libraries # For absoft on Linux ##lapack_dirs = ['/usr/local/lib/atlas', '/opt/absoft/lib'] # For atlas on Gentoo Linux lapack_dirs = [] Though I suppose it doesn't hurt to search non-exisitant directories. By the way. I set the USE_LAPACK environment variable. Is there a way to pass it in as an option to setup.py instead? That seems a better way of keeping with the spirit of distutils. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
On Wed, 2004-09-15 at 11:38, Chris Barker wrote:
great. Do we need to submit a bug report, or is someone going to do this?
I think I'm about to change my mind again; I was sort of right the first time. Sorry to bug the list with this kind of 'thinking out loud.' Listing the vendor specific Fortran libraries is necessary, but if and only if one built ATLAS and LAPACK with your vendor's compiler; those do contain actual Fortran sources. I haven't done serious benchmarks to find out how much faster LAPACK and ATLAS might be with Absoft Fortran (my compiler) than with g77. If there are some benchmarks I can run to experiment with this, I'd be happy to try them. Otherwise maybe we should all just build with g77 and forget about it. The resulting libraries can be called from Absoft-compiled programs with no difficulty. So, I haven't formally reported a bug on Sourceforge yet because I'm not sure how it should read.
By the way, if it is found that different library lists are needed for different systems, it would be nice to have a small selection of list of commented out options:
I agree. As they get submitted, they could be added.
Also: Shouldn't this be inside the "if" above ?
I haven't looked at the code carefully enough.
Though I suppose it doesn't hurt to search non-exisitant directories.
I've hit too many unanticipated side effects to have directories listed which aren't needed. Too much chance of picking up an unintended library.
By the way. I set the USE_LAPACK environment variable. Is there a way to pass it in as an option to setup.py instead?
I would think there should be. When building Scipy with Absoft, one
uses an option "fc_compiler=Absoft" on the setup.py command line.
--
Stephen Walton
participants (3)
-
Chris Barker -
Stephen Walton -
Todd Miller