adding LAPACK and BLAS to the source code

Hey group, Apologies if you get this twice -- we had a mail list hiccup earlier this week, and I don't think this went out correctly. We just had the unpleasant experience of getting SciPy to build on the Sun architecture. In the end, it didn't prove that hard -- it was all the mis-steps that were painful. Most of SciPy builds without a hitch -- it is only the LAPACK and BLAS stuff that causes major problems. The issue is that many LAPACK's are built with vendor specific compilers instead of g77, so getting the correct libraries to link in when compiling with distutils is a major headache. And, once the link works, the headaches continue with chasing down random seg-faults that appear to come from incompatible library ABI issues. We worked with ATLAS for a while, but finally backed off to working with just the standard LAPACK and BLAS. The result was a pain free build because gcc and g77 were used all the way through. I expect building everything with the Sun compilers would also go fairly smoothly using this approach. It seems like most of the questions we get on building SciPy have to do with ATLAS and linear algebra. On windows and Linux, we about have this problem licked. But its still difficult on platforms with multiple compilers. So, I am planning to put LAPACK and BLAS into the SciPy CVS so that they are built with the same compiler as the rest of SciPy and no longer have to be built separately. This will increase the repository size substantially (LAPACK is big), but has little other side effects. We'll set up the building of LAPACK and BLAS so that they are optional -- if you have ATLAS or other LAPACK libraries you'd like to use, you can specify them in the site.cfg package -- or at least specify that system_info should search for them. This search will be turned off by default, however, so that incompatible lapack/blas/atlas libraries are not found. I think this will ease the build process for everyone and still allows for using the optimized libraries. If there are no major objections, please let me know. If not, I'll make the addition Tuesday or Wednesday. Eric

On Sun, 2002-06-23 at 13:45, eric jones wrote:
Hey group,
Apologies if you get this twice -- we had a mail list hiccup earlier this week, and I don't think this went out correctly.
We just had the unpleasant experience of getting SciPy to build on the Sun architecture. In the end, it didn't prove that hard -- it was all the mis-steps that were painful. Most of SciPy builds without a hitch -- it is only the LAPACK and BLAS stuff that causes major problems.
I had thought it would be easier to get optimized LAPACK and BLAS with Sun systems.
The issue is that many LAPACK's are built with vendor specific compilers instead of g77, so getting the correct libraries to link in when compiling with distutils is a major headache. And, once the link works, the headaches continue with chasing down random seg-faults that appear to come from incompatible library ABI issues.
Ah, this is the problem with using those.
We worked with ATLAS for a while, but finally backed off to working with just the standard LAPACK and BLAS.
Why did ATLAS not work?
The result was a pain free build because gcc and g77 were used all the way through. I expect building everything with the Sun compilers would also go fairly smoothly using this approach. It seems like most of the questions we get on building SciPy have to do with ATLAS and linear algebra. On windows and Linux, we about have this problem licked. But its still difficult on platforms with multiple compilers. So, I am planning to put LAPACK and BLAS into the SciPy CVS so that they are built with the same compiler as the rest of SciPy and no longer have to be built separately.
Won't this result in slow code on those machines?
This will increase the repository size substantially (LAPACK is big), but has little other side effects. We'll set up the building of LAPACK and BLAS so that they are optional -- if you have ATLAS or other LAPACK libraries you'd like to use, you can specify them in the site.cfg package -- or at least specify that system_info should search for them. This search will be turned off by default, however, so that incompatible lapack/blas/atlas libraries are not found.
It seems like in CVS the default should be to use the optimized LAPACK given that this is the default most of the developers would want to use. Is the site.cfg file managed by CVS? If so, then our constant changes to this file will undoubtedly result in checking in a version that searches by default. It would seem that a check for certain machines and a subsequent turning off of that option would be better. -Travis

-----Original Message----- From: scipy-dev-admin@scipy.net [mailto:scipy-dev-admin@scipy.net] On Behalf Of Travis Oliphant Sent: Sunday, June 23, 2002 9:05 PM To: scipy-dev@scipy.net Subject: Re: [SciPy-dev] adding LAPACK and BLAS to the source code
Hey group,
Apologies if you get this twice -- we had a mail list hiccup earlier this week, and I don't think this went out correctly.
We just had the unpleasant experience of getting SciPy to build on
On Sun, 2002-06-23 at 13:45, eric jones wrote: the
Sun architecture. In the end, it didn't prove that hard -- it was all the mis-steps that were painful. Most of SciPy builds without a hitch -- it is only the LAPACK and BLAS stuff that causes major problems.
I had thought it would be easier to get optimized LAPACK and BLAS with Sun systems.
The issue is that many LAPACK's are built with vendor specific compilers instead of g77, so getting the correct libraries to link in when compiling with distutils is a major headache. And, once the link works, the headaches continue with chasing down random seg-faults that appear to come from incompatible library ABI issues.
Ah, this is the problem with using those.
Well, really, I'm not sure that its the C ABI. I would have thought (and do think) that gcc is compatible with the native compiler ABI. I'm more suspicious about g77 and f77 library incompatibilities/clashes, especially with IO routines. This is all speculations because we never chased down the real problems -- it just took to long.
We worked with ATLAS for a while, but finally backed off to working
with
just the standard LAPACK and BLAS.
Why did ATLAS not work?
One time it was name mangling issues, another Fortran compiler issues. The deal is that making "mistakes" in the build process is extremely high. If the created library has to incorrect name mangling, or the wrong fortran compiler setting, or whatever, you don't find out until 5 hours later (on Sun).
The result was a pain free build because gcc and g77 were used all the way through. I expect
everything with the Sun compilers would also go fairly smoothly using this approach. It seems like most of the questions we get on building SciPy have to do with ATLAS and linear algebra. On windows and Linux, we about have this problem licked. But its still difficult on
building platforms
with multiple compilers. So, I am planning to put LAPACK and BLAS into the SciPy CVS so that they are built with the same compiler as the rest of SciPy and no longer have to be built separately.
Won't this result in slow code on those machines?
The resulting code was actually pretty fast. There was a factor of 5-10 improvement over Numeric's linear algebra stuff. I'm sure it is still quite a bit slower than ATLAS, but in many cases, just getting the capability is much more important than having the ultimate in speed.
This will increase the repository size substantially (LAPACK is big), but has little
side effects. We'll set up the building of LAPACK and BLAS so that
are optional -- if you have ATLAS or other LAPACK libraries you'd
other they like
to use, you can specify them in the site.cfg package -- or at least specify that system_info should search for them. This search will be turned off by default, however, so that incompatible lapack/blas/atlas libraries are not found.
It seems like in CVS the default should be to use the optimized LAPACK given that this is the default most of the developers would want to use. Is the site.cfg file managed by CVS? If so, then our constant changes to this file will undoubtedly result in checking in a version that searches by default.
I think this will probably work fine on Linux since gcc is the main tool here, and compatibility issues are much less likely to crop up. Perhaps a nm check on the found ATLAS library to find out if the libraries symbols match what SciPy expects would be a good idea though. Another question on Linux that is likely to become more and more important is whether gcc 3.x compiled libraries are compatible with gcc 2.9x ibraries? Outside of the Linux and Windows worlds, defaulting to optimized libraries has problems. On Sun, all the libraries existed (ATLAS, vendor supplied blas, lapack were all in /usr/lib), and were detected by the SciPy, but none of these libraries would work because of the mentioned issues. So, having SciPy automatically detect and choose these optimized libraries is problematic on non-Linux Unix platforms.
It would seem that a check for certain machines and a subsequent
turning
off of that option would be better.
That sounds fine to me -- if Linux choose optimized by default. If other Unix, choose LAPACK/BLAS by default. eric

"EJ" == eric jones <eric@enthought.com> writes:
EJ> We just had the unpleasant experience of getting SciPy to EJ> build on the Sun architecture. In the end, it didn't prove EJ> that hard -- it was all the mis-steps that were painful. Most EJ> of SciPy builds without a hitch -- it is only the LAPACK and EJ> BLAS stuff that causes major problems. Right now my biggest trouble with scipy is that it needs a recent version of Atlas. The version that Debian ships with is too old for scipy. Some of my older code does not work with scipy anymore because my build is broken. I already need to download too many packages from source. While I understand that a newer version of atlas has features that are worth it, sometimes folks dont use the features. By making it mandatory to install atlas it becomes hard to use scipy. Packaging lapack might eliminate these problems but by sticking lapack into scipy's CVS tree aren't we making the scipy cvs tree that much bigger? Sticking it in CVS also means that even if the package is removed it still lingers in the attic (which can be painful sometimes). Also, aren't there any other issues with including lapack in scipy? Aren't we in some sense forking lapack or is lapack no longer under development?? cheers, prabhu

-----Original Message----- From: scipy-dev-admin@scipy.net [mailto:scipy-dev-admin@scipy.net] On Behalf Of Prabhu Ramachandran Sent: Monday, June 24, 2002 1:15 PM To: scipy-dev@scipy.net Subject: [SciPy-dev] adding LAPACK and BLAS to the source code
"EJ" == eric jones <eric@enthought.com> writes:
EJ> We just had the unpleasant experience of getting SciPy to EJ> build on the Sun architecture. In the end, it didn't prove EJ> that hard -- it was all the mis-steps that were painful. Most EJ> of SciPy builds without a hitch -- it is only the LAPACK and EJ> BLAS stuff that causes major problems.
Right now my biggest trouble with scipy is that it needs a recent version of Atlas. The version that Debian ships with is too old for scipy. Some of my older code does not work with scipy anymore because my build is broken. I already need to download too many packages from source. While I understand that a newer version of atlas has features that are worth it, sometimes folks dont use the features. By making it mandatory to install atlas it becomes hard to use scipy. Packaging lapack might eliminate these problems but by sticking lapack into scipy's CVS tree aren't we making the scipy cvs tree that much bigger?
It does grow -- I see this as the major drawback.
Sticking it in CVS also means that even if the package is removed it still lingers in the attic (which can be painful sometimes).
I haven't dealt with this, but (perhaps naively) feel this isn't that huge of an issue. If there is evidence to the contrary, please elaborate. I guess the other deal is that I don't think we'll take it out in the future, accept when a new version of LAPACK comes out.
Also, aren't there any other issues with including lapack in scipy? Aren't we in some sense forking lapack or is lapack no longer under development??
We're not really forking since I doubt we make any modifications -- its purely a logistical issue. We already have code from FFTPACK, ODEPACK, FITPACK, cephes, amos, etc. in SciPy. We have made a few modifications in these to get them working on newer compilers, etc., but nothing to major. There kept there just to make building SciPy easier. LAPACK would be included for the same reason. Eric

"EJ" == eric jones <eric@enthought.com> writes:
>> Sticking it in CVS also means that even if the package is >> removed it still lingers in the attic (which can be painful >> sometimes). EJ> I haven't dealt with this, but (perhaps naively) feel this EJ> isn't that huge of an issue. If there is evidence to the EJ> contrary, please elaborate. I guess the other deal is that I EJ> don't think we'll take it out in the future, accept when a new EJ> version of LAPACK comes out. If you choose to backup your entire CVS tree you'll also end up backing up all of the files in the attic. This can be painful. >> Also, aren't there any other issues with including lapack in >> scipy? Aren't we in some sense forking lapack or is lapack no >> longer under development?? EJ> We're not really forking since I doubt we make any EJ> modifications -- its purely a logistical issue. We already EJ> have code from FFTPACK, ODEPACK, FITPACK, cephes, amos, EJ> etc. in SciPy. We have made a few modifications in these to EJ> get them working on newer compilers, etc., but nothing to EJ> major. There kept there just to make building SciPy easier. EJ> LAPACK would be included for the same reason. Sure, but its one more thing to keep track of. If you really dont think its too big a deal I guess it makes sense to just go ahead and add lapack as well. Maybe you could put all these extra packages into a separate cvs module so that only folks who want it can check it out? This way the extras are separated from the main scipy code. Just a thought. cheers, prabhu

On Tue, 25 Jun 2002, Prabhu Ramachandran wrote:
Sure, but its one more thing to keep track of. If you really dont think its too big a deal I guess it makes sense to just go ahead and add lapack as well. Maybe you could put all these extra packages into a separate cvs module so that only folks who want it can check it out? This way the extras are separated from the main scipy code. Just a thought.
Another thought is to commit lapack source to CVS as a single tar.gz file. And removing BLAS,INSTALL,TESTING, and TIMING directories from the LAPACK distribution (keeping only SRC directory) will reduce the size of LAPACK sources considerably (39MB -> 15MB or 2MB when gzipped). Notice that BLAS sources shipped with LAPACK are not complete. So, the above applies also to BLAS. Pearu

-----Original Message----- From: scipy-dev-admin@scipy.net [mailto:scipy-dev-admin@scipy.net] On Behalf Of Pearu Peterson Sent: Tuesday, June 25, 2002 12:36 AM To: scipy-dev@scipy.net Subject: RE: [SciPy-dev] adding LAPACK and BLAS to the source code
On Tue, 25 Jun 2002, Prabhu Ramachandran wrote:
Sure, but its one more thing to keep track of. If you really dont think its too big a deal I guess it makes sense to just go ahead and add lapack as well. Maybe you could put all these extra packages into a separate cvs module so that only folks who want it can check it out? This way the extras are separated from the main scipy code. Just a thought.
Another thought is to commit lapack source to CVS as a single tar.gz file. And removing BLAS,INSTALL,TESTING, and TIMING directories from the LAPACK distribution (keeping only SRC directory) will reduce the size of LAPACK sources considerably (39MB -> 15MB or 2MB when gzipped).
This sounds like a good idea. The only issue is that Windows doesn't come with tar. There is a zlib module in python that solve the zipped issue. There is also a tarfile module running around that would make tar unnecessary, but last time I tried it, it wasn't that stable. Anyone know if its current state? eric

On Tue, Jun 25, 2002 at 11:51:37AM -0500, eric jones wrote: [snip]
This sounds like a good idea. The only issue is that Windows doesn't come with tar.
WinZIP opens gzipped tarballs just fine. -- Robert Kern Ruddock House President kern@caltech.edu "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter

-----Original Message----- From: scipy-dev-admin@scipy.net [mailto:scipy-dev-admin@scipy.net] On Behalf Of Robert Kern Sent: Tuesday, June 25, 2002 12:07 PM To: scipy-dev@scipy.net Subject: Re: [SciPy-dev] adding LAPACK and BLAS to the source code
On Tue, Jun 25, 2002 at 11:51:37AM -0500, eric jones wrote:
[snip]
This sounds like a good idea. The only issue is that Windows doesn't come with tar.
WinZIP opens gzipped tarballs just fine.
Right, but the build process would have to do this on the fly, and making an os.system call to winzip is not a reliable option. While widely used, it isn't nearly as ubiquitous as tar is on Unix machines. Eric
-- Robert Kern Ruddock House President kern@caltech.edu
"In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter _______________________________________________ Scipy-dev mailing list Scipy-dev@scipy.net http://www.scipy.net/mailman/listinfo/scipy-dev

On Tue, Jun 25, 2002 at 12:11:53PM -0500, eric jones wrote:
-----Original Message----- From: scipy-dev-admin@scipy.net [mailto:scipy-dev-admin@scipy.net] On Behalf Of Robert Kern Sent: Tuesday, June 25, 2002 12:07 PM To: scipy-dev@scipy.net Subject: Re: [SciPy-dev] adding LAPACK and BLAS to the source code
On Tue, Jun 25, 2002 at 11:51:37AM -0500, eric jones wrote:
[snip]
This sounds like a good idea. The only issue is that Windows doesn't come with tar.
WinZIP opens gzipped tarballs just fine.
Right, but the build process would have to do this on the fly, and making an os.system call to winzip is not a reliable option. While widely used, it isn't nearly as ubiquitous as tar is on Unix machines.
True, true. In that case zipping it and using zipfile from the Standard Library would be best.
Eric
-- Robert Kern Ruddock House President kern@caltech.edu "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter

"RK" == Robert Kern <kern@caltech.edu> writes:
>> Right, but the build process would have to do this on the fly, >> and making an os.system call to winzip is not a reliable >> option. While widely used, it isn't nearly as ubiquitous as >> tar is on Unix machines. RK> True, true. In that case zipping it and using zipfile from the RK> Standard Library would be best. Yes, it seems the best portable way of doing this. However there are problems with this approach too. Compressed zip files will require that the zlib module be compiled. My install has zlib but does every Python install include the zlib module? The other problem with using a zip file to distribute lapack along with CVS is that the smallest change of a file inside the zip file will require a fresh download. It would also mess up CVS because CVS is pretty lousy with binary files. So if anything changes it would cause problems. One possibility is that the build utilities look for a zip file in a specific directory and those folks who need to use the lapack sources simply download the new version and stick it in the right place. The builder uses this and compiles everything. This way we keep the zip file out of CVS's hair. cheers, prabhu

"EJ" == eric jones <eric@enthought.com> writes:
EJ> This sounds like a good idea. The only issue is that Windows EJ> doesn't come with tar. There is a zlib module in python that EJ> solve the zipped issue. There is also a tarfile module EJ> running around that would make tar unnecessary, but last time EJ> I tried it, it wasn't that stable. Anyone know if its current EJ> state? I havent tried it but saw the 0.4 announcement. I believe it fixes many issues. Unfortunately I cant seem to reach the web page. FWIW here it is: http://www.gustaebel.de/lars/tarfile/ prabhu
participants (5)
-
eric jones
-
Pearu Peterson
-
Prabhu Ramachandran
-
Robert Kern
-
Travis Oliphant