I'm catching up with half a year of back postings on the numpy list, and the messages about automatically using vecLib on the macintosh sound very interesting. I maintain the official Package Manager databases for MacPython (which allow users to install a handful of common packages with one mouseclick), and Numeric and numarray have been in there since day one. I'm revising all the packages again at the moment (I do that about once a year, or on request), and I had to manually fiddle Numeric 23.6 to actually build on the Mac. This is a bit of a bother, as I have to keep a separate source distribution in stead of just referring to the official one, but more important is the fact that neither Numeric nor numarray support vecLib out of the box. So, a plea for help: I don't know what the usual frequency for Numeric and numarray distributions is, but it would be very helpful for me (and for anyone using Numeric/numarray on the Mac) if new distributions were available that contained the fixes mentioned in the november discussions... -- Jack Jansen, <Jack.Jansen@cwi.nl>, http://www.cwi.nl/~jack If I can't dance I don't want to be part of your revolution -- Emma Goldman
On Thu, 2005-01-06 at 09:54, Jack Jansen wrote:
I'm catching up with half a year of back postings on the numpy list, and the messages about automatically using vecLib on the macintosh sound very interesting.
I maintain the official Package Manager databases for MacPython (which allow users to install a handful of common packages with one mouseclick), and Numeric and numarray have been in there since day one. I'm revising all the packages again at the moment (I do that about once a year, or on request), and I had to manually fiddle Numeric 23.6 to actually build on the Mac. This is a bit of a bother, as I have to keep a separate source distribution in stead of just referring to the official one, but more important is the fact that neither Numeric nor numarray support vecLib out of the box.
So, a plea for help: I don't know what the usual frequency for Numeric and numarray distributions is, but it would be very helpful for me (and for anyone using Numeric/numarray on the Mac) if new distributions were available that contained the fixes mentioned in the november discussions...
numarray-1.2 is relatively near at hand, sometime in the next 2-3 weeks I hope. For numarray-1.2 on the Mac, I think all you will need to do to get a vecLib build is: python setup.py install --use_lapack Regards, Todd
On 6 Jan 2005, at 23:10, Todd Miller wrote:
numarray-1.2 is relatively near at hand, sometime in the next 2-3 weeks I hope. For numarray-1.2 on the Mac, I think all you will need to do to get a vecLib build is:
python setup.py install --use_lapack
Is there a reason to require the "--use_lapack"? I.e. are there any adverse consequences to using it? -- Jack Jansen, <Jack.Jansen@cwi.nl>, http://www.cwi.nl/~jack If I can't dance I don't want to be part of your revolution -- Emma Goldman
Jack Jansen wrote:
On 6 Jan 2005, at 23:10, Todd Miller wrote:
numarray-1.2 is relatively near at hand, sometime in the next 2-3 weeks I hope. For numarray-1.2 on the Mac, I think all you will need to do to get a vecLib build is:
python setup.py install --use_lapack
Is there a reason to require the "--use_lapack"? I.e. are there any adverse consequences to using it?
On other platforms, one has to edit the setup scripts to add the information about where the libraries are. The default fallback is to use the unoptimized version packaged with numarray. The alternative would be to add autoconf-like capabilities to the setup script such that it could determine if the libraries were in the default places (and valid!), then fall back to the lite versions if not. On the Mac, --use_lapack should have no adverse consequences, if I'm reading you right. On other platforms, numarray might fail to build correctly if one hadn't supplied the necessary information. -- Robert Kern rkern@ucsd.edu "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter
On 7 Jan 2005, at 10:53, Robert Kern wrote:
Jack Jansen wrote:
On 6 Jan 2005, at 23:10, Todd Miller wrote:
numarray-1.2 is relatively near at hand, sometime in the next 2-3 weeks I hope. For numarray-1.2 on the Mac, I think all you will need to do to get a vecLib build is:
python setup.py install --use_lapack Is there a reason to require the "--use_lapack"? I.e. are there any adverse consequences to using it?
On other platforms, one has to edit the setup scripts to add the information about where the libraries are. The default fallback is to use the unoptimized version packaged with numarray.
The alternative would be to add autoconf-like capabilities to the setup script such that it could determine if the libraries were in the default places (and valid!), then fall back to the lite versions if not.
Ah, I see. So the problem is really that the library detection code hasn't been written. Hmm, having a look at the code, it seems that it should be fairly simple to fix (but I'm not completely sure I understand the interdependencies between setup.py, generate.py and addons.py, so I don't dare creating a patch). If the whole lapack section of addons was restructured like if os.environ.has_key('LINALG_LIB'): set things up for using that path elif os.path.exists('/usr/local/lib/atlas') use that elif os.path.exists('/System/Library/Frameworks/vecLib.framework') use that else use builtin blas_atlas I think it would have the same functionality as now but without need for the -use_lapack option. OTOH I may be oversimplifying things, I have no idea how these numerical libraries would normally be installed on Linux or other unixen, let alone on Windows. -- Jack Jansen, <Jack.Jansen@cwi.nl>, http://www.cwi.nl/~jack If I can't dance I don't want to be part of your revolution -- Emma Goldman
Jack Jansen wrote:
If the whole lapack section of addons was restructured like if os.environ.has_key('LINALG_LIB'): set things up for using that path elif os.path.exists('/usr/local/lib/atlas') use that elif os.path.exists('/System/Library/Frameworks/vecLib.framework') use that else use builtin blas_atlas
If I may ask, it would be great if /usr/lib/(atlas/ATLAS) were added to these default search paths, like the scipy setup.py file does. In a mixed-architecture environment, where /usr/local is often NFS shared, one must put things like ATLAS in machine-specific locations. One simple solution is to put it directly in /usr/lib/atlas, instead of /usr/local/lib/atlas, since /usr/lib is rarely NFS-shared. This gives a way to share over NFS the bulk of things which are built from source, while leaving architecture-specific things in a location where they don't cause conflicts. Numpy and scipy already have this search path, so hopefully numarray can adopt the same convention as well. It's nice to be able to just unpack those and, without needing to set absolutely anything, simply say './setup.py bdist_rpm' and be done :) Cheers, f
On Fri, 2005-01-07 at 13:33, Fernando Perez wrote:
Jack Jansen wrote:
If the whole lapack section of addons was restructured like if os.environ.has_key('LINALG_LIB'): set things up for using that path elif os.path.exists('/usr/local/lib/atlas') use that elif os.path.exists('/System/Library/Frameworks/vecLib.framework') use that else use builtin blas_atlas
If I may ask, it would be great if /usr/lib/(atlas/ATLAS) were added to these default search paths, like the scipy setup.py file does. In a mixed-architecture environment, where /usr/local is often NFS shared, one must put things like ATLAS in machine-specific locations. One simple solution is to put it directly in /usr/lib/atlas, instead of /usr/local/lib/atlas, since /usr/lib is rarely NFS-shared. This gives a way to share over NFS the bulk of things which are built from source, while leaving architecture-specific things in a location where they don't cause conflicts.
Numpy and scipy already have this search path, so hopefully numarray can adopt the same convention as well. It's nice to be able to just unpack those and, without needing to set absolutely anything, simply say './setup.py bdist_rpm' and be done :)
Cheers,
f
These sound like reasonable ideas but I want to mull it over some and I'm pretty much out of time this week... I'm supposed to be working on the scipy to numarray port. Both ideas look like they may be easy but I'm out of oomph and... they may not. Regards, Todd
Todd Miller wrote: [path ideas]
These sound like reasonable ideas but I want to mull it over some and I'm pretty much out of time this week... I'm supposed to be working on the scipy to numarray port. Both ideas look like they may be easy but I'm out of oomph and... they may not.
No worries. Right now I'm enjoying setting up a yum-based system for handling Atlas/Numeric/scipy on a group of machines with architecture-specific ATLASes (P3, P4, P4+hyperthreading,...). So I sort of have these build issues very much in front of me, but there's no hurry in incorporating them into numarray. The scipy work is definitely a priority. But thanks for considering the input. Cheers, f
Fernando Perez wrote:
No worries. Right now I'm enjoying setting up a yum-based system for handling Atlas/Numeric/scipy on a group of machines with architecture-specific ATLASes (P3, P4, P4+hyperthreading,...).
Ooh, ooh, I want a copy when you're done! Is 'enjoying' the right verb there?
Stephen Walton wrote:
Fernando Perez wrote:
No worries. Right now I'm enjoying setting up a yum-based system for handling Atlas/Numeric/scipy on a group of machines with architecture-specific ATLASes (P3, P4, P4+hyperthreading,...).
Ooh, ooh, I want a copy when you're done!
I actually had you in mind this morning, b/c I remember you've asked about this before. My solution is messy, but I think it's going to work OK. I'll probably post a little writeup about it later. It may be useful to people.
Is 'enjoying' the right verb there?
As they say in mountaineering, "it doesn't have to be fun to be fun" :) Cheers, f
On Jan 7, 2005, at 10:33 AM, Fernando Perez wrote:
Jack Jansen wrote:
If the whole lapack section of addons was restructured like if os.environ.has_key('LINALG_LIB'): set things up for using that path elif os.path.exists('/usr/local/lib/atlas') use that elif os.path.exists('/System/Library/Frameworks/vecLib.framework') use that else use builtin blas_atlas
If I may ask, it would be great if /usr/lib/(atlas/ATLAS) were added to these default search paths, like the scipy setup.py file does.
I would much rather that previous snippet of code look something like: if sys.scipypath: # Or some other flag/global/something use whatever is indicated else: if os.environ.has_key('LINALG_LIB'): set things up for using that path elif os.path.exists('/usr/local/lib/atlas') use that elif os.path.exists('/System/Library/Frameworks/vecLib.framework') use that else use builtin blas_atlas In addition, some of us do not trust anything in /usr for production work. This is to help make our system administrators lives easier. If I only use things from, say /tools, the sysadmins can completely erase and reload workstations for the purposes of bug fixes, security updates, etc. without disturbing my work. This prevents, "GAAAHHH! You upgraded the machine and now everything is using Foo_1.1.1 instead of Foo_1.1.0 and now everything is broken." Most Linux distributions are particularly bad about this. This affliction is also known as "Perl Hell". ;) -a
On Fri, 7 Jan 2005, Jack Jansen wrote:
The alternative would be to add autoconf-like capabilities to the setup script such that it could determine if the libraries were in the default places (and valid!), then fall back to the lite versions if not.
Ah, I see. So the problem is really that the library detection code hasn't been written.
FYI, scipy_distutils has rather general lapack/blas/atlas detection code facilities. It first looks for atlas/lapack libraries, then for blas/lapack libraries, and finally for blas/lapack Fortran sources that scipy_distutils would compile behind the scenes. See scipy_distutils/system_info.py and scipy/Lib/lib/lapack/setup_lapack.py for more information. Pearu
Pearu Peterson wrote:
FYI, scipy_distutils has rather general lapack/blas/atlas detection code facilities. It first looks for atlas/lapack libraries, then for blas/lapack libraries
This makes me happy, as Gentoo Linux puts atlas in: /usr/lib/blas/atlas /usr/lib/lapack/atlas However, I'm all for any system that "just works" on the most common systems, and allows me to specify my weirdo system if need be. Being an OS-X user, I'd be very happy if it "just works" there. I expect to tweak things when I'm running Gentoo. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
On Fri, 2005-01-07 at 04:53, Robert Kern wrote:
Jack Jansen wrote:
On 6 Jan 2005, at 23:10, Todd Miller wrote:
numarray-1.2 is relatively near at hand, sometime in the next 2-3 weeks I hope. For numarray-1.2 on the Mac, I think all you will need to do to get a vecLib build is:
python setup.py install --use_lapack
Is there a reason to require the "--use_lapack"? I.e. are there any adverse consequences to using it?
On other platforms, one has to edit the setup scripts to add the information about where the libraries are. The default fallback is to use the unoptimized version packaged with numarray.
The alternative would be to add autoconf-like capabilities to the setup script such that it could determine if the libraries were in the default places (and valid!), then fall back to the lite versions if not.
On the Mac, --use_lapack should have no adverse consequences, if I'm reading you right. On other platforms, numarray might fail to build correctly if one hadn't supplied the necessary information.
Since I'm not a Mac user, I'll beat a dead horse. Are we all agreed that: 1. vecLib is universally available on OS-X. 2. Using vecLib rather than blaslite is preferred. If so, I'll make it the default. Regards, Todd
Todd Miller wrote:
Since I'm not a Mac user, I'll beat a dead horse. Are we all agreed that:
1. vecLib is universally available on OS-X.
Yes.
2. Using vecLib rather than blaslite is preferred.
Yes.
If so, I'll make it the default.
Woohoo! Thank you! -- Robert Kern rkern@ucsd.edu "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter
participants (8)
-
Andrew P. Lentvorski, Jr. -
Chris Barker -
Fernando Perez -
Jack Jansen -
Pearu Peterson -
Robert Kern -
Stephen Walton -
Todd Miller