numarray patch: use Apple's vecLib on Mac OS X
Hi all, Here's a patch for numarray (against current CVS) to build using Apple's vecLib (optimized BLAS library) on Mac OS X. This was inspired by/stolen from Bob Ippolito's recent Numeric patch ( http://mail.python.org/pipermail/pythonmac-sig/2004-November/ 012077.html ). To build with vecLib, set the environment variable USE_LAPACK=1. I can test any modifications needed to incorporate this into the main release. After building with this patch (both with and without vecLib) I ran Lib/testall.py with no problems -- are there more tests I should do? Cheers! Andrew Index: addons.py =================================================================== RCS file: /cvsroot/numpy/numarray/addons.py,v retrieving revision 1.8 diff -c -r1.8 addons.py *** addons.py 3 Nov 2004 15:06:33 -0000 1.8 --- addons.py 18 Nov 2004 15:43:01 -0000 *************** *** 24,31 **** --- 24,33 ---- os.path.join('Packages/LinearAlgebra2/Src', 'zlapack_lite.c'), os.path.join('Packages/LinearAlgebra2/Src', 'dlapack_lite.c') ] + lapack_include_dirs = ["Packages/LinearAlgebra2/Src", 'Include/numarray'] lapack_libs = [] lapack_dirs = [] + lapack_link_args = [] else: sourcelist = [ os.path.join('Packages/LinearAlgebra2/Src', 'lapack_litemodule.c'), *************** *** 53,58 **** --- 55,70 ---- # lapack_dirs = ['/home/jmiller/work/lib'] + lapack_link_args = [] + lapack_include_dirs = ["Packages/LinearAlgebra2/Src", 'Include/numarray'] + + VECLIB_PATH = '/System/Library/Frameworks/vecLib.framework' + if os.path.exists(VECLIB_PATH): + lapack_link_args = ['-framework', 'vecLib'] + lapack_include_dirs.append( os.path.join(VECLIB_PATH, 'Headers')) + lapack_libs = [] + lapack_dirs = [] + ADDONS_DATA_FILES = [('numarray/ma', ['Packages/MA/Legal.htm'])] ADDONS_PACKAGES = ["numarray.convolve", *************** *** 102,110 **** include_dirs = ['Packages/FFT2/Include','Include/numarray']), Extension('numarray.linear_algebra.lapack_lite2', sourcelist, ! include_dirs = ["Packages/LinearAlgebra2/Src", 'Include/numarray'], library_dirs = lapack_dirs, ! libraries = lapack_libs), Extension("numarray.random_array.ranlib2", ["Packages/RandomArray2/Src/ranlib.c", "Packages/RandomArray2/Src/ranlibmodule.c", --- 114,124 ---- include_dirs = ['Packages/FFT2/Include','Include/numarray']), Extension('numarray.linear_algebra.lapack_lite2', sourcelist, ! include_dirs = lapack_include_dirs, library_dirs = lapack_dirs, ! libraries = lapack_libs, ! extra_link_args = lapack_link_args, ! ), Extension("numarray.random_array.ranlib2", ["Packages/RandomArray2/Src/ranlib.c", "Packages/RandomArray2/Src/ranlibmodule.c",
Hi, I've noticed your patch for MacOS X, and I think that a better solution should use sys.platform instead of a path search, since Apple could always change its framework location. I've also removed the VECLIB_PATH/Headers from the include path because AFAIK Apple gcc already knows where to search for the vecLib framework. Here it is a diff output between my addons.py and the CVS one. HTH, Andrea.
** /Users/andrea/Installs/numarray-1.1.1/addons-mine.py Mon Nov 22 12:39:58 2004 --- /Users/andrea/Installs/numarray-1.1.1/addons-cvs.py Fri Nov 19 10:40:45 2004 *************** *** 63,73 ****
# lapack_dirs = ['/home/jmiller/work/lib']
! if sys.platform == "darwin": ! lapack_include_dirs = ["Packages/LinearAlgebra2/Src",'Include/numarray'] lapack_libs = [] lapack_dirs = [] - lapack_link_args = ['-framework', 'vecLib']
ADDONS_DATA_FILES = [('numarray/ma', ['Packages/MA/Legal.htm'])]
--- 63,74 ----
# lapack_dirs = ['/home/jmiller/work/lib']
! VECLIB_PATH = '/System/Library/Frameworks/vecLib.framework' ! if os.path.exists(VECLIB_PATH): ! lapack_link_args = ['-framework', 'vecLib'] ! lapack_include_dirs.append( os.path.join(VECLIB_PATH, 'Headers')) lapack_libs = [] lapack_dirs = []
ADDONS_DATA_FILES = [('numarray/ma', ['Packages/MA/Legal.htm'])]
On 18 Nov 2004, at 16:53, Andrew Straw wrote:
Hi all,
Here's a patch for numarray (against current CVS) to build using Apple's vecLib (optimized BLAS library) on Mac OS X. This was inspired by/stolen from Bob Ippolito's recent Numeric patch ( http://mail.python.org/pipermail/pythonmac-sig/2004-November/ 012077.html ). To build with vecLib, set the environment variable USE_LAPACK=1. I can test any modifications needed to incorporate this into the main release. After building with this patch (both with and without vecLib) I ran Lib/testall.py with no problems -- are there more tests I should do?
Cheers! Andrew
Index: addons.py =================================================================== RCS file: /cvsroot/numpy/numarray/addons.py,v retrieving revision 1.8 diff -c -r1.8 addons.py *** addons.py 3 Nov 2004 15:06:33 -0000 1.8 --- addons.py 18 Nov 2004 15:43:01 -0000 *************** *** 24,31 **** --- 24,33 ---- os.path.join('Packages/LinearAlgebra2/Src', 'zlapack_lite.c'), os.path.join('Packages/LinearAlgebra2/Src', 'dlapack_lite.c') ] + lapack_include_dirs = ["Packages/LinearAlgebra2/Src", 'Include/numarray'] lapack_libs = [] lapack_dirs = [] + lapack_link_args = [] else: sourcelist = [ os.path.join('Packages/LinearAlgebra2/Src', 'lapack_litemodule.c'), *************** *** 53,58 **** --- 55,70 ----
# lapack_dirs = ['/home/jmiller/work/lib']
+ lapack_link_args = [] + lapack_include_dirs = ["Packages/LinearAlgebra2/Src", 'Include/numarray'] + + VECLIB_PATH = '/System/Library/Frameworks/vecLib.framework' + if os.path.exists(VECLIB_PATH): + lapack_link_args = ['-framework', 'vecLib'] + lapack_include_dirs.append( os.path.join(VECLIB_PATH, 'Headers')) + lapack_libs = [] + lapack_dirs = [] + ADDONS_DATA_FILES = [('numarray/ma', ['Packages/MA/Legal.htm'])]
ADDONS_PACKAGES = ["numarray.convolve", *************** *** 102,110 **** include_dirs = ['Packages/FFT2/Include','Include/numarray']), Extension('numarray.linear_algebra.lapack_lite2', sourcelist, ! include_dirs = ["Packages/LinearAlgebra2/Src", 'Include/numarray'], library_dirs = lapack_dirs, ! libraries = lapack_libs), Extension("numarray.random_array.ranlib2", ["Packages/RandomArray2/Src/ranlib.c", "Packages/RandomArray2/Src/ranlibmodule.c", --- 114,124 ---- include_dirs = ['Packages/FFT2/Include','Include/numarray']), Extension('numarray.linear_algebra.lapack_lite2', sourcelist, ! include_dirs = lapack_include_dirs, library_dirs = lapack_dirs, ! libraries = lapack_libs, ! extra_link_args = lapack_link_args, ! ), Extension("numarray.random_array.ranlib2", ["Packages/RandomArray2/Src/ranlib.c", "Packages/RandomArray2/Src/ranlibmodule.c",
------------------------------------------------------- This SF.Net email is sponsored by: InterSystems CACHE FREE OODBMS DOWNLOAD - A multidimensional database that combines robust object and relational technologies, making it a perfect match for Java, C++,COM, XML, ODBC and JDBC. www.intersystems.com/match8 _______________________________________________ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
On Mon, 2004-11-22 at 06:49, Andrea Riciputi wrote:
Hi, I've noticed your patch for MacOS X, and I think that a better solution should use sys.platform instead of a path search, since Apple could always change its framework location. I've also removed the VECLIB_PATH/Headers from the include path because AFAIK Apple gcc already knows where to search for the vecLib framework. Here it is a diff output between my addons.py and the CVS one.
Not to look a patch-horse in the mouth, but IMHO, the sys.platform idea is good but the VECLIB_PATH elimination maybe less so. The reason I question the VECLIB_PATH change is that we're removing explicit and generally harmless extra information in exchange for two assumptions: (1) gcc knows the path already (2) the numarray Mac user is using gcc. Other opinions? Mac users please speak up if you think the VECLIB_PATH change is a good one or it won't get done. Thanks for the suggestions. I'll make the sys.platform change later today. Regards, Todd
HTH, Andrea.
** /Users/andrea/Installs/numarray-1.1.1/addons-mine.py Mon Nov 22 12:39:58 2004 --- /Users/andrea/Installs/numarray-1.1.1/addons-cvs.py Fri Nov 19 10:40:45 2004 *************** *** 63,73 ****
# lapack_dirs = ['/home/jmiller/work/lib']
! if sys.platform == "darwin": ! lapack_include_dirs = ["Packages/LinearAlgebra2/Src",'Include/numarray'] lapack_libs = [] lapack_dirs = [] - lapack_link_args = ['-framework', 'vecLib']
ADDONS_DATA_FILES = [('numarray/ma', ['Packages/MA/Legal.htm'])]
--- 63,74 ----
# lapack_dirs = ['/home/jmiller/work/lib']
! VECLIB_PATH = '/System/Library/Frameworks/vecLib.framework' ! if os.path.exists(VECLIB_PATH): ! lapack_link_args = ['-framework', 'vecLib'] ! lapack_include_dirs.append( os.path.join(VECLIB_PATH, 'Headers')) lapack_libs = [] lapack_dirs = []
ADDONS_DATA_FILES = [('numarray/ma', ['Packages/MA/Legal.htm'])]
On 18 Nov 2004, at 16:53, Andrew Straw wrote:
Hi all,
Here's a patch for numarray (against current CVS) to build using Apple's vecLib (optimized BLAS library) on Mac OS X. This was inspired by/stolen from Bob Ippolito's recent Numeric patch ( http://mail.python.org/pipermail/pythonmac-sig/2004-November/ 012077.html ). To build with vecLib, set the environment variable USE_LAPACK=1. I can test any modifications needed to incorporate this into the main release. After building with this patch (both with and without vecLib) I ran Lib/testall.py with no problems -- are there more tests I should do?
Cheers! Andrew
Index: addons.py =================================================================== RCS file: /cvsroot/numpy/numarray/addons.py,v retrieving revision 1.8 diff -c -r1.8 addons.py *** addons.py 3 Nov 2004 15:06:33 -0000 1.8 --- addons.py 18 Nov 2004 15:43:01 -0000 *************** *** 24,31 **** --- 24,33 ---- os.path.join('Packages/LinearAlgebra2/Src', 'zlapack_lite.c'), os.path.join('Packages/LinearAlgebra2/Src', 'dlapack_lite.c') ] + lapack_include_dirs = ["Packages/LinearAlgebra2/Src", 'Include/numarray'] lapack_libs = [] lapack_dirs = [] + lapack_link_args = [] else: sourcelist = [ os.path.join('Packages/LinearAlgebra2/Src', 'lapack_litemodule.c'), *************** *** 53,58 **** --- 55,70 ----
# lapack_dirs = ['/home/jmiller/work/lib']
+ lapack_link_args = [] + lapack_include_dirs = ["Packages/LinearAlgebra2/Src", 'Include/numarray'] + + VECLIB_PATH = '/System/Library/Frameworks/vecLib.framework' + if os.path.exists(VECLIB_PATH): + lapack_link_args = ['-framework', 'vecLib'] + lapack_include_dirs.append( os.path.join(VECLIB_PATH, 'Headers')) + lapack_libs = [] + lapack_dirs = [] + ADDONS_DATA_FILES = [('numarray/ma', ['Packages/MA/Legal.htm'])]
ADDONS_PACKAGES = ["numarray.convolve", *************** *** 102,110 **** include_dirs = ['Packages/FFT2/Include','Include/numarray']), Extension('numarray.linear_algebra.lapack_lite2', sourcelist, ! include_dirs = ["Packages/LinearAlgebra2/Src", 'Include/numarray'], library_dirs = lapack_dirs, ! libraries = lapack_libs), Extension("numarray.random_array.ranlib2", ["Packages/RandomArray2/Src/ranlib.c", "Packages/RandomArray2/Src/ranlibmodule.c", --- 114,124 ---- include_dirs = ['Packages/FFT2/Include','Include/numarray']), Extension('numarray.linear_algebra.lapack_lite2', sourcelist, ! include_dirs = lapack_include_dirs, library_dirs = lapack_dirs, ! libraries = lapack_libs, ! extra_link_args = lapack_link_args, ! ), Extension("numarray.random_array.ranlib2", ["Packages/RandomArray2/Src/ranlib.c", "Packages/RandomArray2/Src/ranlibmodule.c",
------------------------------------------------------- This SF.Net email is sponsored by: InterSystems CACHE FREE OODBMS DOWNLOAD - A multidimensional database that combines robust object and relational technologies, making it a perfect match for Java, C++,COM, XML, ODBC and JDBC. www.intersystems.com/match8 _______________________________________________ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
------------------------------------------------------- SF email is sponsored by - The IT Product Guide Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://productguide.itmanagersjournal.com/ _______________________________________________ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion --
I see your point but AFAIK the links option "-framework vecLib" will work *only* with Apple gcc (ld) and Apple gcc (ld) does know where frameworks are. From Apple ld man page:
-framework name[,suffix] Specifies a framework to link against. Frameworks are dynamic shared libraries, but they are stored in different locations, and therefore must be searched for differently. When this option is specified, ld searches for framework `name.framework/name' first in any directories specified with the -F option, then in the standard framework directories /Library/Frameworks, /Net- work/Library/Frameworks, and /System/Library/Frameworks. The placement of the -framework option is significant, as it deter- mines when and how the framework is searched. If the optional suffix is specified the framework is first searched for the name with the suffix and then without.
-Fdir In Apple's version of GCC only, add the directory dir to the head of the list of directories to be searched for frameworks.
The framework search algorithm is, for an inclusion of <Fmwk/Header.h>, to look for files named path/Fmwk.framework/Head- ers/Header.h or path/Fmwk.framework/PrivateHeaders/Header.h where path includes /System/Library/Frameworks/ /Library/Frameworks/, and /Local/Library/Frameworks/, plus any additional paths specified by -F.
HTH, Andrea. On 22 Nov 2004, at 17:08, Todd Miller wrote:
Not to look a patch-horse in the mouth, but IMHO, the sys.platform idea is good but the VECLIB_PATH elimination maybe less so. The reason I question the VECLIB_PATH change is that we're removing explicit and generally harmless extra information in exchange for two assumptions: (1) gcc knows the path already (2) the numarray Mac user is using gcc. Other opinions? Mac users please speak up if you think the VECLIB_PATH change is a good one or it won't get done.
Thanks for the suggestions. I'll make the sys.platform change later today.
Regards, Todd
Andrea Riciputi wrote:
I see your point but AFAIK the links option "-framework vecLib" will work *only* with Apple gcc (ld) and Apple gcc (ld) does know where frameworks are. From Apple ld man page:
IBM's xlc will also find the appropriate framework, but as you can see this is only a link-time option and has nothing to do with finding headers. Using #include <vecLib/vecLib.h> will find the appropriate header without any command-line options. Using #include <clapack.h> will not. Fortunately, neither one is used nor needed by lapack_litemodule.c, so for this case, no include-path needs to be specified. -- Robert Kern rkern@ucsd.edu "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter
Hi all, I forwarded this thread to Bob Ippolito, the original author of that patch. He's not on this list. I do suggest that any questions about OS-X be sent to the Pythonmac-SIG list, as there are some folks there that really know what they are doing. Anyway, here's his comments: Bob Ippolito wrote: Two things: 1) Apple can't (or rather, won't ever) change their framework location. 2) Darwin might be Darwin and not Mac OS X, which won't have vecLib. 3) Checking for both is silly, because you won't reasonably ever have the path to vecLib on a non-Darwin platform.. and if you somehow do, it'll probably work anyway. I can give more flexible code with regard to locating frameworks in an identical manner to what gcc and dyld do, but that is only going to be applicable if you work at Apple, or you are building against an SDK, neither of which happen very often in practice. With regard to building against a SDK, the API used by Numeric that is provided by vecLib isn't going to change, ever, unless Numeric wants more stuff that is provided only in OS X 10.4+, for example. This would be an issue when and if it happens, but in that case you would probably link against the NEW name (Accelerate.framework). I'm not sure if the vecLib API is supported by OS X 10.1, but someone else is going to have to care about that because I sure don't. Alternate linkers isn't really a concern because the three command-line compilers that matter probably all support the -framework flag. Apple's GCC (obviously..) and IBM's XLC (according to Robert) certainly do, and more likely than not, MetroWerks CodeWarrior also does (someone else would have to confirm this). If Fink or Gentoo ships some purely GNU-based compiler/linker that doesn't support -framework, I'd consider that a bug. I'm not sure how they'd get many things to compile correctly without it, unless they're emulating it by specifying paths directly to the Mach-O MH_DYLIB files (/System/Library/Frameworks/vecLib.framework/vecLib, for example) PLUS using non-framework-style header paths everywhere, since <vecLib/vecLib.h> would never work if you -I to anywhere in the framework tree. Does an unpatched GNU toolchain even understand Mach-O in the first place? Most likely a non-issue in any case, but I'm not about to Finkify any machine to test that theory. In summary, I wrote the patch in that way for the specific reason of doing it as correctly as possible without overcomplicating things. I hope that people would trust that I know what I'm doing. -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
On Mon, 2004-11-22 at 16:11 -0800, Chris Barker wrote:
Hi all,
I forwarded this thread to Bob Ippolito, the original author of that patch. He's not on this list. I do suggest that any questions about OS-X be sent to the Pythonmac-SIG list,
I'll try...
as there are some folks there that really know what they are doing. Anyway, here's his comments:
Bob Ippolito wrote:
Two things: 1) Apple can't (or rather, won't ever) change their framework location. 2) Darwin might be Darwin and not Mac OS X, which won't have vecLib. 3) Checking for both is silly, because you won't reasonably ever have the path to vecLib on a non-Darwin platform.. and if you somehow do, it'll probably work anyway.
That's enough for me. I'm thinking we should just keep Andrew's port of Bob's patch, already committed to CVS, as is. If any one has any further comments about this please let me know. I realize I'm probably blowing around like a weather vane, but I'm no Mac expert, so I'm relying on your collective guidance. Thanks everyone for their input: Bob, Andrew, Andrea, Robert, and Chris. Regards, Todd [snip]
On Thu, 2004-11-18 at 10:53, Andrew Straw wrote:
Hi all,
Hi Andrew, sorry about not getting back to this sooner. I actually committed this some time last week but forgot to follow up on the e-mail. Anyway, well done. Patches that work are one of my favorite things.
Here's a patch for numarray (against current CVS) to build using Apple's vecLib (optimized BLAS library) on Mac OS X. This was inspired by/stolen from Bob Ippolito's recent Numeric patch ( http://mail.python.org/pipermail/pythonmac-sig/2004-November/ 012077.html ). To build with vecLib, set the environment variable USE_LAPACK=1. I can test any modifications needed to incorporate this into the main release. After building with this patch (both with and without vecLib) I ran Lib/testall.py with no problems -- are there more tests I should do?
No. All of the self-tests for numarray, with the exception of the f2py support, are integrated into numarray.testall. The tests can always be improved, but they are intended to be centralized so that should always be the only place you have to look. Thanks again for the patch. Regards, Todd
Cheers! Andrew
Index: addons.py =================================================================== RCS file: /cvsroot/numpy/numarray/addons.py,v retrieving revision 1.8 diff -c -r1.8 addons.py *** addons.py 3 Nov 2004 15:06:33 -0000 1.8 --- addons.py 18 Nov 2004 15:43:01 -0000 *************** *** 24,31 **** --- 24,33 ---- os.path.join('Packages/LinearAlgebra2/Src', 'zlapack_lite.c'), os.path.join('Packages/LinearAlgebra2/Src', 'dlapack_lite.c') ] + lapack_include_dirs = ["Packages/LinearAlgebra2/Src", 'Include/numarray'] lapack_libs = [] lapack_dirs = [] + lapack_link_args = [] else: sourcelist = [ os.path.join('Packages/LinearAlgebra2/Src', 'lapack_litemodule.c'), *************** *** 53,58 **** --- 55,70 ----
# lapack_dirs = ['/home/jmiller/work/lib']
+ lapack_link_args = [] + lapack_include_dirs = ["Packages/LinearAlgebra2/Src", 'Include/numarray'] + + VECLIB_PATH = '/System/Library/Frameworks/vecLib.framework' + if os.path.exists(VECLIB_PATH): + lapack_link_args = ['-framework', 'vecLib'] + lapack_include_dirs.append( os.path.join(VECLIB_PATH, 'Headers')) + lapack_libs = [] + lapack_dirs = [] + ADDONS_DATA_FILES = [('numarray/ma', ['Packages/MA/Legal.htm'])]
ADDONS_PACKAGES = ["numarray.convolve", *************** *** 102,110 **** include_dirs = ['Packages/FFT2/Include','Include/numarray']), Extension('numarray.linear_algebra.lapack_lite2', sourcelist, ! include_dirs = ["Packages/LinearAlgebra2/Src", 'Include/numarray'], library_dirs = lapack_dirs, ! libraries = lapack_libs), Extension("numarray.random_array.ranlib2", ["Packages/RandomArray2/Src/ranlib.c", "Packages/RandomArray2/Src/ranlibmodule.c", --- 114,124 ---- include_dirs = ['Packages/FFT2/Include','Include/numarray']), Extension('numarray.linear_algebra.lapack_lite2', sourcelist, ! include_dirs = lapack_include_dirs, library_dirs = lapack_dirs, ! libraries = lapack_libs, ! extra_link_args = lapack_link_args, ! ), Extension("numarray.random_array.ranlib2", ["Packages/RandomArray2/Src/ranlib.c", "Packages/RandomArray2/Src/ranlibmodule.c",
------------------------------------------------------- This SF.Net email is sponsored by: InterSystems CACHE FREE OODBMS DOWNLOAD - A multidimensional database that combines robust object and relational technologies, making it a perfect match for Java, C++,COM, XML, ODBC and JDBC. www.intersystems.com/match8 _______________________________________________ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
--
participants (5)
-
Andrea Riciputi -
Andrew Straw -
Chris Barker -
Robert Kern -
Todd Miller