numarray patch: use Apple's vecLib on Mac OS X
![](https://secure.gravatar.com/avatar/d5321459a9b36ca748932987de93e083.jpg?s=120&d=mm&r=g)
Hi all, Here's a patch for numarray (against current CVS) to build using Apple's vecLib (optimized BLAS library) on Mac OS X. This was inspired by/stolen from Bob Ippolito's recent Numeric patch ( http://mail.python.org/pipermail/pythonmac-sig/2004-November/ 012077.html ). To build with vecLib, set the environment variable USE_LAPACK=1. I can test any modifications needed to incorporate this into the main release. After building with this patch (both with and without vecLib) I ran Lib/testall.py with no problems -- are there more tests I should do? Cheers! Andrew Index: addons.py =================================================================== RCS file: /cvsroot/numpy/numarray/addons.py,v retrieving revision 1.8 diff -c -r1.8 addons.py *** addons.py 3 Nov 2004 15:06:33 -0000 1.8 --- addons.py 18 Nov 2004 15:43:01 -0000 *************** *** 24,31 **** --- 24,33 ---- os.path.join('Packages/LinearAlgebra2/Src', 'zlapack_lite.c'), os.path.join('Packages/LinearAlgebra2/Src', 'dlapack_lite.c') ] + lapack_include_dirs = ["Packages/LinearAlgebra2/Src", 'Include/numarray'] lapack_libs = [] lapack_dirs = [] + lapack_link_args = [] else: sourcelist = [ os.path.join('Packages/LinearAlgebra2/Src', 'lapack_litemodule.c'), *************** *** 53,58 **** --- 55,70 ---- # lapack_dirs = ['/home/jmiller/work/lib'] + lapack_link_args = [] + lapack_include_dirs = ["Packages/LinearAlgebra2/Src", 'Include/numarray'] + + VECLIB_PATH = '/System/Library/Frameworks/vecLib.framework' + if os.path.exists(VECLIB_PATH): + lapack_link_args = ['-framework', 'vecLib'] + lapack_include_dirs.append( os.path.join(VECLIB_PATH, 'Headers')) + lapack_libs = [] + lapack_dirs = [] + ADDONS_DATA_FILES = [('numarray/ma', ['Packages/MA/Legal.htm'])] ADDONS_PACKAGES = ["numarray.convolve", *************** *** 102,110 **** include_dirs = ['Packages/FFT2/Include','Include/numarray']), Extension('numarray.linear_algebra.lapack_lite2', sourcelist, ! include_dirs = ["Packages/LinearAlgebra2/Src", 'Include/numarray'], library_dirs = lapack_dirs, ! libraries = lapack_libs), Extension("numarray.random_array.ranlib2", ["Packages/RandomArray2/Src/ranlib.c", "Packages/RandomArray2/Src/ranlibmodule.c", --- 114,124 ---- include_dirs = ['Packages/FFT2/Include','Include/numarray']), Extension('numarray.linear_algebra.lapack_lite2', sourcelist, ! include_dirs = lapack_include_dirs, library_dirs = lapack_dirs, ! libraries = lapack_libs, ! extra_link_args = lapack_link_args, ! ), Extension("numarray.random_array.ranlib2", ["Packages/RandomArray2/Src/ranlib.c", "Packages/RandomArray2/Src/ranlibmodule.c",
![](https://secure.gravatar.com/avatar/09764ad794a843b7c030ae5ace320f18.jpg?s=120&d=mm&r=g)
Hi, I've noticed your patch for MacOS X, and I think that a better solution should use sys.platform instead of a path search, since Apple could always change its framework location. I've also removed the VECLIB_PATH/Headers from the include path because AFAIK Apple gcc already knows where to search for the vecLib framework. Here it is a diff output between my addons.py and the CVS one. HTH, Andrea.
On 18 Nov 2004, at 16:53, Andrew Straw wrote:
![](https://secure.gravatar.com/avatar/faf9400121dca9940496a7473b1d8179.jpg?s=120&d=mm&r=g)
On Mon, 2004-11-22 at 06:49, Andrea Riciputi wrote:
Not to look a patch-horse in the mouth, but IMHO, the sys.platform idea is good but the VECLIB_PATH elimination maybe less so. The reason I question the VECLIB_PATH change is that we're removing explicit and generally harmless extra information in exchange for two assumptions: (1) gcc knows the path already (2) the numarray Mac user is using gcc. Other opinions? Mac users please speak up if you think the VECLIB_PATH change is a good one or it won't get done. Thanks for the suggestions. I'll make the sys.platform change later today. Regards, Todd
![](https://secure.gravatar.com/avatar/ec366db3649cf13f4061b519193849d6.jpg?s=120&d=mm&r=g)
Andrea Riciputi wrote:
IBM's xlc will also find the appropriate framework, but as you can see this is only a link-time option and has nothing to do with finding headers. Using #include <vecLib/vecLib.h> will find the appropriate header without any command-line options. Using #include <clapack.h> will not. Fortunately, neither one is used nor needed by lapack_litemodule.c, so for this case, no include-path needs to be specified. -- Robert Kern rkern@ucsd.edu "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter
![](https://secure.gravatar.com/avatar/5dde29b54a3f1b76b2541d0a4a9b232c.jpg?s=120&d=mm&r=g)
Hi all, I forwarded this thread to Bob Ippolito, the original author of that patch. He's not on this list. I do suggest that any questions about OS-X be sent to the Pythonmac-SIG list, as there are some folks there that really know what they are doing. Anyway, here's his comments: Bob Ippolito wrote: Two things: 1) Apple can't (or rather, won't ever) change their framework location. 2) Darwin might be Darwin and not Mac OS X, which won't have vecLib. 3) Checking for both is silly, because you won't reasonably ever have the path to vecLib on a non-Darwin platform.. and if you somehow do, it'll probably work anyway. I can give more flexible code with regard to locating frameworks in an identical manner to what gcc and dyld do, but that is only going to be applicable if you work at Apple, or you are building against an SDK, neither of which happen very often in practice. With regard to building against a SDK, the API used by Numeric that is provided by vecLib isn't going to change, ever, unless Numeric wants more stuff that is provided only in OS X 10.4+, for example. This would be an issue when and if it happens, but in that case you would probably link against the NEW name (Accelerate.framework). I'm not sure if the vecLib API is supported by OS X 10.1, but someone else is going to have to care about that because I sure don't. Alternate linkers isn't really a concern because the three command-line compilers that matter probably all support the -framework flag. Apple's GCC (obviously..) and IBM's XLC (according to Robert) certainly do, and more likely than not, MetroWerks CodeWarrior also does (someone else would have to confirm this). If Fink or Gentoo ships some purely GNU-based compiler/linker that doesn't support -framework, I'd consider that a bug. I'm not sure how they'd get many things to compile correctly without it, unless they're emulating it by specifying paths directly to the Mach-O MH_DYLIB files (/System/Library/Frameworks/vecLib.framework/vecLib, for example) PLUS using non-framework-style header paths everywhere, since <vecLib/vecLib.h> would never work if you -I to anywhere in the framework tree. Does an unpatched GNU toolchain even understand Mach-O in the first place? Most likely a non-issue in any case, but I'm not about to Finkify any machine to test that theory. In summary, I wrote the patch in that way for the specific reason of doing it as correctly as possible without overcomplicating things. I hope that people would trust that I know what I'm doing. -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
![](https://secure.gravatar.com/avatar/faf9400121dca9940496a7473b1d8179.jpg?s=120&d=mm&r=g)
On Mon, 2004-11-22 at 16:11 -0800, Chris Barker wrote:
I'll try...
That's enough for me. I'm thinking we should just keep Andrew's port of Bob's patch, already committed to CVS, as is. If any one has any further comments about this please let me know. I realize I'm probably blowing around like a weather vane, but I'm no Mac expert, so I'm relying on your collective guidance. Thanks everyone for their input: Bob, Andrew, Andrea, Robert, and Chris. Regards, Todd [snip]
![](https://secure.gravatar.com/avatar/faf9400121dca9940496a7473b1d8179.jpg?s=120&d=mm&r=g)
On Thu, 2004-11-18 at 10:53, Andrew Straw wrote:
Hi all,
Hi Andrew, sorry about not getting back to this sooner. I actually committed this some time last week but forgot to follow up on the e-mail. Anyway, well done. Patches that work are one of my favorite things.
No. All of the self-tests for numarray, with the exception of the f2py support, are integrated into numarray.testall. The tests can always be improved, but they are intended to be centralized so that should always be the only place you have to look. Thanks again for the patch. Regards, Todd
--
![](https://secure.gravatar.com/avatar/09764ad794a843b7c030ae5ace320f18.jpg?s=120&d=mm&r=g)
Hi, I've noticed your patch for MacOS X, and I think that a better solution should use sys.platform instead of a path search, since Apple could always change its framework location. I've also removed the VECLIB_PATH/Headers from the include path because AFAIK Apple gcc already knows where to search for the vecLib framework. Here it is a diff output between my addons.py and the CVS one. HTH, Andrea.
On 18 Nov 2004, at 16:53, Andrew Straw wrote:
![](https://secure.gravatar.com/avatar/faf9400121dca9940496a7473b1d8179.jpg?s=120&d=mm&r=g)
On Mon, 2004-11-22 at 06:49, Andrea Riciputi wrote:
Not to look a patch-horse in the mouth, but IMHO, the sys.platform idea is good but the VECLIB_PATH elimination maybe less so. The reason I question the VECLIB_PATH change is that we're removing explicit and generally harmless extra information in exchange for two assumptions: (1) gcc knows the path already (2) the numarray Mac user is using gcc. Other opinions? Mac users please speak up if you think the VECLIB_PATH change is a good one or it won't get done. Thanks for the suggestions. I'll make the sys.platform change later today. Regards, Todd
![](https://secure.gravatar.com/avatar/ec366db3649cf13f4061b519193849d6.jpg?s=120&d=mm&r=g)
Andrea Riciputi wrote:
IBM's xlc will also find the appropriate framework, but as you can see this is only a link-time option and has nothing to do with finding headers. Using #include <vecLib/vecLib.h> will find the appropriate header without any command-line options. Using #include <clapack.h> will not. Fortunately, neither one is used nor needed by lapack_litemodule.c, so for this case, no include-path needs to be specified. -- Robert Kern rkern@ucsd.edu "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter
![](https://secure.gravatar.com/avatar/5dde29b54a3f1b76b2541d0a4a9b232c.jpg?s=120&d=mm&r=g)
Hi all, I forwarded this thread to Bob Ippolito, the original author of that patch. He's not on this list. I do suggest that any questions about OS-X be sent to the Pythonmac-SIG list, as there are some folks there that really know what they are doing. Anyway, here's his comments: Bob Ippolito wrote: Two things: 1) Apple can't (or rather, won't ever) change their framework location. 2) Darwin might be Darwin and not Mac OS X, which won't have vecLib. 3) Checking for both is silly, because you won't reasonably ever have the path to vecLib on a non-Darwin platform.. and if you somehow do, it'll probably work anyway. I can give more flexible code with regard to locating frameworks in an identical manner to what gcc and dyld do, but that is only going to be applicable if you work at Apple, or you are building against an SDK, neither of which happen very often in practice. With regard to building against a SDK, the API used by Numeric that is provided by vecLib isn't going to change, ever, unless Numeric wants more stuff that is provided only in OS X 10.4+, for example. This would be an issue when and if it happens, but in that case you would probably link against the NEW name (Accelerate.framework). I'm not sure if the vecLib API is supported by OS X 10.1, but someone else is going to have to care about that because I sure don't. Alternate linkers isn't really a concern because the three command-line compilers that matter probably all support the -framework flag. Apple's GCC (obviously..) and IBM's XLC (according to Robert) certainly do, and more likely than not, MetroWerks CodeWarrior also does (someone else would have to confirm this). If Fink or Gentoo ships some purely GNU-based compiler/linker that doesn't support -framework, I'd consider that a bug. I'm not sure how they'd get many things to compile correctly without it, unless they're emulating it by specifying paths directly to the Mach-O MH_DYLIB files (/System/Library/Frameworks/vecLib.framework/vecLib, for example) PLUS using non-framework-style header paths everywhere, since <vecLib/vecLib.h> would never work if you -I to anywhere in the framework tree. Does an unpatched GNU toolchain even understand Mach-O in the first place? Most likely a non-issue in any case, but I'm not about to Finkify any machine to test that theory. In summary, I wrote the patch in that way for the specific reason of doing it as correctly as possible without overcomplicating things. I hope that people would trust that I know what I'm doing. -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
![](https://secure.gravatar.com/avatar/faf9400121dca9940496a7473b1d8179.jpg?s=120&d=mm&r=g)
On Mon, 2004-11-22 at 16:11 -0800, Chris Barker wrote:
I'll try...
That's enough for me. I'm thinking we should just keep Andrew's port of Bob's patch, already committed to CVS, as is. If any one has any further comments about this please let me know. I realize I'm probably blowing around like a weather vane, but I'm no Mac expert, so I'm relying on your collective guidance. Thanks everyone for their input: Bob, Andrew, Andrea, Robert, and Chris. Regards, Todd [snip]
![](https://secure.gravatar.com/avatar/faf9400121dca9940496a7473b1d8179.jpg?s=120&d=mm&r=g)
On Thu, 2004-11-18 at 10:53, Andrew Straw wrote:
Hi all,
Hi Andrew, sorry about not getting back to this sooner. I actually committed this some time last week but forgot to follow up on the e-mail. Anyway, well done. Patches that work are one of my favorite things.
No. All of the self-tests for numarray, with the exception of the f2py support, are integrated into numarray.testall. The tests can always be improved, but they are intended to be centralized so that should always be the only place you have to look. Thanks again for the patch. Regards, Todd
--
participants (5)
-
Andrea Riciputi
-
Andrew Straw
-
Chris Barker
-
Robert Kern
-
Todd Miller