Revisiting numpy/scipy on 64 bit OSX
Hi, I have been playing with 64 bit numpy/scipy on OSX 10.5 Intel again. I thought everything worked after the last time we discussed this, but I noticed that I had Scipy import failures in for example optimize since the gfortran I used creates 32 bit code per default. So I build a gcc 4.2.4 on OSX, then build python in 64 bit mode (fixing configure.in since Python assumes flags only available with the Apple gcc on Darwin), added a wrapper around gfortran that injects a "-m64" compile flag. Then I checkout out and build numpy (r5671), scipy (r4661) and the current nose release and ran the tests. I had some more failures than I expected (details below). Any thoughts on the failures? I am about to build ATLAS and a couple other dependencies on that box. William Stein is at Scipy 08 and there seems to be at least some interest in 64 bit OSX binaries. If anyone wants one let me know and I can put one together once Scipy 0.7 and Numpy 1.1.2 are out. Cheers, Michael numpy failed tests: **************************************************************** File "/Users/mabshoff/64bit_numpy_scipy/lib/python2.5/site-packages/numpy/lib/tests/test_polynomial.py", line 29, in test_polynomial Failed example: p / q Expected: (poly1d([ 0.33333333]), poly1d([ 1.33333333, 2.66666667])) Got: (poly1d([ 0.333]), poly1d([ 1.333, 2.667])) ********************************************************************** File "/Users/mabshoff/64bit_numpy_scipy/lib/python2.5/site-packages/numpy/lib/tests/test_polynomial.py", line 51, in test_polynomial Failed example: p.integ() Expected: poly1d([ 0.33333333, 1. , 3. , 0. ]) Got: poly1d([ 0.333, 1. , 3. , 0. ]) **********************************************************************File "/Users/mabshoff/64bit_numpy_scipy/lib/python2.5/site-packages/numpy/lib/tests/test_polynomial.py", line 53, in test_polynomialFailed example: p.integ(1) Expected: poly1d([ 0.33333333, 1. , 3. , 0. ]) Got: poly1d([ 0.333, 1. , 3. , 0. ]) ********************************************************************** File "/Users/mabshoff/64bit_numpy_scipy/lib/python2.5/site-packages/numpy/lib/tests/test_polynomial.py", line 55, in test_polynomial Failed example: p.integ(5) Expected: poly1d([ 0.00039683, 0.00277778, 0.025 , 0. , 0. , 0. , 0. , 0. ]) Got: poly1d([ 0. , 0.003, 0.025, 0. , 0. , 0. , 0. , 0. ]) ******************************** File "/Users/mabshoff/64bit_numpy_scipy/lib/python2.5/site-packages/numpy/lib/tests/test_ufunclike.py", line 48, in test_ufunclike Failed example: U.log2(a) Expected: array([ 2.169925 , 1.20163386, 2.70043972]) Got: array([ 2.17 , 1.202, 2.7 ]) ********************************************************************** File "/Users/mabshoff/64bit_numpy_scipy/lib/python2.5/site-packages/numpy/lib/tests/test_ufunclike.py", line 53, in test_ufunclike Failed example: U.log2(a, y) Expected: array([ 2.169925 , 1.20163386, 2.70043972]) Got: array([ 2.17 , 1.202, 2.7 ]) ********************************************************************** File "/Users/mabshoff/64bit_numpy_scipy/lib/python2.5/site-packages/numpy/lib/tests/test_ufunclike.py", line 55, in test_ufunclike Failed example: y Expected: array([ 2.169925 , 1.20163386, 2.70043972]) Got: array([ 2.17 , 1.202, 2.7 ]) ====================================================================== FAIL: test_umath.TestComplexFunctions.test_it ---------------------------------------------------------------------- Traceback (most recent call last): File "/Users/mabshoff/64bit_numpy_scipy/lib/python2.5/site-packages/nose/case.py", line 182, in runTest self.test(*self.arg) File "/Users/mabshoff/64bit_numpy_scipy/lib/python2.5/site-packages/numpy/core/tests/test_umath.py", line 196, in test_it assert_almost_equal(fz.real, fr, err_msg='real part %s'%f) File "/Users/mabshoff/64bit_numpy_scipy/lib/python2.5/site-packages/numpy/testing/utils.py", line 207, in assert_almost_equal assert round(abs(desired - actual),decimal) == 0, msg AssertionError: Items are not equal: real part <ufunc 'log10'> ACTUAL: -0.3010299956639812 DESIRED: 0.0 scipy: ====================================================================== ERROR: test_nonsymmetric_modes (test_arpack.TestEigenNonSymmetric) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Users/mabshoff/64bit_numpy_scipy/lib/python2.5/site-packages/scipy/sparse/linalg/eigen/arpack/tests/test_arpack.py", line 204, in test_nonsymmetric_modes self.eval_evec(m,typ,k,which) File "/Users/mabshoff/64bit_numpy_scipy/lib/python2.5/site-packages/scipy/sparse/linalg/eigen/arpack/tests/test_arpack.py", line 186, in eval_evec eval,evec=eigen(a,k,which=which,**kwds) File "/Users/mabshoff/64bit_numpy_scipy/lib/python2.5/site-packages/scipy/sparse/linalg/eigen/arpack/arpack.py", line 220, in eigen raise RuntimeError("Error info=%d in arpack"%info) RuntimeError: Error info=-8 in arpack ====================================================================== ERROR: test_starting_vector (test_arpack.TestEigenNonSymmetric) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Users/mabshoff/64bit_numpy_scipy/lib/python2.5/site-packages/scipy/sparse/linalg/eigen/arpack/tests/test_arpack.py", line 214, in test_starting_vector self.eval_evec(self.symmetric[0],typ,k,which='LM',v0=v0) File "/Users/mabshoff/64bit_numpy_scipy/lib/python2.5/site-packages/scipy/sparse/linalg/eigen/arpack/tests/test_arpack.py", line 186, in eval_evec eval,evec=eigen(a,k,which=which,**kwds) File "/Users/mabshoff/64bit_numpy_scipy/lib/python2.5/site-packages/scipy/sparse/linalg/eigen/arpack/arpack.py", line 220, in eigen raise RuntimeError("Error info=%d in arpack"%info) RuntimeError: Error info=-8 in arpack ====================================================================== FAIL: test_implicit (test_odr.TestODR) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Users/mabshoff/64bit_numpy_scipy/lib/python2.5/site-packages/scipy/odr/tests/test_odr.py", line 118, in test_implicit 2.0778813389755596e-03]]), File "/Users/mabshoff/64bit_numpy_scipy/lib/python2.5/site-packages/numpy/testing/utils.py", line 304, in assert_array_almost_equal header='Arrays are not almost equal') File "/Users/mabshoff/64bit_numpy_scipy/lib/python2.5/site-packages/numpy/testing/utils.py", line 289, in assert_array_compare assert cond, msg AssertionError: Arrays are not almost equal (mismatch 4.0%) x: array([[ 2.10892628e+00, -1.94376722e+00, 7.02635228e-02, -4.71752493e-02, 5.25155371e-02], [ -1.94376722e+00, 2.04814914e+00, -6.16004809e-02,... y: array([[ 2.10892746e+00, -1.94376864e+00, 7.02635509e-02, -4.71752674e-02, 5.25155759e-02], [ -1.94376864e+00, 2.04815092e+00, -6.16005159e-02,...
2008/8/21 Michael Abshoff <michael.abshoff@googlemail.com>:
William Stein is at Scipy 08 and there seems to be at least some interest in 64 bit OSX binaries. If anyone wants one let me know and I can put one together once Scipy 0.7 and Numpy 1.1.2 are out.
There is still interest in having a 64-bit OSX buildbot, which would allow us to support this platform. Would you still be able to host it, as per our previous discussion? Regards Stéfan
Stéfan van der Walt wrote:
2008/8/21 Michael Abshoff <michael.abshoff@googlemail.com>:
William Stein is at Scipy 08 and there seems to be at least some interest in 64 bit OSX binaries. If anyone wants one let me know and I can put one together once Scipy 0.7 and Numpy 1.1.2 are out.
There is still interest in having a 64-bit OSX buildbot, which would allow us to support this platform. Would you still be able to host it, as per our previous discussion?
Yep, I really got distracted the last months doing other things, but now the numpy and scipy upgrades in Sage are high priority since it fixes a number of build issues and segfaults on Solaris and OSX 64 bit :)
Regards Stéfan
Cheers, Michael
_______________________________________________ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
2008/8/21 Michael Abshoff <michael.abshoff@googlemail.com>:
Stéfan van der Walt wrote:
2008/8/21 Michael Abshoff <michael.abshoff@googlemail.com>:
William Stein is at Scipy 08 and there seems to be at least some interest in 64 bit OSX binaries. If anyone wants one let me know and I can put one together once Scipy 0.7 and Numpy 1.1.2 are out.
There is still interest in having a 64-bit OSX buildbot, which would allow us to support this platform. Would you still be able to host it, as per our previous discussion?
Yep, I really got distracted the last months doing other things, but now the numpy and scipy upgrades in Sage are high priority since it fixes a number of build issues and segfaults on Solaris and OSX 64 bit :)
Would you like to host one or two buildbots? I currently have "Solaris 10 SPARC" reserved for you. Would you like another one for MacOSX 64? Cheers Stéfan
I've been experimenting with both a non-framework, non-universal 64-bit build and a 4-way universal build of the python (2.6) trunk with numpy 1.1.1. The non-framework 64 build appears to give me exactly the same results from numpy.test() as the standard 32-bit version (as well as allowing large arrays like numpy.zeros((1000,1000,1000),'d') ), which is <unittest._TextTestResult run=1300 errors=0 failures=0> Our numerical models also seem to run fine with it using 8-16G. The 4-way universal python gives the same results in 32-bit but when running in 64-bit I get an error in the tests below, which I haven't had time to look at. It also gives the error
a = numpy.zeros((1000,1000,1000),'d') Traceback (most recent call last): File "<stdin>", line 1, in <module> ValueError: dimensions too large.
Anyway, this all goes to say I'm interested in making sure numpy is dependable in 64-bit on OSX 10.5 and preferably for 4-way universal builds, but I haven't seen test failures like yours, at least not recently. In my experience so far on the Mac it's still important to make sure Xcode and the OS are up to date (or to use a completely segregated set of consistently build gnu compilers like you did) and to make sure the fink and mac ports stuff is not in your path before building python and numpy. Otherwise configure and distutils end up making bad choices that I don't have the skill to root out. I've had various versions of non-framework 64-bit builds of python running with numpy and Numeric for a while now (a year maybe). The framework and universal builds of python have only worked recently, and I'm still not confident in them. I don't know what's involved in the buildbots. If it's simple, doesn't take much time to set up, and works behind a tight firewall I might be able to set some up. Chris Numpy is installed in /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-package s/numpy Numpy version 1.1.1 Python version 2.6b2+ (trunk:65908, Aug 20 2008, 08:47:16) [GCC 4.0.1 (Apple Inc. build 5484)] Found 18/18 tests for numpy.core.tests.test_defmatrix Found 3/3 tests for numpy.core.tests.test_errstate Found 3/3 tests for numpy.core.tests.test_memmap Found 290/290 tests for numpy.core.tests.test_multiarray Found 72/72 tests for numpy.core.tests.test_numeric Found 36/36 tests for numpy.core.tests.test_numerictypes Found 12/12 tests for numpy.core.tests.test_records Found 145/145 tests for numpy.core.tests.test_regression Found 7/7 tests for numpy.core.tests.test_scalarmath Found 2/2 tests for numpy.core.tests.test_ufunc Found 16/16 tests for numpy.core.tests.test_umath Found 63/63 tests for numpy.core.tests.test_unicode Found 4/4 tests for numpy.distutils.tests.test_fcompiler_gnu Found 5/5 tests for numpy.distutils.tests.test_misc_util Found 2/2 tests for numpy.fft.tests.test_fftpack Found 3/3 tests for numpy.fft.tests.test_helper Found 25/25 tests for numpy.lib.tests.test__datasource Found 10/10 tests for numpy.lib.tests.test_arraysetops Found 1/1 tests for numpy.lib.tests.test_financial Found 55/55 tests for numpy.lib.tests.test_function_base Found 5/5 tests for numpy.lib.tests.test_getlimits Found 6/6 tests for numpy.lib.tests.test_index_tricks /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-package s/numpy/lib/tests/test_io.py:11: SyntaxWarning: assertion is always true, perhaps remove parentheses? assert(c.readlines(), Found 15/15 tests for numpy.lib.tests.test_io Found 1/1 tests for numpy.lib.tests.test_machar Found 4/4 tests for numpy.lib.tests.test_polynomial Found 1/1 tests for numpy.lib.tests.test_regression Found 49/49 tests for numpy.lib.tests.test_shape_base Found 15/15 tests for numpy.lib.tests.test_twodim_base Found 43/43 tests for numpy.lib.tests.test_type_check Found 1/1 tests for numpy.lib.tests.test_ufunclike Found 89/89 tests for numpy.linalg.tests.test_linalg Found 3/3 tests for numpy.linalg.tests.test_regression Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packag es/numpy/__init__.py", line 126, in test return NumpyTest().test(*args, **kw) File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packag es/numpy/testing/numpytest.py", line 579, in test level, verbosity) File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packag es/numpy/testing/numpytest.py", line 524, in _test_suite_from_all_tests verbosity=verbosity) File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packag es/numpy/testing/numpytest.py", line 417, in _get_suite_list suite = obj(mthname) File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packag es/numpy/ma/tests/test_core.py", line 36, in __init__ self.setUp() File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packag es/numpy/ma/tests/test_core.py", line 49, in setUp xf = numpy.where(m1, 1.e+20, x) ValueError: invalid entry in choice array On 8/21/08 4:15 PM, "Michael Abshoff" <michael.abshoff@googlemail.com> wrote:
Hi,
I have been playing with 64 bit numpy/scipy on OSX 10.5 Intel again. I thought everything worked after the last time we discussed this, but I noticed that I had Scipy import failures in for example optimize since the gfortran I used creates 32 bit code per default.
So I build a gcc 4.2.4 on OSX, then build python in 64 bit mode (fixing configure.in since Python assumes flags only available with the Apple gcc on Darwin), added a wrapper around gfortran that injects a "-m64" compile flag. Then I checkout out and build numpy (r5671), scipy (r4661) and the current nose release and ran the tests. I had some more failures than I expected (details below).
Any thoughts on the failures? I am about to build ATLAS and a couple other dependencies on that box.
William Stein is at Scipy 08 and there seems to be at least some interest in 64 bit OSX binaries. If anyone wants one let me know and I can put one together once Scipy 0.7 and Numpy 1.1.2 are out.
Cheers,
Michael
numpy failed tests:
**************************************************************** File "/Users/mabshoff/64bit_numpy_scipy/lib/python2.5/site-packages/numpy/lib/tests /test_polynomial.py", line 29, in test_polynomial Failed example: p / q Expected: (poly1d([ 0.33333333]), poly1d([ 1.33333333, 2.66666667])) Got: (poly1d([ 0.333]), poly1d([ 1.333, 2.667])) ********************************************************************** File "/Users/mabshoff/64bit_numpy_scipy/lib/python2.5/site-packages/numpy/lib/tests /test_polynomial.py", line 51, in test_polynomial Failed example: p.integ() Expected: poly1d([ 0.33333333, 1. , 3. , 0. ]) Got: poly1d([ 0.333, 1. , 3. , 0. ]) **********************************************************************File "/Users/mabshoff/64bit_numpy_scipy/lib/python2.5/site-packages/numpy/lib/tests /test_polynomial.py", line 53, in test_polynomialFailed example: p.integ(1) Expected: poly1d([ 0.33333333, 1. , 3. , 0. ]) Got: poly1d([ 0.333, 1. , 3. , 0. ]) ********************************************************************** File "/Users/mabshoff/64bit_numpy_scipy/lib/python2.5/site-packages/numpy/lib/tests /test_polynomial.py", line 55, in test_polynomial Failed example: p.integ(5) Expected: poly1d([ 0.00039683, 0.00277778, 0.025 , 0. , 0. , 0. , 0. , 0. ]) Got: poly1d([ 0. , 0.003, 0.025, 0. , 0. , 0. , 0. , 0. ])
******************************** File "/Users/mabshoff/64bit_numpy_scipy/lib/python2.5/site-packages/numpy/lib/tests /test_ufunclike.py", line 48, in test_ufunclike Failed example: U.log2(a) Expected: array([ 2.169925 , 1.20163386, 2.70043972]) Got: array([ 2.17 , 1.202, 2.7 ]) ********************************************************************** File "/Users/mabshoff/64bit_numpy_scipy/lib/python2.5/site-packages/numpy/lib/tests /test_ufunclike.py", line 53, in test_ufunclike Failed example: U.log2(a, y) Expected: array([ 2.169925 , 1.20163386, 2.70043972]) Got: array([ 2.17 , 1.202, 2.7 ]) ********************************************************************** File "/Users/mabshoff/64bit_numpy_scipy/lib/python2.5/site-packages/numpy/lib/tests /test_ufunclike.py", line 55, in test_ufunclike Failed example: y Expected: array([ 2.169925 , 1.20163386, 2.70043972]) Got: array([ 2.17 , 1.202, 2.7 ]) ====================================================================== FAIL: test_umath.TestComplexFunctions.test_it ---------------------------------------------------------------------- Traceback (most recent call last): File "/Users/mabshoff/64bit_numpy_scipy/lib/python2.5/site-packages/nose/case.py", line 182, in runTest self.test(*self.arg) File "/Users/mabshoff/64bit_numpy_scipy/lib/python2.5/site-packages/numpy/core/test s/test_umath.py", line 196, in test_it assert_almost_equal(fz.real, fr, err_msg='real part %s'%f) File "/Users/mabshoff/64bit_numpy_scipy/lib/python2.5/site-packages/numpy/testing/u tils.py", line 207, in assert_almost_equal assert round(abs(desired - actual),decimal) == 0, msg AssertionError: Items are not equal: real part <ufunc 'log10'> ACTUAL: -0.3010299956639812 DESIRED: 0.0
scipy:
====================================================================== ERROR: test_nonsymmetric_modes (test_arpack.TestEigenNonSymmetric) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Users/mabshoff/64bit_numpy_scipy/lib/python2.5/site-packages/scipy/sparse/li nalg/eigen/arpack/tests/test_arpack.py", line 204, in test_nonsymmetric_modes self.eval_evec(m,typ,k,which) File "/Users/mabshoff/64bit_numpy_scipy/lib/python2.5/site-packages/scipy/sparse/li nalg/eigen/arpack/tests/test_arpack.py", line 186, in eval_evec eval,evec=eigen(a,k,which=which,**kwds) File "/Users/mabshoff/64bit_numpy_scipy/lib/python2.5/site-packages/scipy/sparse/li nalg/eigen/arpack/arpack.py", line 220, in eigen raise RuntimeError("Error info=%d in arpack"%info) RuntimeError: Error info=-8 in arpack
====================================================================== ERROR: test_starting_vector (test_arpack.TestEigenNonSymmetric) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Users/mabshoff/64bit_numpy_scipy/lib/python2.5/site-packages/scipy/sparse/li nalg/eigen/arpack/tests/test_arpack.py", line 214, in test_starting_vector self.eval_evec(self.symmetric[0],typ,k,which='LM',v0=v0) File "/Users/mabshoff/64bit_numpy_scipy/lib/python2.5/site-packages/scipy/sparse/li nalg/eigen/arpack/tests/test_arpack.py", line 186, in eval_evec eval,evec=eigen(a,k,which=which,**kwds) File "/Users/mabshoff/64bit_numpy_scipy/lib/python2.5/site-packages/scipy/sparse/li nalg/eigen/arpack/arpack.py", line 220, in eigen raise RuntimeError("Error info=%d in arpack"%info) RuntimeError: Error info=-8 in arpack ====================================================================== FAIL: test_implicit (test_odr.TestODR) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Users/mabshoff/64bit_numpy_scipy/lib/python2.5/site-packages/scipy/odr/tests /test_odr.py", line 118, in test_implicit 2.0778813389755596e-03]]), File "/Users/mabshoff/64bit_numpy_scipy/lib/python2.5/site-packages/numpy/testing/u tils.py", line 304, in assert_array_almost_equal header='Arrays are not almost equal') File "/Users/mabshoff/64bit_numpy_scipy/lib/python2.5/site-packages/numpy/testing/u tils.py", line 289, in assert_array_compare assert cond, msg AssertionError: Arrays are not almost equal
(mismatch 4.0%) x: array([[ 2.10892628e+00, -1.94376722e+00, 7.02635228e-02, -4.71752493e-02, 5.25155371e-02], [ -1.94376722e+00, 2.04814914e+00, -6.16004809e-02,... y: array([[ 2.10892746e+00, -1.94376864e+00, 7.02635509e-02, -4.71752674e-02, 5.25155759e-02], [ -1.94376864e+00, 2.04815092e+00, -6.16005159e-02,... _______________________________________________ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
On Fri, Aug 22, 2008 at 07:00, Chris Kees <christopher.e.kees@usace.army.mil> wrote:
I've been experimenting with both a non-framework, non-universal 64-bit build and a 4-way universal build of the python (2.6) trunk with numpy 1.1.1. The non-framework 64 build appears to give me exactly the same results from numpy.test() as the standard 32-bit version (as well as allowing large arrays like numpy.zeros((1000,1000,1000),'d') ), which is
<unittest._TextTestResult run=1300 errors=0 failures=0>
Our numerical models also seem to run fine with it using 8-16G. The 4-way universal python gives the same results in 32-bit but when running in 64-bit I get an error in the tests below, which I haven't had time to look at. It also gives the error
a = numpy.zeros((1000,1000,1000),'d') Traceback (most recent call last): File "<stdin>", line 1, in <module> ValueError: dimensions too large.
Much of our configuration occurs by compiling small C programs and executing them. Probably, one of these got run in 32-bit mode, and that fooled the numpy build into thinking that it was for 32-bit only. Unfortunately, what you are trying to do is tantamount to cross-compiling, and neither distutils nor the additions we have built on top of it work very well with cross-compiling. It's possible that we could special case the configuration on OS X, though. Instead of trusting the results of the executables, we can probably recognize each of the 4 OS X variants through #ifdefs and reset the discovered results. This isn't easily extended to all platforms (which is why we went with the executable approach in the first place), but OS X on both 32-bit and 64-bit will be increasingly common but still manageable. I would welcome contributions in this area. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco
Robert Kern wrote:
On Fri, Aug 22, 2008 at 07:00, Chris Kees <christopher.e.kees@usace.army.mil> wrote:
Hi,
I've been experimenting with both a non-framework, non-universal 64-bit build and a 4-way universal build of the python (2.6) trunk with numpy 1.1.1. The non-framework 64 build appears to give me exactly the same results from numpy.test() as the standard 32-bit version (as well as allowing large arrays like numpy.zeros((1000,1000,1000),'d') ), which is
<unittest._TextTestResult run=1300 errors=0 failures=0>
I used a gcc 4.2.4 build from sources and last time I used XCode (and in addition a gfortran build from 4.2.3 sources) most things went fine. Note that I am using Python 2.5.2 and that I had to make configure.in not add some default BASEFLAGS since those added flags only exist for the Apple version of gcc. This was also numpy and scipy svn tip :)
Our numerical models also seem to run fine with it using 8-16G. The 4-way universal python gives the same results in 32-bit but when running in 64-bit I get an error in the tests below, which I haven't had time to look at. It also gives the error
a = numpy.zeros((1000,1000,1000),'d') Traceback (most recent call last): File "<stdin>", line 1, in <module> ValueError: dimensions too large.
Much of our configuration occurs by compiling small C programs and executing them. Probably, one of these got run in 32-bit mode, and that fooled the numpy build into thinking that it was for 32-bit only.
Yeah, building universal binaries is still fraught with issues since much code out there uses values derived at configure time for endianess and other values. IIRC Apple patches Python to make some of those constants functions, but that recollection might be wrong in case of Python. I usually use lipo to make universal binaries theses days to get around that limitation, but that means four compilations instead of two. My main goal in all of this is a 64 bit Sage on OSX (which I am reasonable close to fully working), but due to above mentioned problems for example with gmp it seems unlikely that I can produce a universal version directly and lipo is a way out of this.
Unfortunately, what you are trying to do is tantamount to cross-compiling, and neither distutils nor the additions we have built on top of it work very well with cross-compiling. It's possible that we could special case the configuration on OS X, though. Instead of trusting the results of the executables, we can probably recognize each of the 4 OS X variants through #ifdefs and reset the discovered results. This isn't easily extended to all platforms (which is why we went with the executable approach in the first place), but OS X on both 32-bit and 64-bit will be increasingly common but still manageable. I would welcome contributions in this area.
I am actually fairly confident that 64 bit Intel will dominate the Apple userbase in the short term. Every laptop and workstation on sale by Apple now and in the last two years or so has been 64 bit capable and for console applications OSX 10.4 or higher will do. So I see little benefit from doing 32 bit on OSX except for legacy support :). I know that Apple hardware tends to stick around longer, so even Sage will support 32 bit OSX for a while. Cheers, Michael
On 8/22/08 11:34 AM, "Michael Abshoff" <michael.abshoff@googlemail.com> wrote:
Robert Kern wrote:
On Fri, Aug 22, 2008 at 07:00, Chris Kees <christopher.e.kees@usace.army.mil> wrote:
Hi,
I've been experimenting with both a non-framework, non-universal 64-bit build and a 4-way universal build of the python (2.6) trunk with numpy 1.1.1. The non-framework 64 build appears to give me exactly the same results from numpy.test() as the standard 32-bit version (as well as allowing large arrays like numpy.zeros((1000,1000,1000),'d') ), which is
<unittest._TextTestResult run=1300 errors=0 failures=0>
I used a gcc 4.2.4 build from sources and last time I used XCode (and in addition a gfortran build from 4.2.3 sources) most things went fine. Note that I am using Python 2.5.2 and that I had to make configure.in not add some default BASEFLAGS since those added flags only exist for the Apple version of gcc. This was also numpy and scipy svn tip :)
Our numerical models also seem to run fine with it using 8-16G. The 4-way universal python gives the same results in 32-bit but when running in 64-bit I get an error in the tests below, which I haven't had time to look at. It also gives the error
a = numpy.zeros((1000,1000,1000),'d') Traceback (most recent call last): File "<stdin>", line 1, in <module> ValueError: dimensions too large.
Much of our configuration occurs by compiling small C programs and executing them. Probably, one of these got run in 32-bit mode, and that fooled the numpy build into thinking that it was for 32-bit only.
Yeah, building universal binaries is still fraught with issues since much code out there uses values derived at configure time for endianess and other values. IIRC Apple patches Python to make some of those constants functions, but that recollection might be wrong in case of Python. I usually use lipo to make universal binaries theses days to get around that limitation, but that means four compilations instead of two.
My main goal in all of this is a 64 bit Sage on OSX (which I am reasonable close to fully working), but due to above mentioned problems for example with gmp it seems unlikely that I can produce a universal version directly and lipo is a way out of this.
Unfortunately, what you are trying to do is tantamount to cross-compiling, and neither distutils nor the additions we have built on top of it work very well with cross-compiling. It's possible that we could special case the configuration on OS X, though. Instead of trusting the results of the executables, we can probably recognize each of the 4 OS X variants through #ifdefs and reset the discovered results. This isn't easily extended to all platforms (which is why we went with the executable approach in the first place), but OS X on both 32-bit and 64-bit will be increasingly common but still manageable. I would welcome contributions in this area.
This sounds like a reasonable approach to me. I wouldn't be disappointed if everything but x86_64 went away, but I'm afraid it's not going to happen soon enough to avoid dealing with i386 and ppc64 in some way. It would be convenient for me if I could just distributed a 4-way universal python/numpy along with my modules. Unfortunately I don't know enough about distutils and the numpy additions to be confident that I can make the fixes. Can we can get all the information we need by comparing the 4-way universal build with a pure x86_64 build? -Chris
I am actually fairly confident that 64 bit Intel will dominate the Apple userbase in the short term. Every laptop and workstation on sale by Apple now and in the last two years or so has been 64 bit capable and for console applications OSX 10.4 or higher will do. So I see little benefit from doing 32 bit on OSX except for legacy support :). I know that Apple hardware tends to stick around longer, so even Sage will support 32 bit OSX for a while.
Cheers,
Michael _______________________________________________ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
participants (4)
-
Chris Kees -
Michael Abshoff -
Robert Kern -
Stéfan van der Walt