Switch to using ATLAS for OSX binary wheels
Hi, Summary : I'm planning to upload OSX wheels for numpy and scipy using the ATLAS blas / lapack library instead of the default OSX Accelerate framework. We've run into some trouble with a segfault in recent OSX Accelerate: https://github.com/numpy/numpy/issues/4007 and Accelerate also doesn't play well with multiprocessing. https://github.com/numpy/numpy/issues/4776 Because there's nothing I love more than half-day compilation runs on my laptop, I've built ATLAS binaries with gcc 4.8, and linked numpy and scipy to them to make OSX wheels. These pass all tests in i386 and x86_64 mode, including numpy, scipy, matplotlib, pandas: https://travis-ci.org/matthew-brett/scipy-stack-osx-testing/builds/27442987 The build process needs some automating, but it's recorded here: https://github.com/matthew-brett/numpy-atlas-binaries It's possible to get travis-ci to build these guys from a bare machine and then upload them somewhere, but I haven't tried to do that. Meanwhile Sturla kindly worked up a patch to numpy to work round the Accelerate segfault [1]. I haven't tested that, but given I'd already built the wheels, I prefer the ATLAS builds because they work with multiprocessing. I propose uploading these wheels as the default for numpy and scipy. Does anyone have any objection or comments before I go ahead and do that? Cheers, Matthew [1] http://mail.scipy.org/pipermail/numpy-discussion/2014-June/070333.html
Matthew Brett <matthew.brett@gmail.com> wrote:
Meanwhile Sturla kindly worked up a patch to numpy to work round the Accelerate segfault [1]. I haven't tested that, but given I'd already built the wheels, I prefer the ATLAS builds because they work with multiprocessing.
It is an ugly hack. If it is used it would be better to keep it in a separate branch rather than in NumPy master. I am not able to post a PR because my fork of NumPy is unavailable on GitHub.
I propose uploading these wheels as the default for numpy and scipy. Does anyone have any objection or comments before I go ahead and do that?
ATLAS should be fine. Performance will not be anything like the alternatives (Accelerate and OpenBLAS), but correctness is more important. I am not sure why ATLAS is preferred over OpenBLAS, though, but I don't really care. Perhaps we could ask Intel permission to use MKL in the binary wheels? The Accelerate vs. multiptocessing issue will never be fixed for Python 2.7 and poissibly Python 3.3 (it is fixed in Python 3.4). The error is in multiprocessing, so Apple will not patch Accelerate. New features will not be added to Python 2.7 standard lib, so there is no chance of getting the required fix backported there either. Stalemate. Sturla
On Fri, Jun 13, 2014 at 3:03 PM, Sturla Molden <sturla.molden@gmail.com> wrote:
Perhaps we could ask Intel permission to use MKL in the binary wheels?
We have already asked and obtained that permission, under the condition that we put some attribution to Intel MKL on our website (which we already have at http://scipy.org/scipylib/donations.html). I would not be in favor of distributing only MKL binaries, but distributing those in addition to ATLAS/Accelerate ones would be fine. Ralf
On Fri, Jun 13, 2014 at 8:18 AM, Ralf Gommers <ralf.gommers@gmail.com> wrote:
We have already asked and obtained that permission, under the condition that we put some attribution to Intel MKL on our website (which we already have at http://scipy.org/scipylib/donations.html). I would not be in favor of distributing only MKL binaries, but distributing those in addition to ATLAS/Accelerate ones would be fine.
I'm curious, why not? But in any case, if the numpy project itself is distributing more than one binary type, then we still need to decide what goes on PyPy. I'm confused about the whole pip optional features thing, but could you put up both and have folks access it with, for example: pip install numpy[mkl] ? -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
On Fri, Jun 13, 2014 at 5:53 PM, Chris Barker <chris.barker@noaa.gov> wrote:
On Fri, Jun 13, 2014 at 8:18 AM, Ralf Gommers <ralf.gommers@gmail.com> wrote:
We have already asked and obtained that permission, under the condition that we put some attribution to Intel MKL on our website (which we already have at http://scipy.org/scipylib/donations.html). I would not be in favor of distributing only MKL binaries, but distributing those in addition to ATLAS/Accelerate ones would be fine.
I'm curious, why not?
Because I think that we should provide a set of binaries that people can reproduce without needing to buy/obtain licenses. Good for debugging, rotating release manager duties, etc.
But in any case, if the numpy project itself is distributing more than one binary type, then we still need to decide what goes on PyPy. I'm confused about the whole pip optional features thing, but could you put up both and have folks access it with, for example:
pip install numpy[mkl]
?
I think that's possible. Ralf
-Chris
--
Christopher Barker, Ph.D. Oceanographer
Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception
Chris.Barker@noaa.gov
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
On 13 June 2014 18:00, Ralf Gommers <ralf.gommers@gmail.com> wrote:
pip install numpy[mkl]
?
I think that's possible.
MKL are fairly famous, but perhaps it would be legally safer to use [mkl-nonfree] (or something of the sort) to signal the licence. But maybe I am bikeshedding here.
Chris Barker <chris.barker@noaa.gov> wrote:
I'm curious, why not?
Because an MKL license is required to redistribute MKL. If someone wants to include the binaries in their product they must acquire a license. An MKL-based binary wheel would be for end-users that wants to install and use NumPy. It would not be for those who redistribute NumPy as a component of a larger application. ATLAS and OpenBLAS are free, and Accelerate Framework is a part of Apple's operating systems. Those binaries can be redistributed by third parties without Intel's licensisng restrictions. But if you are an end-user installing NumPy for your own work you'd want the MKL binary wheel instead. Sturla
Ralf Gommers <ralf.gommers@gmail.com> wrote:
We have already asked and obtained that permission, under the condition that we put some attribution to Intel MKL on our website (which we already have at <a href="http://scipy.org/scipylib/donations.html">http://scipy.org/scipylib/donations.html</a>). I would not be in favor of distributing only MKL binaries, but distributing those in addition to ATLAS/Accelerate ones would be fine.
That is great news :) If we can distribute MKL and ATLAS binaries it will make everyone happy. I suppose this applies to Windows as well? Sturla
On Fri, Jun 13, 2014 at 2:07 PM, Matthew Brett <matthew.brett@gmail.com> wrote:
Hi,
Summary : I'm planning to upload OSX wheels for numpy and scipy using the ATLAS blas / lapack library instead of the default OSX Accelerate framework.
We've run into some trouble with a segfault in recent OSX Accelerate:
https://github.com/numpy/numpy/issues/4007
and Accelerate also doesn't play well with multiprocessing.
https://github.com/numpy/numpy/issues/4776
Because there's nothing I love more than half-day compilation runs on my laptop, I've built ATLAS binaries with gcc 4.8, and linked numpy and scipy to them to make OSX wheels. These pass all tests in i386 and x86_64 mode, including numpy, scipy, matplotlib, pandas:
https://travis-ci.org/matthew-brett/scipy-stack-osx-testing/builds/27442987
The build process needs some automating, but it's recorded here:
https://github.com/matthew-brett/numpy-atlas-binaries
It's possible to get travis-ci to build these guys from a bare machine and then upload them somewhere, but I haven't tried to do that.
Meanwhile Sturla kindly worked up a patch to numpy to work round the Accelerate segfault [1]. I haven't tested that, but given I'd already built the wheels, I prefer the ATLAS builds because they work with multiprocessing.
I propose uploading these wheels as the default for numpy and scipy. Does anyone have any objection or comments before I go ahead and do that?
From your README and wscript I don't see what numpy version you're using to compile scipy against. I got the impression that you used 1.8.1, but it should be numpy 1.5.1 for the 2.7 build, and 1.7.1 for 3.x.
I've tried the scipy 0.14.0 python2.7 wheel, but I get import errors (see below). Your wheels should work with all common Python installs (mine is homebrew) right? Ralf $ python2.7 -c "import scipy; scipy.test()" Running unit tests for scipy NumPy version 1.9.0.dev-056ab73 NumPy is installed in /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy SciPy version 0.14.0 SciPy is installed in /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy Python version 2.7.5 (default, Jun 18 2013, 21:21:44) [GCC 4.2.1 Compatible Apple LLVM 4.2 (clang-425.0.28)] nose version 1.3.0 E...............EEEEEE............EEEEEEEEEE ====================================================================== ERROR: Failure: ImportError (dlopen(/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so, 2): Symbol not found: _PyModule_Create2 Referenced from: /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so Expected in: flat namespace in /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose/loader.py", line 413, in loadTestsFromName addr.filename, addr.module) File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose/importer.py", line 47, in importFromPath return self.importFromDir(dir_path, fqname) File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose/importer.py", line 94, in importFromDir mod = load_module(part_fqname, fh, filename, desc) File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/__init__.py", line 27, in <module> from . import vq, hierarchy File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/hierarchy.py", line 175, in <module> from . import _hierarchy_wrap ImportError: dlopen(/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so, 2): Symbol not found: _PyModule_Create2 Referenced from: /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so Expected in: flat namespace in /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so ... <42 more errors> ...
Cheers,
Matthew
[1] http://mail.scipy.org/pipermail/numpy-discussion/2014-June/070333.html _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
On Friday, June 13, 2014, Ralf Gommers <ralf.gommers@gmail.com> wrote:
On Fri, Jun 13, 2014 at 2:07 PM, Matthew Brett <matthew.brett@gmail.com <javascript:_e(%7B%7D,'cvml','matthew.brett@gmail.com');>> wrote:
Hi,
Summary : I'm planning to upload OSX wheels for numpy and scipy using the ATLAS blas / lapack library instead of the default OSX Accelerate framework.
We've run into some trouble with a segfault in recent OSX Accelerate:
https://github.com/numpy/numpy/issues/4007
and Accelerate also doesn't play well with multiprocessing.
https://github.com/numpy/numpy/issues/4776
Because there's nothing I love more than half-day compilation runs on my laptop, I've built ATLAS binaries with gcc 4.8, and linked numpy and scipy to them to make OSX wheels. These pass all tests in i386 and x86_64 mode, including numpy, scipy, matplotlib, pandas:
https://travis-ci.org/matthew-brett/scipy-stack-osx-testing/builds/27442987
The build process needs some automating, but it's recorded here:
https://github.com/matthew-brett/numpy-atlas-binaries
It's possible to get travis-ci to build these guys from a bare machine and then upload them somewhere, but I haven't tried to do that.
Meanwhile Sturla kindly worked up a patch to numpy to work round the Accelerate segfault [1]. I haven't tested that, but given I'd already built the wheels, I prefer the ATLAS builds because they work with multiprocessing.
I propose uploading these wheels as the default for numpy and scipy. Does anyone have any objection or comments before I go ahead and do that?
From your README and wscript I don't see what numpy version you're using to compile scipy against. I got the impression that you used 1.8.1, but it should be numpy 1.5.1 for the 2.7 build, and 1.7.1 for 3.x.
I've tried the scipy 0.14.0 python2.7 wheel, but I get import errors (see below). Your wheels should work with all common Python installs (mine is homebrew) right?
Ralf
$ python2.7 -c "import scipy; scipy.test()" Running unit tests for scipy NumPy version 1.9.0.dev-056ab73 NumPy is installed in /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy SciPy version 0.14.0 SciPy is installed in /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy Python version 2.7.5 (default, Jun 18 2013, 21:21:44) [GCC 4.2.1 Compatible Apple LLVM 4.2 (clang-425.0.28)] nose version 1.3.0 E...............EEEEEE............EEEEEEEEEE ====================================================================== ERROR: Failure: ImportError (dlopen(/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so, 2): Symbol not found: _PyModule_Create2 Referenced from: /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so Expected in: flat namespace in /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose/loader.py", line 413, in loadTestsFromName addr.filename, addr.module) File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose/importer.py", line 47, in importFromPath return self.importFromDir(dir_path, fqname) File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose/importer.py", line 94, in importFromDir mod = load_module(part_fqname, fh, filename, desc) File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/__init__.py", line 27, in <module> from . import vq, hierarchy File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/hierarchy.py", line 175, in <module> from . import _hierarchy_wrap ImportError: dlopen(/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so, 2): Symbol not found: _PyModule_Create2 Referenced from: /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so Expected in: flat namespace in /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so
... <42 more errors>
That is strange homebrew is one of tests in the grid and the installation path looks strange. Can you try downloading the wheel from the url and installing from the local file? Cheers, Matthew
On Sat, Jun 14, 2014 at 11:56 AM, Matthew Brett <matthew.brett@gmail.com> wrote:
On Friday, June 13, 2014, Ralf Gommers <ralf.gommers@gmail.com> wrote:
On Fri, Jun 13, 2014 at 2:07 PM, Matthew Brett <matthew.brett@gmail.com> wrote:
Hi,
Summary : I'm planning to upload OSX wheels for numpy and scipy using the ATLAS blas / lapack library instead of the default OSX Accelerate framework.
We've run into some trouble with a segfault in recent OSX Accelerate:
https://github.com/numpy/numpy/issues/4007
and Accelerate also doesn't play well with multiprocessing.
https://github.com/numpy/numpy/issues/4776
Because there's nothing I love more than half-day compilation runs on my laptop, I've built ATLAS binaries with gcc 4.8, and linked numpy and scipy to them to make OSX wheels. These pass all tests in i386 and x86_64 mode, including numpy, scipy, matplotlib, pandas:
https://travis-ci.org/matthew-brett/scipy-stack-osx-testing/builds/27442987
The build process needs some automating, but it's recorded here:
https://github.com/matthew-brett/numpy-atlas-binaries
It's possible to get travis-ci to build these guys from a bare machine and then upload them somewhere, but I haven't tried to do that.
Meanwhile Sturla kindly worked up a patch to numpy to work round the Accelerate segfault [1]. I haven't tested that, but given I'd already built the wheels, I prefer the ATLAS builds because they work with multiprocessing.
I propose uploading these wheels as the default for numpy and scipy. Does anyone have any objection or comments before I go ahead and do that?
From your README and wscript I don't see what numpy version you're using to compile scipy against. I got the impression that you used 1.8.1, but it should be numpy 1.5.1 for the 2.7 build, and 1.7.1 for 3.x.
I've tried the scipy 0.14.0 python2.7 wheel, but I get import errors (see below). Your wheels should work with all common Python installs (mine is homebrew) right?
Ralf
$ python2.7 -c "import scipy; scipy.test()" Running unit tests for scipy NumPy version 1.9.0.dev-056ab73 NumPy is installed in /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy SciPy version 0.14.0 SciPy is installed in /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy Python version 2.7.5 (default, Jun 18 2013, 21:21:44) [GCC 4.2.1 Compatible Apple LLVM 4.2 (clang-425.0.28)] nose version 1.3.0 E...............EEEEEE............EEEEEEEEEE ====================================================================== ERROR: Failure: ImportError (dlopen(/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so, 2): Symbol not found: _PyModule_Create2 Referenced from: /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so Expected in: flat namespace in /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose/loader.py", line 413, in loadTestsFromName addr.filename, addr.module) File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose/importer.py", line 47, in importFromPath return self.importFromDir(dir_path, fqname) File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose/importer.py", line 94, in importFromDir mod = load_module(part_fqname, fh, filename, desc) File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/__init__.py", line 27, in <module> from . import vq, hierarchy File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/hierarchy.py", line 175, in <module> from . import _hierarchy_wrap ImportError: dlopen(/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so, 2): Symbol not found: _PyModule_Create2 Referenced from: /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so Expected in: flat namespace in /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so
... <42 more errors>
That is strange homebrew is one of tests in the grid and the installation path looks strange.
Can you try downloading the wheel from the url and installing from the local file?
That's what I did (easier than remembering the magic pip incantation). The install path looks fine to me, maybe homebrew changed it recently? I can try to update my install, will take a few days though. Ralf
Cheers,
Matthew
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Hi, On Sun, Jun 15, 2014 at 10:51 PM, Ralf Gommers <ralf.gommers@gmail.com> wrote:
On Sat, Jun 14, 2014 at 11:56 AM, Matthew Brett <matthew.brett@gmail.com> wrote:
On Friday, June 13, 2014, Ralf Gommers <ralf.gommers@gmail.com> wrote:
On Fri, Jun 13, 2014 at 2:07 PM, Matthew Brett <matthew.brett@gmail.com> wrote:
Hi,
Summary : I'm planning to upload OSX wheels for numpy and scipy using the ATLAS blas / lapack library instead of the default OSX Accelerate framework.
We've run into some trouble with a segfault in recent OSX Accelerate:
https://github.com/numpy/numpy/issues/4007
and Accelerate also doesn't play well with multiprocessing.
https://github.com/numpy/numpy/issues/4776
Because there's nothing I love more than half-day compilation runs on my laptop, I've built ATLAS binaries with gcc 4.8, and linked numpy and scipy to them to make OSX wheels. These pass all tests in i386 and x86_64 mode, including numpy, scipy, matplotlib, pandas:
https://travis-ci.org/matthew-brett/scipy-stack-osx-testing/builds/27442987
The build process needs some automating, but it's recorded here:
https://github.com/matthew-brett/numpy-atlas-binaries
It's possible to get travis-ci to build these guys from a bare machine and then upload them somewhere, but I haven't tried to do that.
Meanwhile Sturla kindly worked up a patch to numpy to work round the Accelerate segfault [1]. I haven't tested that, but given I'd already built the wheels, I prefer the ATLAS builds because they work with multiprocessing.
I propose uploading these wheels as the default for numpy and scipy. Does anyone have any objection or comments before I go ahead and do that?
From your README and wscript I don't see what numpy version you're using to compile scipy against. I got the impression that you used 1.8.1, but it should be numpy 1.5.1 for the 2.7 build, and 1.7.1 for 3.x.
I've tried the scipy 0.14.0 python2.7 wheel, but I get import errors (see below). Your wheels should work with all common Python installs (mine is homebrew) right?
Ralf
$ python2.7 -c "import scipy; scipy.test()" Running unit tests for scipy NumPy version 1.9.0.dev-056ab73 NumPy is installed in /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy SciPy version 0.14.0 SciPy is installed in /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy Python version 2.7.5 (default, Jun 18 2013, 21:21:44) [GCC 4.2.1 Compatible Apple LLVM 4.2 (clang-425.0.28)] nose version 1.3.0 E...............EEEEEE............EEEEEEEEEE
====================================================================== ERROR: Failure: ImportError (dlopen(/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so, 2): Symbol not found: _PyModule_Create2 Referenced from: /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so Expected in: flat namespace in /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so)
---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose/loader.py", line 413, in loadTestsFromName addr.filename, addr.module) File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose/importer.py", line 47, in importFromPath return self.importFromDir(dir_path, fqname) File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose/importer.py", line 94, in importFromDir mod = load_module(part_fqname, fh, filename, desc) File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/__init__.py", line 27, in <module> from . import vq, hierarchy File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/hierarchy.py", line 175, in <module> from . import _hierarchy_wrap ImportError: dlopen(/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so, 2): Symbol not found: _PyModule_Create2 Referenced from: /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so Expected in: flat namespace in /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so
... <42 more errors>
That is strange homebrew is one of tests in the grid and the installation path looks strange.
Can you try downloading the wheel from the url and installing from the local file?
That's what I did (easier than remembering the magic pip incantation). The install path looks fine to me, maybe homebrew changed it recently? I can try to update my install, will take a few days though.
Yes, sorry - that does look like the normal homebrew install path, I didn't realize it had the exotic framework parts to it. I just ran these commands on my machine: SPI=https://nipy.bic.berkeley.edu/scipy_installers BREW_BIN=/usr/local/Cellar/python/2.7.6/bin curl -O $SPI/numpy-1.8.1-cp27-none-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.whl curl -O $SPI/scipy-0.14.0-cp27-none-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.whl $BREW_BIN/pip install numpy-1.8.1-cp27-none-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.whl $BREW_BIN/pip install scipy-0.14.0-cp27-none-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.whl $BREW_BIN/python -c "import scipy; scipy.test()" and the scipy tests passed. I built the scipy wheel against numpy 1.8.1 - but - aren't the numpies binary compatible? What difference would I expect to see if I'd built scipy against numpy 1.5.1 or 1.7 ? Cheers, Matthew
On Mon, Jun 16, 2014 at 7:10 AM, Matthew Brett <matthew.brett@gmail.com> wrote:
Hi,
On Sun, Jun 15, 2014 at 10:51 PM, Ralf Gommers <ralf.gommers@gmail.com> wrote:
On Sat, Jun 14, 2014 at 11:56 AM, Matthew Brett <matthew.brett@gmail.com> wrote:
On Friday, June 13, 2014, Ralf Gommers <ralf.gommers@gmail.com> wrote:
On Fri, Jun 13, 2014 at 2:07 PM, Matthew Brett <matthew.brett@gmail.com> wrote:
Hi,
Summary : I'm planning to upload OSX wheels for numpy and scipy using the ATLAS blas / lapack library instead of the default OSX Accelerate framework.
We've run into some trouble with a segfault in recent OSX Accelerate:
https://github.com/numpy/numpy/issues/4007
and Accelerate also doesn't play well with multiprocessing.
https://github.com/numpy/numpy/issues/4776
Because there's nothing I love more than half-day compilation runs on my laptop, I've built ATLAS binaries with gcc 4.8, and linked numpy and scipy to them to make OSX wheels. These pass all tests in i386 and x86_64 mode, including numpy, scipy, matplotlib, pandas:
https://travis-ci.org/matthew-brett/scipy-stack-osx-testing/builds/27442987
The build process needs some automating, but it's recorded here:
https://github.com/matthew-brett/numpy-atlas-binaries
It's possible to get travis-ci to build these guys from a bare machine and then upload them somewhere, but I haven't tried to do that.
Meanwhile Sturla kindly worked up a patch to numpy to work round the Accelerate segfault [1]. I haven't tested that, but given I'd already built the wheels, I prefer the ATLAS builds because they work with multiprocessing.
I propose uploading these wheels as the default for numpy and scipy. Does anyone have any objection or comments before I go ahead and do that?
From your README and wscript I don't see what numpy version you're using to compile scipy against. I got the impression that you used 1.8.1, but it should be numpy 1.5.1 for the 2.7 build, and 1.7.1 for 3.x.
I've tried the scipy 0.14.0 python2.7 wheel, but I get import errors (see below). Your wheels should work with all common Python installs (mine is homebrew) right?
Ralf
$ python2.7 -c "import scipy; scipy.test()" Running unit tests for scipy NumPy version 1.9.0.dev-056ab73 NumPy is installed in /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy SciPy version 0.14.0 SciPy is installed in /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy Python version 2.7.5 (default, Jun 18 2013, 21:21:44) [GCC 4.2.1 Compatible Apple LLVM 4.2 (clang-425.0.28)] nose version 1.3.0 E...............EEEEEE............EEEEEEEEEE
====================================================================== ERROR: Failure: ImportError (dlopen(/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so, 2): Symbol not found: _PyModule_Create2 Referenced from: /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so Expected in: flat namespace in /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so)
---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose/loader.py", line 413, in loadTestsFromName addr.filename, addr.module) File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose/importer.py", line 47, in importFromPath return self.importFromDir(dir_path, fqname) File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose/importer.py", line 94, in importFromDir mod = load_module(part_fqname, fh, filename, desc) File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/__init__.py", line 27, in <module> from . import vq, hierarchy File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/hierarchy.py", line 175, in <module> from . import _hierarchy_wrap ImportError: dlopen(/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so, 2): Symbol not found: _PyModule_Create2 Referenced from: /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so Expected in: flat namespace in /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so
... <42 more errors>
That is strange homebrew is one of tests in the grid and the installation path looks strange.
Can you try downloading the wheel from the url and installing from the local file?
That's what I did (easier than remembering the magic pip incantation). The install path looks fine to me, maybe homebrew changed it recently? I can try to update my install, will take a few days though.
Yes, sorry - that does look like the normal homebrew install path, I didn't realize it had the exotic framework parts to it.
I just ran these commands on my machine:
SPI=https://nipy.bic.berkeley.edu/scipy_installers BREW_BIN=/usr/local/Cellar/python/2.7.6/bin curl -O $SPI/numpy-1.8.1-cp27-none-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.whl curl -O $SPI/scipy-0.14.0-cp27-none-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.whl $BREW_BIN/pip install numpy-1.8.1-cp27-none-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.whl $BREW_BIN/pip install scipy-0.14.0-cp27-none-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.whl $BREW_BIN/python -c "import scipy; scipy.test()"
and the scipy tests passed.
I built the scipy wheel against numpy 1.8.1 - but - aren't the numpies binary compatible? What difference would I expect to see if I'd built scipy against numpy 1.5.1 or 1.7 ?
I summarized here what David explained a while ago about the difference between forward and backwards binary compatibility http://stackoverflow.com/questions/17709641/valueerror-numpy-dtype-has-the-w... Josef
Cheers,
Matthew _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Hi, On Mon, Jun 16, 2014 at 12:51 PM, <josef.pktd@gmail.com> wrote:
On Mon, Jun 16, 2014 at 7:10 AM, Matthew Brett <matthew.brett@gmail.com> wrote:
Hi,
On Sun, Jun 15, 2014 at 10:51 PM, Ralf Gommers <ralf.gommers@gmail.com> wrote:
On Sat, Jun 14, 2014 at 11:56 AM, Matthew Brett <matthew.brett@gmail.com> wrote:
On Friday, June 13, 2014, Ralf Gommers <ralf.gommers@gmail.com> wrote:
On Fri, Jun 13, 2014 at 2:07 PM, Matthew Brett <matthew.brett@gmail.com> wrote:
Hi,
Summary : I'm planning to upload OSX wheels for numpy and scipy using the ATLAS blas / lapack library instead of the default OSX Accelerate framework.
We've run into some trouble with a segfault in recent OSX Accelerate:
https://github.com/numpy/numpy/issues/4007
and Accelerate also doesn't play well with multiprocessing.
https://github.com/numpy/numpy/issues/4776
Because there's nothing I love more than half-day compilation runs on my laptop, I've built ATLAS binaries with gcc 4.8, and linked numpy and scipy to them to make OSX wheels. These pass all tests in i386 and x86_64 mode, including numpy, scipy, matplotlib, pandas:
https://travis-ci.org/matthew-brett/scipy-stack-osx-testing/builds/27442987
The build process needs some automating, but it's recorded here:
https://github.com/matthew-brett/numpy-atlas-binaries
It's possible to get travis-ci to build these guys from a bare machine and then upload them somewhere, but I haven't tried to do that.
Meanwhile Sturla kindly worked up a patch to numpy to work round the Accelerate segfault [1]. I haven't tested that, but given I'd already built the wheels, I prefer the ATLAS builds because they work with multiprocessing.
I propose uploading these wheels as the default for numpy and scipy. Does anyone have any objection or comments before I go ahead and do that?
From your README and wscript I don't see what numpy version you're using to compile scipy against. I got the impression that you used 1.8.1, but it should be numpy 1.5.1 for the 2.7 build, and 1.7.1 for 3.x.
I've tried the scipy 0.14.0 python2.7 wheel, but I get import errors (see below). Your wheels should work with all common Python installs (mine is homebrew) right?
Ralf
$ python2.7 -c "import scipy; scipy.test()" Running unit tests for scipy NumPy version 1.9.0.dev-056ab73 NumPy is installed in /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy SciPy version 0.14.0 SciPy is installed in /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy Python version 2.7.5 (default, Jun 18 2013, 21:21:44) [GCC 4.2.1 Compatible Apple LLVM 4.2 (clang-425.0.28)] nose version 1.3.0 E...............EEEEEE............EEEEEEEEEE
====================================================================== ERROR: Failure: ImportError (dlopen(/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so, 2): Symbol not found: _PyModule_Create2 Referenced from: /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so Expected in: flat namespace in /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so)
---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose/loader.py", line 413, in loadTestsFromName addr.filename, addr.module) File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose/importer.py", line 47, in importFromPath return self.importFromDir(dir_path, fqname) File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose/importer.py", line 94, in importFromDir mod = load_module(part_fqname, fh, filename, desc) File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/__init__.py", line 27, in <module> from . import vq, hierarchy File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/hierarchy.py", line 175, in <module> from . import _hierarchy_wrap ImportError: dlopen(/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so, 2): Symbol not found: _PyModule_Create2 Referenced from: /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so Expected in: flat namespace in /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so
... <42 more errors>
That is strange homebrew is one of tests in the grid and the installation path looks strange.
Can you try downloading the wheel from the url and installing from the local file?
That's what I did (easier than remembering the magic pip incantation). The install path looks fine to me, maybe homebrew changed it recently? I can try to update my install, will take a few days though.
Yes, sorry - that does look like the normal homebrew install path, I didn't realize it had the exotic framework parts to it.
I just ran these commands on my machine:
SPI=https://nipy.bic.berkeley.edu/scipy_installers BREW_BIN=/usr/local/Cellar/python/2.7.6/bin curl -O $SPI/numpy-1.8.1-cp27-none-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.whl curl -O $SPI/scipy-0.14.0-cp27-none-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.whl $BREW_BIN/pip install numpy-1.8.1-cp27-none-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.whl $BREW_BIN/pip install scipy-0.14.0-cp27-none-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.whl $BREW_BIN/python -c "import scipy; scipy.test()"
and the scipy tests passed.
I built the scipy wheel against numpy 1.8.1 - but - aren't the numpies binary compatible? What difference would I expect to see if I'd built scipy against numpy 1.5.1 or 1.7 ?
I summarized here what David explained a while ago about the difference between forward and backwards binary compatibility
http://stackoverflow.com/questions/17709641/valueerror-numpy-dtype-has-the-w...
I see - thanks for the summary. I will recompile. Cheers, Matthew
Hi, On Mon, Jun 16, 2014 at 1:00 PM, Matthew Brett <matthew.brett@gmail.com> wrote:
Hi,
On Mon, Jun 16, 2014 at 12:51 PM, <josef.pktd@gmail.com> wrote:
On Mon, Jun 16, 2014 at 7:10 AM, Matthew Brett <matthew.brett@gmail.com> wrote:
Hi,
On Sun, Jun 15, 2014 at 10:51 PM, Ralf Gommers <ralf.gommers@gmail.com> wrote:
On Sat, Jun 14, 2014 at 11:56 AM, Matthew Brett <matthew.brett@gmail.com> wrote:
On Friday, June 13, 2014, Ralf Gommers <ralf.gommers@gmail.com> wrote:
On Fri, Jun 13, 2014 at 2:07 PM, Matthew Brett <matthew.brett@gmail.com> wrote: > > Hi, > > Summary : I'm planning to upload OSX wheels for numpy and scipy using > the ATLAS blas / lapack library instead of the default OSX Accelerate > framework. > > We've run into some trouble with a segfault in recent OSX Accelerate: > > https://github.com/numpy/numpy/issues/4007 > > and Accelerate also doesn't play well with multiprocessing. > > https://github.com/numpy/numpy/issues/4776 > > Because there's nothing I love more than half-day compilation runs on > my laptop, I've built ATLAS binaries with gcc 4.8, and linked numpy > and scipy to them to make OSX wheels. These pass all tests in i386 > and x86_64 mode, including numpy, scipy, matplotlib, pandas: > > > https://travis-ci.org/matthew-brett/scipy-stack-osx-testing/builds/27442987 > > The build process needs some automating, but it's recorded here: > > https://github.com/matthew-brett/numpy-atlas-binaries > > It's possible to get travis-ci to build these guys from a bare machine > and then upload them somewhere, but I haven't tried to do that. > > Meanwhile Sturla kindly worked up a patch to numpy to work round the > Accelerate segfault [1]. I haven't tested that, but given I'd already > built the wheels, I prefer the ATLAS builds because they work with > multiprocessing. > > I propose uploading these wheels as the default for numpy and scipy. > Does anyone have any objection or comments before I go ahead and do > that?
From your README and wscript I don't see what numpy version you're using to compile scipy against. I got the impression that you used 1.8.1, but it should be numpy 1.5.1 for the 2.7 build, and 1.7.1 for 3.x.
I've tried the scipy 0.14.0 python2.7 wheel, but I get import errors (see below). Your wheels should work with all common Python installs (mine is homebrew) right?
Ralf
$ python2.7 -c "import scipy; scipy.test()" Running unit tests for scipy NumPy version 1.9.0.dev-056ab73 NumPy is installed in /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy SciPy version 0.14.0 SciPy is installed in /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy Python version 2.7.5 (default, Jun 18 2013, 21:21:44) [GCC 4.2.1 Compatible Apple LLVM 4.2 (clang-425.0.28)] nose version 1.3.0 E...............EEEEEE............EEEEEEEEEE
====================================================================== ERROR: Failure: ImportError (dlopen(/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so, 2): Symbol not found: _PyModule_Create2 Referenced from: /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so Expected in: flat namespace in /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so)
---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose/loader.py", line 413, in loadTestsFromName addr.filename, addr.module) File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose/importer.py", line 47, in importFromPath return self.importFromDir(dir_path, fqname) File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose/importer.py", line 94, in importFromDir mod = load_module(part_fqname, fh, filename, desc) File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/__init__.py", line 27, in <module> from . import vq, hierarchy File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/hierarchy.py", line 175, in <module> from . import _hierarchy_wrap ImportError: dlopen(/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so, 2): Symbol not found: _PyModule_Create2 Referenced from: /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so Expected in: flat namespace in /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so
... <42 more errors>
That is strange homebrew is one of tests in the grid and the installation path looks strange.
Can you try downloading the wheel from the url and installing from the local file?
That's what I did (easier than remembering the magic pip incantation). The install path looks fine to me, maybe homebrew changed it recently? I can try to update my install, will take a few days though.
Yes, sorry - that does look like the normal homebrew install path, I didn't realize it had the exotic framework parts to it.
I just ran these commands on my machine:
SPI=https://nipy.bic.berkeley.edu/scipy_installers BREW_BIN=/usr/local/Cellar/python/2.7.6/bin curl -O $SPI/numpy-1.8.1-cp27-none-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.whl curl -O $SPI/scipy-0.14.0-cp27-none-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.whl $BREW_BIN/pip install numpy-1.8.1-cp27-none-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.whl $BREW_BIN/pip install scipy-0.14.0-cp27-none-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.whl $BREW_BIN/python -c "import scipy; scipy.test()"
and the scipy tests passed.
I built the scipy wheel against numpy 1.8.1 - but - aren't the numpies binary compatible? What difference would I expect to see if I'd built scipy against numpy 1.5.1 or 1.7 ?
I summarized here what David explained a while ago about the difference between forward and backwards binary compatibility
http://stackoverflow.com/questions/17709641/valueerror-numpy-dtype-has-the-w...
I see - thanks for the summary. I will recompile.
Last call - binaries recompiled, tested, more comments here: https://github.com/numpy/numpy/issues/4007 I'm planning to upload these binaries to pypi later on today, Cheers, Matthew
On Thu, Jun 19, 2014 at 11:39 AM, Matthew Brett <matthew.brett@gmail.com> wrote:
Hi,
On Mon, Jun 16, 2014 at 1:00 PM, Matthew Brett <matthew.brett@gmail.com> wrote:
Hi,
On Mon, Jun 16, 2014 at 12:51 PM, <josef.pktd@gmail.com> wrote:
On Mon, Jun 16, 2014 at 7:10 AM, Matthew Brett <matthew.brett@gmail.com> wrote:
Hi,
On Sun, Jun 15, 2014 at 10:51 PM, Ralf Gommers <ralf.gommers@gmail.com> wrote:
On Sat, Jun 14, 2014 at 11:56 AM, Matthew Brett <matthew.brett@gmail.com> wrote:
On Friday, June 13, 2014, Ralf Gommers <ralf.gommers@gmail.com> wrote: > > > > > On Fri, Jun 13, 2014 at 2:07 PM, Matthew Brett <matthew.brett@gmail.com> > wrote: >> >> Hi, >> >> Summary : I'm planning to upload OSX wheels for numpy and scipy using >> the ATLAS blas / lapack library instead of the default OSX Accelerate >> framework. >> >> We've run into some trouble with a segfault in recent OSX Accelerate: >> >> https://github.com/numpy/numpy/issues/4007 >> >> and Accelerate also doesn't play well with multiprocessing. >> >> https://github.com/numpy/numpy/issues/4776 >> >> Because there's nothing I love more than half-day compilation runs on >> my laptop, I've built ATLAS binaries with gcc 4.8, and linked numpy >> and scipy to them to make OSX wheels. These pass all tests in i386 >> and x86_64 mode, including numpy, scipy, matplotlib, pandas: >> >> >> https://travis-ci.org/matthew-brett/scipy-stack-osx-testing/builds/27442987 >> >> The build process needs some automating, but it's recorded here: >> >> https://github.com/matthew-brett/numpy-atlas-binaries >> >> It's possible to get travis-ci to build these guys from a bare machine >> and then upload them somewhere, but I haven't tried to do that. >> >> Meanwhile Sturla kindly worked up a patch to numpy to work round the >> Accelerate segfault [1]. I haven't tested that, but given I'd already >> built the wheels, I prefer the ATLAS builds because they work with >> multiprocessing. >> >> I propose uploading these wheels as the default for numpy and scipy. >> Does anyone have any objection or comments before I go ahead and do >> that? > > > From your README and wscript I don't see what numpy version you're using > to compile scipy against. I got the impression that you used 1.8.1, but it > should be numpy 1.5.1 for the 2.7 build, and 1.7.1 for 3.x. > > I've tried the scipy 0.14.0 python2.7 wheel, but I get import errors (see > below). Your wheels should work with all common Python installs (mine is > homebrew) right? > > Ralf > > > $ python2.7 -c "import scipy; scipy.test()" > Running unit tests for scipy > NumPy version 1.9.0.dev-056ab73 > NumPy is installed in > /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy > SciPy version 0.14.0 > SciPy is installed in > /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy > Python version 2.7.5 (default, Jun 18 2013, 21:21:44) [GCC 4.2.1 > Compatible Apple LLVM 4.2 (clang-425.0.28)] > nose version 1.3.0 > E...............EEEEEE............EEEEEEEEEE > > ====================================================================== > ERROR: Failure: ImportError > (dlopen(/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so, > 2): Symbol not found: _PyModule_Create2 > Referenced from: > /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so > Expected in: flat namespace > in > /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so) > > ---------------------------------------------------------------------- > Traceback (most recent call last): > File > "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose/loader.py", > line 413, in loadTestsFromName > addr.filename, addr.module) > File > "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose/importer.py", > line 47, in importFromPath > return self.importFromDir(dir_path, fqname) > File > "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose/importer.py", > line 94, in importFromDir > mod = load_module(part_fqname, fh, filename, desc) > File > "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/__init__.py", > line 27, in <module> > from . import vq, hierarchy > File > "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/hierarchy.py", > line 175, in <module> > from . import _hierarchy_wrap > ImportError: > dlopen(/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so, > 2): Symbol not found: _PyModule_Create2 > Referenced from: > /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so > Expected in: flat namespace > in > /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so > > ... > <42 more errors>
That is strange homebrew is one of tests in the grid and the installation path looks strange.
Can you try downloading the wheel from the url and installing from the local file?
That's what I did (easier than remembering the magic pip incantation). The install path looks fine to me, maybe homebrew changed it recently? I can try to update my install, will take a few days though.
Yes, sorry - that does look like the normal homebrew install path, I didn't realize it had the exotic framework parts to it.
I just ran these commands on my machine:
SPI=https://nipy.bic.berkeley.edu/scipy_installers BREW_BIN=/usr/local/Cellar/python/2.7.6/bin curl -O $SPI/numpy-1.8.1-cp27-none-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.whl curl -O $SPI/scipy-0.14.0-cp27-none-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.whl $BREW_BIN/pip install numpy-1.8.1-cp27-none-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.whl $BREW_BIN/pip install scipy-0.14.0-cp27-none-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.whl $BREW_BIN/python -c "import scipy; scipy.test()"
and the scipy tests passed.
I built the scipy wheel against numpy 1.8.1 - but - aren't the numpies binary compatible? What difference would I expect to see if I'd built scipy against numpy 1.5.1 or 1.7 ?
I summarized here what David explained a while ago about the difference between forward and backwards binary compatibility
http://stackoverflow.com/questions/17709641/valueerror-numpy-dtype-has-the-w...
I see - thanks for the summary. I will recompile.
Last call - binaries recompiled, tested, more comments here:
https://github.com/numpy/numpy/issues/4007
I'm planning to upload these binaries to pypi later on today,
Done - please test and let me know of any problems, Cheers, Matthew
Just successfully tested on Python 3.4 from python.org / OSX 10.9 and all sklearn tests pass, including a tests that involves multiprocessing and that used to crash with Accelerate. Thanks very much! -- Olivier
On Thu, Jun 19, 2014 at 6:31 PM, Olivier Grisel <olivier.grisel@ensta.org> wrote:
Just successfully tested on Python 3.4 from python.org / OSX 10.9 and all sklearn tests pass, including a tests that involves multiprocessing and that used to crash with Accelerate.
Thanks very much!
De rien - thanks for your help with the Rackspace account, of which more in another email, Cheers, matthew
On 13.06.2014 14:07, Matthew Brett wrote:
Hi,
Summary : I'm planning to upload OSX wheels for numpy and scipy using the ATLAS blas / lapack library instead of the default OSX Accelerate framework.
hi, thanks for doing this. Have you built a generic atlas binary? atlas tunes it self to the capabilities of the machine its building on. So if the machine used to create be binary is newer than the oldest mac machine we want to support it can produce errors due to the use of illegal instructions (avx/sse4 etc.) So far I know the unknown machine type will produce a somewhat generic binary (but not for ARM).
On Friday, June 13, 2014, Julian Taylor <jtaylor.debian@googlemail.com> wrote:
On 13.06.2014 14:07, Matthew Brett wrote:
Hi,
Summary : I'm planning to upload OSX wheels for numpy and scipy using the ATLAS blas / lapack library instead of the default OSX Accelerate framework.
hi, thanks for doing this.
Have you built a generic atlas binary? atlas tunes it self to the capabilities of the machine its building on. So if the machine used to create be binary is newer than the oldest mac machine we want to support it can produce errors due to the use of illegal instructions (avx/sse4 etc.) So far I know the unknown machine type will produce a somewhat generic binary (but not for ARM). _______________________________________
Sorry am traveling and dont have the reference to hand, but all intel macs have sse2. So I built the atlas binaries with sse2 only... Cheers, Matthew
participants (8)
-
Chris Barker -
Daπid -
josef.pktd＠gmail.com -
Julian Taylor -
Matthew Brett -
Olivier Grisel -
Ralf Gommers -
Sturla Molden