Using OpenBLAS for manylinux wheels
Hi, Olivier Grisel and I are working on building and testing manylinux wheels for numpy and scipy. We first thought that we should use ATLAS BLAS, but Olivier found that my build of these could be very slow [1]. I set up a testing grid [2] which found test errors for numpy and scipy using ATLAS wheels. On the other hand, the same testing grid finds no errors or failures [3] using latest OpenBLAS (0.2.17) and running tests for: numpy scipy scikit-learn numexpr pandas statsmodels This is on the travis-ci ubuntu VMs. Please do test on your own machines with something like this script [4]: source test_manylinux.sh We have worried in the past about the reliability of OpenBLAS, but I find these tests reassuring. Are there any other tests of OpenBLAS that we should run to assure ourselves that it is safe to use? Matthew [1] https://github.com/matthew-brett/manylinux-builds/issues/4#issue-143530908 [2] https://travis-ci.org/matthew-brett/manylinux-testing/builds/118780781 [3] I disabled a few pandas tests which were failing for reasons not related to BLAS. Some of the statsmodels test runs time out. [4] https://gist.github.com/matthew-brett/2fd9d9a29e022c297634
I just tested those new openblas-based wheels on the linux virtualbox setup that I used to report the following segfault back in February: https://mail.scipy.org/pipermail/numpy-discussion/2016-February/074866.html https://mail.scipy.org/pipermail/numpy-discussion/2016-February/074870.html I cannot reproduce this problem anymore with this new version of OpenBLAS. All scikit-learn tests pass (I had originally encountered the segfault when runing the scikit-learn tests). I also ran the numpy and scipy tests successfully on that machine. What I find reassuring is that the upstream OpenBLAS has set up a buildbot based CI to test OpenBLAS on many CPU architectures and is running the scipy test continuously to detect regressions early on: https://github.com/xianyi/OpenBLAS/issues/785 http://build.openblas.net/waterfall https://github.com/xianyi/OpenBLAS-CI/ -- Olivier Grisel
On 03/28/2016 04:33 PM, Matthew Brett wrote:
Please do test on your own machines with something like this script [4]: Matthew,
I ran the tests after installing the wheels on my machine running Ubuntu 14.04. Three numpy tests failed with the GFORTRAN_1.4 error you mentioned in post to the wheel-builds list recently. All other tests passed. I can reproduce these failing tests in a Docker container if it is helpful. # python -c 'import numpy; numpy.test("full")' Running unit tests for numpy NumPy version 1.11.0 NumPy relaxed strides checking option: False NumPy is installed in /usr/local/lib/python2.7/dist-packages/numpy Python version 2.7.6 (default, Jun 22 2015, 17:58:13) [GCC 4.8.2] nose version 1.3.7 ... ====================================================================== ERROR: test_kind.TestKind.test_all ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/nose/case.py", line 381, in setUp try_run(self.inst, ('setup', 'setUp')) File "/usr/local/lib/python2.7/dist-packages/nose/util.py", line 471, in try_run return func() File "/usr/local/lib/python2.7/dist-packages/numpy/f2py/tests/util.py", line 358, in setUp module_name=self.module_name) File "/usr/local/lib/python2.7/dist-packages/numpy/f2py/tests/util.py", line 78, in wrapper memo[key] = func(*a, **kw) File "/usr/local/lib/python2.7/dist-packages/numpy/f2py/tests/util.py", line 149, in build_module __import__(module_name) ImportError: /usr/local/lib/python2.7/dist-packages/numpy/core/../.libs/libgfortran.so.3: version `GFORTRAN_1.4' not found (required by /tmp/tmptYznnz/_test_ext_module_5405.so) ====================================================================== ERROR: test_mixed.TestMixed.test_all ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/nose/case.py", line 381, in setUp try_run(self.inst, ('setup', 'setUp')) File "/usr/local/lib/python2.7/dist-packages/nose/util.py", line 471, in try_run return func() File "/usr/local/lib/python2.7/dist-packages/numpy/f2py/tests/util.py", line 358, in setUp module_name=self.module_name) File "/usr/local/lib/python2.7/dist-packages/numpy/f2py/tests/util.py", line 78, in wrapper memo[key] = func(*a, **kw) File "/usr/local/lib/python2.7/dist-packages/numpy/f2py/tests/util.py", line 149, in build_module __import__(module_name) ImportError: /usr/local/lib/python2.7/dist-packages/numpy/core/../.libs/libgfortran.so.3: version `GFORTRAN_1.4' not found (required by /tmp/tmptYznnz/_test_ext_module_5405.so) ====================================================================== ERROR: test_mixed.TestMixed.test_docstring ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/nose/case.py", line 381, in setUp try_run(self.inst, ('setup', 'setUp')) File "/usr/local/lib/python2.7/dist-packages/nose/util.py", line 471, in try_run return func() File "/usr/local/lib/python2.7/dist-packages/numpy/f2py/tests/util.py", line 358, in setUp module_name=self.module_name) File "/usr/local/lib/python2.7/dist-packages/numpy/f2py/tests/util.py", line 84, in wrapper raise ret ImportError: /usr/local/lib/python2.7/dist-packages/numpy/core/../.libs/libgfortran.so.3: version `GFORTRAN_1.4' not found (required by /tmp/tmptYznnz/_test_ext_module_5405.so) ---------------------------------------------------------------------- Ran 6322 tests in 136.678s FAILED (KNOWNFAIL=6, SKIP=11, errors=3) Cheers, - Jonathan Helmus
The problem with the gfortran failures will be tackled by renaming the vendored libgfortran.so library, see: https://github.com/pypa/auditwheel/issues/24 This is orthogonal to the ATLAS vs OpenBLAS decision though. -- Olivier
On Nix/NixOS we've been using OpenBLAS 0.2.14 for some time now because we had some segmentation faults with 0.2.15 and scipy/scikitlearn. I've tested the packages you listed, and more, with OpenBLAS 0.2.17 and encountered no problems. On Wed, Mar 30, 2016 at 1:32 PM, Olivier Grisel <olivier.grisel@ensta.org> wrote:
The problem with the gfortran failures will be tackled by renaming the vendored libgfortran.so library, see:
https://github.com/pypa/auditwheel/issues/24
This is orthogonal to the ATLAS vs OpenBLAS decision though.
-- Olivier _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
If OpenBLAS is looking like the easiest to support solution, then no objections here. (If 0.2.17 is genuinely working well, then maybe we want to switch to it on Windows too. I know Xianyi disabled some of the problematic kernels for us -- maybe that's enough. Mostly I just don't want to end up in the situation where we're trying to support a bunch of differently broken builds on different platforms.) On Mar 28, 2016 2:34 PM, "Matthew Brett" <matthew.brett@gmail.com> wrote:
Hi,
Olivier Grisel and I are working on building and testing manylinux wheels for numpy and scipy.
We first thought that we should use ATLAS BLAS, but Olivier found that my build of these could be very slow [1]. I set up a testing grid [2] which found test errors for numpy and scipy using ATLAS wheels.
On the other hand, the same testing grid finds no errors or failures [3] using latest OpenBLAS (0.2.17) and running tests for:
numpy scipy scikit-learn numexpr pandas statsmodels
This is on the travis-ci ubuntu VMs.
Please do test on your own machines with something like this script [4]:
source test_manylinux.sh
We have worried in the past about the reliability of OpenBLAS, but I find these tests reassuring.
Are there any other tests of OpenBLAS that we should run to assure ourselves that it is safe to use?
Matthew
[1] https://github.com/matthew-brett/manylinux-builds/issues/4#issue-143530908 [2] https://travis-ci.org/matthew-brett/manylinux-testing/builds/118780781 [3] I disabled a few pandas tests which were failing for reasons not related to BLAS. Some of the statsmodels test runs time out. [4] https://gist.github.com/matthew-brett/2fd9d9a29e022c297634 _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
I would like to see OpenBLAS support for numpy on windows. The latest OpenBLAS windows builds numpy support for are on https://bitbucket.org/carlkl/mingw-w64-for-python/downloads now. Scipy wheels should work regardless if numpy was build with MSVC or with mingwpy. It is only mandantory to agree about the BLAS implemenation used. Carl 2016-03-30 23:30 GMT+02:00 Nathaniel Smith <njs@pobox.com>:
If OpenBLAS is looking like the easiest to support solution, then no objections here. (If 0.2.17 is genuinely working well, then maybe we want to switch to it on Windows too. I know Xianyi disabled some of the problematic kernels for us -- maybe that's enough. Mostly I just don't want to end up in the situation where we're trying to support a bunch of differently broken builds on different platforms.) On Mar 28, 2016 2:34 PM, "Matthew Brett" <matthew.brett@gmail.com> wrote:
Hi,
Olivier Grisel and I are working on building and testing manylinux wheels for numpy and scipy.
We first thought that we should use ATLAS BLAS, but Olivier found that my build of these could be very slow [1]. I set up a testing grid [2] which found test errors for numpy and scipy using ATLAS wheels.
On the other hand, the same testing grid finds no errors or failures [3] using latest OpenBLAS (0.2.17) and running tests for:
numpy scipy scikit-learn numexpr pandas statsmodels
This is on the travis-ci ubuntu VMs.
Please do test on your own machines with something like this script [4]:
source test_manylinux.sh
We have worried in the past about the reliability of OpenBLAS, but I find these tests reassuring.
Are there any other tests of OpenBLAS that we should run to assure ourselves that it is safe to use?
Matthew
[1] https://github.com/matthew-brett/manylinux-builds/issues/4#issue-143530908 [2] https://travis-ci.org/matthew-brett/manylinux-testing/builds/118780781 [3] I disabled a few pandas tests which were failing for reasons not related to BLAS. Some of the statsmodels test runs time out. [4] https://gist.github.com/matthew-brett/2fd9d9a29e022c297634 _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
On Wed, Mar 30, 2016 at 2:39 PM, Carl Kleffner <cmkleffner@gmail.com> wrote:
I would like to see OpenBLAS support for numpy on windows. The latest OpenBLAS windows builds numpy support for are on https://bitbucket.org/carlkl/mingw-w64-for-python/downloads now. Scipy wheels should work regardless if numpy was build with MSVC or with mingwpy. It is only mandantory to agree about the BLAS implemenation used.
How do the tests look for `numpy.test("full")` and `scipy.test("full") with OpenBLAS 0.2.17 on Windows? Sorry if you said before. Thanks, Matthew
On Mon, Mar 28, 2016 at 2:33 PM, Matthew Brett <matthew.brett@gmail.com> wrote:
Hi,
Olivier Grisel and I are working on building and testing manylinux wheels for numpy and scipy.
We first thought that we should use ATLAS BLAS, but Olivier found that my build of these could be very slow [1]. I set up a testing grid [2] which found test errors for numpy and scipy using ATLAS wheels.
On the other hand, the same testing grid finds no errors or failures [3] using latest OpenBLAS (0.2.17) and running tests for:
numpy scipy scikit-learn numexpr pandas statsmodels
This is on the travis-ci ubuntu VMs.
Please do test on your own machines with something like this script [4]:
source test_manylinux.sh
We have worried in the past about the reliability of OpenBLAS, but I find these tests reassuring.
Are there any other tests of OpenBLAS that we should run to assure ourselves that it is safe to use?
Here is an update on progress: We've now done a lot of testing on the Linux OpenBLAS wheels. They pass all tests on Linux, with Intel kernels: https://travis-ci.org/matthew-brett/manylinux-testing/builds/120825485 http://nipy.bic.berkeley.edu/builders/manylinux-2.7-debian/builds/22 http://nipy.bic.berkeley.edu/builders/manylinux-2.7-fedora/builds/10 Xianyi, the maintainer of OpenBLAS, is very helpfully running the OpenBLAS buildbot nightly tests with numpy and scipy: http://build.openblas.net/builders There is still one BLAS-related failure on these tests on AMD chips: https://github.com/xianyi/OpenBLAS-CI/issues/10 I propose to hold off distributing the OpenBLAS wheels until the OpenBLAS tests are clean on the OpenBLAS buildbots - any objections? Cheers, Matthew
Xianyi, the maintainer of OpenBLAS, is very helpfully running the OpenBLAS buildbot nightly tests with numpy and scipy:
http://build.openblas.net/builders
There is still one BLAS-related failure on these tests on AMD chips:
https://github.com/xianyi/OpenBLAS-CI/issues/10
I propose to hold off distributing the OpenBLAS wheels until the OpenBLAS tests are clean on the OpenBLAS buildbots - any objections?
I agree. If someone can understand the fortran code of that scipy test failure to extract a minimalistic reproduction case that only use a few BLAS calls that would help. -- Olivier http://twitter.com/ogrisel - http://github.com/ogrisel
On Apr 5, 2016 10:23 AM, "Matthew Brett" <matthew.brett@gmail.com> wrote:
On Mon, Mar 28, 2016 at 2:33 PM, Matthew Brett <matthew.brett@gmail.com>
wrote:
Hi,
Olivier Grisel and I are working on building and testing manylinux wheels for numpy and scipy.
We first thought that we should use ATLAS BLAS, but Olivier found that my build of these could be very slow [1]. I set up a testing grid [2] which found test errors for numpy and scipy using ATLAS wheels.
On the other hand, the same testing grid finds no errors or failures [3] using latest OpenBLAS (0.2.17) and running tests for:
numpy scipy scikit-learn numexpr pandas statsmodels
This is on the travis-ci ubuntu VMs.
Please do test on your own machines with something like this script [4]:
source test_manylinux.sh
We have worried in the past about the reliability of OpenBLAS, but I find these tests reassuring.
Are there any other tests of OpenBLAS that we should run to assure ourselves that it is safe to use?
Here is an update on progress:
We've now done a lot of testing on the Linux OpenBLAS wheels. They pass all tests on Linux, with Intel kernels:
https://travis-ci.org/matthew-brett/manylinux-testing/builds/120825485 http://nipy.bic.berkeley.edu/builders/manylinux-2.7-debian/builds/22 http://nipy.bic.berkeley.edu/builders/manylinux-2.7-fedora/builds/10
Xianyi, the maintainer of OpenBLAS, is very helpfully running the OpenBLAS buildbot nightly tests with numpy and scipy:
http://build.openblas.net/builders
There is still one BLAS-related failure on these tests on AMD chips:
https://github.com/xianyi/OpenBLAS-CI/issues/10
I propose to hold off distributing the OpenBLAS wheels until the OpenBLAS tests are clean on the OpenBLAS buildbots - any objections?
Alternatively, would it make sense to add a local patch to our openblas builds to blacklist the piledriver kernel and then distribute them now? (I'm not immediately sure what would be involved in doing this but it seems unlikely that it would require anything tricky?) -n
2016-04-05 19:44 GMT+02:00 Nathaniel Smith <njs@pobox.com>:
I propose to hold off distributing the OpenBLAS wheels until the OpenBLAS tests are clean on the OpenBLAS buildbots - any objections?
Alternatively, would it make sense to add a local patch to our openblas builds to blacklist the piledriver kernel and then distribute them now? (I'm not immediately sure what would be involved in doing this but it seems unlikely that it would require anything tricky?)
I tried to force use the NEHALEM or the BARCELONA driver on a PILEDRIVER AMD box and while it fixes the original test failure in isolve it causes this other scipy test to fail: ====================================================================== FAIL: test_nanmedian_all_axis (test_stats.TestNanFunc) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib/python2.7/site-packages/scipy/stats/tests/test_stats.py", line 242, in test_nanmedian_all_axis assert_equal(len(w), 4) File "/usr/local/lib/python2.7/site-packages/numpy/testing/utils.py", line 375, in assert_equal raise AssertionError(msg) AssertionError: Items are not equal: ACTUAL: 1 DESIRED: 4 -- Olivier http://twitter.com/ogrisel - http://github.com/ogrisel
On Wed, Apr 6, 2016 at 2:04 AM, Olivier Grisel <olivier.grisel@ensta.org> wrote:
2016-04-05 19:44 GMT+02:00 Nathaniel Smith <njs@pobox.com>:
I propose to hold off distributing the OpenBLAS wheels until the OpenBLAS tests are clean on the OpenBLAS buildbots - any objections?
Alternatively, would it make sense to add a local patch to our openblas builds to blacklist the piledriver kernel and then distribute them now? (I'm not immediately sure what would be involved in doing this but it seems unlikely that it would require anything tricky?)
I tried to force use the NEHALEM or the BARCELONA driver on a PILEDRIVER AMD box and while it fixes the original test failure in isolve it causes this other scipy test to fail:
====================================================================== FAIL: test_nanmedian_all_axis (test_stats.TestNanFunc) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib/python2.7/site-packages/scipy/stats/tests/test_stats.py", line 242, in test_nanmedian_all_axis assert_equal(len(w), 4) File "/usr/local/lib/python2.7/site-packages/numpy/testing/utils.py", line 375, in assert_equal raise AssertionError(msg) AssertionError: Items are not equal: ACTUAL: 1 DESIRED: 4
I'm reading this email next to https://github.com/xianyi/OpenBLAS-CI/issues/10#issuecomment-206195714 and I'm confused :-). -n -- Nathaniel J. Smith -- https://vorpus.org
Yes sorry I forgot to update the thread. Actually I am no longer sure how I go this error. I am re-running the full test suite because I cannot reproduce it when running the test_stats.py module alone. -- Olivier
I updated the issue: https://github.com/xianyi/OpenBLAS-CI/issues/10#issuecomment-206195714 The random test_nanmedian_all_axis failure is unrelated to openblas and should be ignored. -- Olivier
On Wed, Apr 6, 2016 at 6:47 AM, Olivier Grisel <olivier.grisel@ensta.org> wrote:
I updated the issue:
https://github.com/xianyi/OpenBLAS-CI/issues/10#issuecomment-206195714
The random test_nanmedian_all_axis failure is unrelated to openblas and should be ignored.
It looks like all is well now, at least for the OpenBLAS buildbots : http://build.openblas.net/builders Matthew
participants (6)
-
Carl Kleffner -
Freddy Rietdijk -
Jonathan Helmus -
Matthew Brett -
Nathaniel Smith -
Olivier Grisel