NumPy's BLAS library on macOS?
Would someone please answer installation questions about NumPy's BLAS on macOS? I'm not finding the answers in the release notes <https://github.com/numpy/numpy/releases>, the PR <https://github.com/numpy/numpy/pull/18874> source, the docs <https://numpy.org/doc/1.21/>, or Stack Overflow <https://stackoverflow.com/search?tab=newest&q=%5bnumpy%5d%20openblas>. Q1. The NumPy 1.21.0 release note <https://github.com/numpy/numpy/releases/tag/v1.21.0> says "This change enables the Accelerate Framework as an option on macOS." How to set that option on/off? Q2. How to determine if NumPy uses Accelerate vs. its internal copy of OpenBLAS? After installing a wheel, `numpy.show_config()` shows the openblas_info library_dirs et al as '/usr/local/lib'. Neither '/usr/local/lib/' nor 'site-packages/numpy/' contains a *blas*.so library (for Python 3.8.* on macOS 10.14.6) but the doc <https://numpy.org/install/> says "The OpenBLAS libraries are included in the wheel." Q3. How to pip install NumPy 1.21.0 in a way that ensures it uses its embedded OpenBLAS on macOS as on Linux? I'm aiming for as portable results as possible. Or should we link NumPy to an external OpenBLAS via `pip install numpy --no-binary numpy==1.21.0` with `~/.numpy-site.cfg`? (Ditto for SciPy.) Q4. Can the new NPY_* environment variables select specific BLAS & LAPACK libraries through pip install, and perhaps install faster than building NumPy, SciPy, etc. from source? How to do that? Q5. Is NumPy's embedded OpenBLAS compiled by gcc or clang? Is that controllable via `pip install`? Thank you!
On Wed, Jul 7, 2021 at 9:56 PM Jerry Morrison < jerry.morrison+numpy@gmail.com> wrote:
Would someone please answer installation questions about NumPy's BLAS on macOS? I'm not finding the answers in the release notes <https://github.com/numpy/numpy/releases>, the PR <https://github.com/numpy/numpy/pull/18874> source, the docs <https://numpy.org/doc/1.21/>, or Stack Overflow <https://stackoverflow.com/search?tab=newest&q=%5bnumpy%5d%20openblas>.
Q1. The NumPy 1.21.0 release note <https://github.com/numpy/numpy/releases/tag/v1.21.0> says "This change enables the Accelerate Framework as an option on macOS." How to set that option on/off?
It's autodetected at build time. If you have no other BLAS installed, it will be used. Or explicitly select it with NPY_BLAS_ORDER/NPY_LAPACK_ORDER
Q2. How to determine if NumPy uses Accelerate vs. its internal copy of OpenBLAS? After installing a wheel, `numpy.show_config()` shows the openblas_info library_dirs et al as '/usr/local/lib'. Neither '/usr/local/lib/' nor 'site-packages/numpy/' contains a *blas*.so library (for Python 3.8.* on macOS 10.14.6) but the doc <https://numpy.org/install/> says "The OpenBLAS libraries are included in the wheel."
It's a build-time option, you cannot select it at runtime.
Q3. How to pip install NumPy 1.21.0 in a way that ensures it uses its embedded OpenBLAS on macOS as on Linux? I'm aiming for as portable results as possible. Or should we link NumPy to an external OpenBLAS via `pip install numpy --no-binary numpy==1.21.0` with `~/.numpy-site.cfg`? (Ditto for SciPy.)
If you install a wheel, you will always get the bundled OpenBLAS on every platform for which we have binary wheels.
Q4. Can the new NPY_* environment variables select specific BLAS & LAPACK libraries through pip install, and perhaps install faster than building NumPy, SciPy, etc. from source? How to do that?
This question seems a little bit confused. Those env vars just select the BLAS/LAPACK library. It will not affect build time - we're never building BLAS or LAPACK itself from source.
Q5. Is NumPy's embedded OpenBLAS compiled by gcc or clang? Is that controllable via `pip install`?
gcc/gfortran. and no, you cannot control it through pip Cheers, Ralf
Thank you! _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
Got it! *Summary:* * Installing a numpy wheel (e.g. `pip install numpy==1.21.0`) uses its embedded OpenBLAS on every platform that has a wheel. That OpenBLAS is always compiled with gcc/gfortran. In this case, `np.show_config()` reports `library_dirs = ['/usr/local/lib']` even though there's no libblas in that directory. * Installing numpy from source (e.g. `pip install numpy==1.21.0 --no-binary numpy)` looks for BLAS & LAPACK libraries at build time as influenced by the environment vars NPY_BLAS_ORDER/NPY_LAPACK_ORDER or by the file ~/.numpy-site.cfg. On macOS, 'accelerate' is in the default search order after 'openblas'. On macOS < 11.3, importing numpy that's linked to Accelerate will detect an Accelerate bug and raise a RuntimeError. On Wed, Jul 7, 2021 at 1:32 PM Ralf Gommers <ralf.gommers@gmail.com> wrote:
On Wed, Jul 7, 2021 at 9:56 PM Jerry Morrison < jerry.morrison+numpy@gmail.com> wrote:
Would someone please answer installation questions about NumPy's BLAS on macOS? I'm not finding the answers in the release notes <https://github.com/numpy/numpy/releases>, the PR <https://github.com/numpy/numpy/pull/18874> source, the docs <https://numpy.org/doc/1.21/>, or Stack Overflow <https://stackoverflow.com/search?tab=newest&q=%5bnumpy%5d%20openblas>.
Q1. The NumPy 1.21.0 release note <https://github.com/numpy/numpy/releases/tag/v1.21.0> says "This change enables the Accelerate Framework as an option on macOS." How to set that option on/off?
It's autodetected at build time. If you have no other BLAS installed, it will be used. Or explicitly select it with NPY_BLAS_ORDER/NPY_LAPACK_ORDER
Q2. How to determine if NumPy uses Accelerate vs. its internal copy of OpenBLAS? After installing a wheel, `numpy.show_config()` shows the openblas_info library_dirs et al as '/usr/local/lib'. Neither '/usr/local/lib/' nor 'site-packages/numpy/' contains a *blas*.so library (for Python 3.8.* on macOS 10.14.6) but the doc <https://numpy.org/install/> says "The OpenBLAS libraries are included in the wheel."
It's a build-time option, you cannot select it at runtime.
Q3. How to pip install NumPy 1.21.0 in a way that ensures it uses its embedded OpenBLAS on macOS as on Linux? I'm aiming for as portable results as possible. Or should we link NumPy to an external OpenBLAS via `pip install numpy --no-binary numpy==1.21.0` with `~/.numpy-site.cfg`? (Ditto for SciPy.)
If you install a wheel, you will always get the bundled OpenBLAS on every platform for which we have binary wheels.
Q4. Can the new NPY_* environment variables select specific BLAS & LAPACK libraries through pip install, and perhaps install faster than building NumPy, SciPy, etc. from source? How to do that?
This question seems a little bit confused. Those env vars just select the BLAS/LAPACK library. It will not affect build time - we're never building BLAS or LAPACK itself from source.
Q5. Is NumPy's embedded OpenBLAS compiled by gcc or clang? Is that controllable via `pip install`?
gcc/gfortran. and no, you cannot control it through pip
Cheers, Ralf
Thank you! _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
On 8/7/21 2:23 am, Jerry Morrison wrote:
Got it!
*Summary:* * Installing a numpy wheel (e.g. `pip install numpy==1.21.0`) uses its embedded OpenBLAS on every platform that has a wheel. That OpenBLAS is always compiled with gcc/gfortran. In this case, `np.show_config()` reports `library_dirs = ['/usr/local/lib']` even though there's no libblas in that directory.
* Installing numpy from source (e.g. `pip install numpy==1.21.0 --no-binary numpy)` looks for BLAS & LAPACK libraries at build time as influenced by the environment vars NPY_BLAS_ORDER/NPY_LAPACK_ORDER or by the file ~/.numpy-site.cfg. On macOS, 'accelerate' is in the default search order after 'openblas'. On macOS < 11.3, importing numpy that's linked to Accelerate will detect an Accelerate bug and raise a RuntimeError.
That seems correct, although admittedly show_config could do a better job. The problem is that not every BLAS implementation provides a convenient method to self-report. It might be nice to document all this somewhere more permanent, the docstring for show_config might be a good place to start. Matti
On Wed, Jul 7, 2021 at 10:14 PM Matti Picus <matti.picus@gmail.com> wrote:
On 8/7/21 2:23 am, Jerry Morrison wrote:
Got it!
*Summary:* * Installing a numpy wheel (e.g. `pip install numpy==1.21.0`) uses its embedded OpenBLAS on every platform that has a wheel. That OpenBLAS is always compiled with gcc/gfortran. In this case, `np.show_config()` reports `library_dirs = ['/usr/local/lib']` even though there's no libblas in that directory.
* Installing numpy from source (e.g. `pip install numpy==1.21.0 --no-binary numpy)` looks for BLAS & LAPACK libraries at build time as influenced by the environment vars NPY_BLAS_ORDER/NPY_LAPACK_ORDER or by the file ~/.numpy-site.cfg. On macOS, 'accelerate' is in the default search order after 'openblas'. On macOS < 11.3, importing numpy that's linked to Accelerate will detect an Accelerate bug and raise a RuntimeError.
That seems correct, although admittedly show_config could do a better job. The problem is that not every BLAS implementation provides a convenient method to self-report.
For implementations that don't self-report, could show_config detect that it's using embedded OpenBLAS or a system Accelerate library?
It might be nice to document all this somewhere more permanent, the docstring for show_config might be a good place to start.
Agreed. On the https://numpy.org/install/ installation page? Do you want a PR? (How to get the translations?)
On Thu, Jul 8, 2021 at 11:47 PM Jerry Morrison < jerry.morrison+numpy@gmail.com> wrote:
On Wed, Jul 7, 2021 at 10:14 PM Matti Picus <matti.picus@gmail.com> wrote:
On 8/7/21 2:23 am, Jerry Morrison wrote:
Got it!
*Summary:* * Installing a numpy wheel (e.g. `pip install numpy==1.21.0`) uses its embedded OpenBLAS on every platform that has a wheel. That OpenBLAS is always compiled with gcc/gfortran. In this case, `np.show_config()` reports `library_dirs = ['/usr/local/lib']` even though there's no libblas in that directory.
* Installing numpy from source (e.g. `pip install numpy==1.21.0 --no-binary numpy)` looks for BLAS & LAPACK libraries at build time as influenced by the environment vars NPY_BLAS_ORDER/NPY_LAPACK_ORDER or by the file ~/.numpy-site.cfg. On macOS, 'accelerate' is in the default search order after 'openblas'. On macOS < 11.3, importing numpy that's linked to Accelerate will detect an Accelerate bug and raise a RuntimeError.
That seems correct, although admittedly show_config could do a better job. The problem is that not every BLAS implementation provides a convenient method to self-report.
For implementations that don't self-report, could show_config detect that it's using embedded OpenBLAS or a system Accelerate library?
It's not about "self reporting". What `show_config` shows for BLAS/LAPACK is the build time configuration, not the one at runtime. The paths are present in `numpy/__config.py__`, which is a file generated by the build process.
It might be nice to document all this somewhere more permanent, the docstring for show_config might be a good place to start.
Agreed. On the https://numpy.org/install/ installation page? Do you want a PR? (How to get the translations?)
Thanks, a PR would be nice. It's too detailed for the website; Matti's suggestion was in the docstring of `show_config`, which is defined at https://github.com/numpy/numpy/blob/main/numpy/distutils/misc_util.py#L2332 Cheers, Ralf
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
participants (3)
-
Jerry Morrison -
Matti Picus -
Ralf Gommers