Telling meson build which CBLAS/LAPACK (LAPACKE?) to use via pkgconfig module
Hi, I'm involved in packaging NumPy for http://pkgsrc.org/. We install a set of possible BLAS/CBLAS/LAPACK/LAPACKE packages side-by-side in the same prefix. This includes multiple variants of OpenBLAS with regard to multithreading (and indexing). For this purpose, we point software to use the build-time chosen BLAS implementation via BLAS_LIBS and similar variables, or, as seems to be appropriate for the new meson build of NumPy, via .pc files. A package depends on the generic BLAS library family and central user configuration chooses which one the packages should use during build. What would be the correct way to force the NumPy build to just use our BLAS choice, avoiding any automatisms that might surprise us? How agnostic is NumPy regarding the offered BLAS? Does it have to know that it is using OpenMP-parallelized OpenBLAS vs. the serial one (I'd imagine just setting OMP_NUM_THREADS handles paralellism) or MKL, Netlib, … ? It doesn't scale to have to tell it "openblas" or "netlib", as there is no universal vocabulary to name the variants, and NumPy doesn't even know openblas_openmp from serial openblas or openblas_pthread (right?). Basically, I want to do meson setup -Dcblas_pc=$CBLAS_PC with CBLAS_PC being the module name of one of $prefix/lib/pkgconfig/cblas.pc $prefix/lib/pkgconfig/openblas_pthread.pc $prefix/lib/pkgconfig/openblas_openmp.pc $prefix/lib/pkgconfig/openblas64_openmp.pc so that pkg-config does its thing without the NumPy build guessing around. Is that feasible already? Is it easily supportable with some changes to the build? I dream of a world where package build scripts don't have to add dozens of idiosyncratic lines to detect these libs. I'd like things to work like for CMake's FindBLAS with -DBLA_PREFER_PKGCONFIG and -DBLA_PKGCONFIG_BLAS=$BLAS_PC (see https://cmake.org/cmake/help/latest/module/FindBLAS.html since version 3.25). Can we have that? And: NumPy needs CBLAS … does it also need LAPACKE instead of LAPACK? These are differing libraries, possibly coming in differing binaries, even if your OpenBLAS builds also combine them. So I guess it should be -Dcblas_pc and -Dlapacke_pc, both being possibly identical. A build of the reference Netlib implementation provides four distinct libraries and .pc files: $prefix/lib/pkgconfig/cblas.pc $prefix/lib/pkgconfig/blas.pc $prefix/lib/pkgconfig/lapacke.pc $prefix/lib/pkgconfig/lapack.pc We do support installing openblas64 and friends alongside the others and I imagine just setting an ILP64 option and repective symbol suffix (none as of yet, as it's not a settled thing upstream) for the NumPy build if a 64 variant is chosen by the user. I wonder a bit if there are possible pitfalls combining other libraries with Python and indirectly some incompatible BLAS variant via NumPy … but one point of our user choice is that they could ensure that all packages really use the same BLAS. Alrighty then, Thomas -- Dr. Thomas Orgis HPC @ Universität Hamburg
On Mon, Nov 27, 2023 at 2:10 PM Dr. Thomas Orgis < thomas.orgis@uni-hamburg.de> wrote:
Hi,
I'm involved in packaging NumPy for http://pkgsrc.org/. We install a set of possible BLAS/CBLAS/LAPACK/LAPACKE packages side-by-side in the same prefix. This includes multiple variants of OpenBLAS with regard to multithreading (and indexing). For this purpose, we point software to use the build-time chosen BLAS implementation via BLAS_LIBS and similar variables, or, as seems to be appropriate for the new meson build of NumPy, via .pc files.
A package depends on the generic BLAS library family and central user configuration chooses which one the packages should use during build.
What would be the correct way to force the NumPy build to just use our BLAS choice, avoiding any automatisms that might surprise us?
How agnostic is NumPy regarding the offered BLAS? Does it have to know that it is using OpenMP-parallelized OpenBLAS vs. the serial one (I'd imagine just setting OMP_NUM_THREADS handles paralellism)
The NumPy build does not know anything about this. It will just build, and it will simply call the OpenBLAS functionality - whether those execute under the hood in parallel or not, or with OpenMP or pthreads, is unknown. When a user or downstream library wants to control that parallelism, they can use an environment variable or https://github.com/joblib/threadpoolctl. or MKL, Netlib, … ? By default, the NumPy build will try all of those, in the order given by the `blas-order` and `lapack-order` build options (see `meson_options.txt in the root of the repo). It doesn't scale to have to tell it "openblas" or "netlib",
as there is no universal vocabulary to name the variants, and NumPy doesn't even know openblas_openmp from serial openblas or openblas_pthread (right?).
Basically, I want to do
meson setup -Dcblas_pc=$CBLAS_PC
with CBLAS_PC being the module name of one of
$prefix/lib/pkgconfig/cblas.pc $prefix/lib/pkgconfig/openblas_pthread.pc $prefix/lib/pkgconfig/openblas_openmp.pc $prefix/lib/pkgconfig/openblas64_openmp.pc
so that pkg-config does its thing without the NumPy build guessing around. Is that feasible already? Is it easily supportable with some changes to the build? I dream of a world where package build scripts don't have to add dozens of idiosyncratic lines to detect these libs.
Yes, that is possible. You should be building with a build frontend (pip or pypa/build) and then the invocation will include `-C-Dblas=<your .pc name> -C-Dlapack=<your .pc name>`. See http://scipy.github.io/devdocs/building/blas_lapack.html for more guidance.
I'd like things to work like for CMake's FindBLAS with -DBLA_PREFER_PKGCONFIG and -DBLA_PKGCONFIG_BLAS=$BLAS_PC (see
https://cmake.org/cmake/help/latest/module/FindBLAS.html
since version 3.25).
Can we have that?
Yes, that is implemented since NumPy 1.26.2 and in the main branch.
And: NumPy needs CBLAS … does it also need LAPACKE instead of LAPACK?
No need for LAPACKE.
These are differing libraries, possibly coming in differing binaries, even if your OpenBLAS builds also combine them. So I guess it should be -Dcblas_pc and -Dlapacke_pc, both being possibly identical. A build of the reference Netlib implementation provides four distinct libraries and .pc files:
$prefix/lib/pkgconfig/cblas.pc $prefix/lib/pkgconfig/blas.pc $prefix/lib/pkgconfig/lapacke.pc $prefix/lib/pkgconfig/lapack.pc
We do support installing openblas64 and friends alongside the others and I imagine just setting an ILP64 option and repective symbol suffix (none as of yet, as it's not a settled thing upstream) for the NumPy build if a 64 variant is chosen by the user. I wonder a bit if there are possible pitfalls combining other libraries with Python and indirectly some incompatible BLAS variant via NumPy … but one point of our user choice is that they could ensure that all packages really use the same BLAS.
You have to opt in to ILP64, via a `-Duse-ilp64` flag. It will not work to craft a blas.pc which points at a 64-bit BLAS. Cheers, Ralf
Alrighty then,
Thomas
-- Dr. Thomas Orgis HPC @ Universität Hamburg _______________________________________________ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-leave@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: ralf.gommers@googlemail.com
Am Mon, 27 Nov 2023 14:58:45 +0100 schrieb Ralf Gommers <ralf.gommers@gmail.com>:
The NumPy build does not know anything about this. It will just build, and it will simply call the OpenBLAS functionality
Great!
Yes, that is possible. You should be building with a build frontend (pip or pypa/build) and then the invocation will include `-C-Dblas=<your .pc name> -C-Dlapack=<your .pc name>`.
I'm confused about these frontends, I must say. I imagined that if you're using meson, one could just call meson setup/build? That being said: I am not sure now myself how the pkgsrc build actually works right now. There's common machinery to 'build python stuff' and the part about meson-based packages is rather fresh and not documented yet. The build output starts with * Building wheel... + /data/pkg/bin/python3.11 /data/projekte/pkgsrc/work/math/py-numpy/work/numpy-1.26.2/vendored-meson/meson/meson.py setup /data/projekte/pkgsrc/work/math/py-numpy/work/numpy-1.26.2 /data/projekte/pkgsrc/work/math/py-numpy/work/numpy-1.26.2/.mesonpy-_lv … so some wrapped call to a vendored copy of meson that NumPy ships. Adding -Dblas=$CBLAS_PC to that command should do the trick, no? (however that is effected)
And: NumPy needs CBLAS … does it also need LAPACKE instead of LAPACK?
No need for LAPACKE.
Good, if also somewhat weird;-) I'm curious, though: When you need the CBLAS API, why is the dependency called blas and not cblas? In practice, most accelerated libraries offer all APIs in one binary and -Dlapack is already redundant, but when we use the netlib reference, blas, cblas, lapack, and lapacke are distinct entities. Calling cblas just blas where lapack _does_ mean the Fortran one, is rather confusing.
You have to opt in to ILP64, via a `-Duse-ilp64` flag. It will not work to craft a blas.pc which points at a 64-bit BLAS.
So -Dblas=openblas64 -Dlapack=openblas64 -Duse-ilp64 would do it, right? Alrighty then, Thomas PS: You might want to fix that one: ../../numpy/meson.build:124: WARNING: Project targets '>=1.2.99' but uses feature introduced in '1.3.0': dep 'blas' custom lookup. -- Dr. Thomas Orgis HPC @ Universität Hamburg
On Mon, Nov 27, 2023 at 6:51 PM Dr. Thomas Orgis < thomas.orgis@uni-hamburg.de> wrote:
Am Mon, 27 Nov 2023 14:58:45 +0100 schrieb Ralf Gommers <ralf.gommers@gmail.com>:
The NumPy build does not know anything about this. It will just build, and it will simply call the OpenBLAS functionality
Great!
Yes, that is possible. You should be building with a build frontend (pip or pypa/build) and then the invocation will include `-C-Dblas=<your .pc name> -C-Dlapack=<your .pc name>`.
I'm confused about these frontends, I must say. I imagined that if you're using meson, one could just call meson setup/build? That being said: I am not sure now myself how the pkgsrc build actually works right now. There's common machinery to 'build python stuff' and the part about meson-based packages is rather fresh and not documented yet.
You have to go through a "build frontend" to produce a wheel, which then gets installed/repackaged for your distro. If you call meson/ninja directly, you will not get the Python package metadata that meson-python produces. And you do need that, others there are some things that will break (e.g., using `importlib` APIs for introspecting Python packages). So what your machinery should be doing is building with `pip install . --no-build-isolation` or `python -m build --no-isolation`.
The build output starts with
* Building wheel... + /data/pkg/bin/python3.11 /data/projekte/pkgsrc/work/math/py-numpy/work/numpy-1.26.2/vendored-meson/meson/meson.py setup /data/projekte/pkgsrc/work/math/py-numpy/work/numpy-1.26.2 /data/projekte/pkgsrc/work/math/py-numpy/work/numpy-1.26.2/.mesonpy-_lv
… so some wrapped call to a vendored copy of meson that NumPy ships.
Yes, we need that for extra BLAS/LAPACK and SIMD related functionality that is still in the process of being upstreamed into Meson.
Adding -Dblas=$CBLAS_PC to that command should do the trick, no? (however that is effected)
Sounds like it, assuming CBLAS_PC is the name of a library.
And: NumPy needs CBLAS … does it also need LAPACKE instead of LAPACK?
No need for LAPACKE.
Good, if also somewhat weird;-) I'm curious, though: When you need the CBLAS API, why is the dependency called blas and not cblas? In practice, most accelerated libraries offer all APIs in one binary and -Dlapack is already redundant, but when we use the netlib reference, blas, cblas, lapack, and lapacke are distinct entities. Calling cblas just blas where lapack _does_ mean the Fortran one, is rather confusing.
Partly a matter of history since we always did it like this, but I think there's more to it. The two libraries are called BLAS and LAPACK, those offer distinct functionality. CBLAS and LAPACKE are basically much less important implementation details, and typically shipped in the same library because they're interfaces to the exact same functionality. We're not "calling CBLAS just BLAS" here, but rather: BLAS is the main name and has the functionality you want. CBLAS is an optional interface to that, and if you want it you have to ask for it with (in Meson): dependency('blas', modules: ['cblas']) It doesn't make much sense for us to expose CBLAS (or LAPACKE) as a separate thing in our own build interface.
You have to opt in to ILP64, via a `-Duse-ilp64` flag. It will not work to craft a blas.pc which points at a 64-bit BLAS.
So -Dblas=openblas64 -Dlapack=openblas64 -Duse-ilp64 would do it, right?
Exactly.
Alrighty then,
Thomas
PS: You might want to fix that one:
../../numpy/meson.build:124: WARNING: Project targets '>=1.2.99' but uses feature introduced in '1.3.0': dep 'blas' custom lookup.
Yeah, that'll go away when we update the vendored copy, will be done in the next few days. Cheers, Ralf
-- Dr. Thomas Orgis HPC @ Universität Hamburg _______________________________________________ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-leave@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: ralf.gommers@gmail.com
Thanks a lot for the patience and explanations. Am Mon, 27 Nov 2023 19:59:13 +0100 schrieb Ralf Gommers <ralf.gommers@gmail.com>:
You have to go through a "build frontend" to produce a wheel, which then gets installed/repackaged for your distro.
This is obviously happening in pkgsrc.
The build output starts with
* Building wheel...
I only need to add those -C-Dblas= parameters at the correct place.
… so some wrapped call to a vendored copy of meson that NumPy ships.
Yes, we need that for extra BLAS/LAPACK and SIMD related functionality that is still in the process of being upstreamed into Meson.
… and I dream of the world where BLAS is 'just a library' without any special support in build systems.
Adding -Dblas=$CBLAS_PC to that command should do the trick, no? (however that is effected)
Sounds like it, assuming CBLAS_PC is the name of a library.
It's the name to give to pkg-config to identify the module for CBLAS.
offer distinct functionality. CBLAS and LAPACKE are basically much less important implementation details, and typically shipped in the same library because they're interfaces to the exact same functionality.
Yes and no. The Netlib reference implementation comes (or at least can be built that way) in separate pieces, the code of which is what the optimized BLAS vendors like to use to build those combined libraries. So in my case, CBLAS_PC can be just cblas, with the effect $ pkg-config --libs cblas -L/data/pkg/lib -lcblas and $ pkg-config --libs --static cblas -L/data/pkg/lib -lcblas -lblas The reference libcblas is a separate library which could be used to wrap around any plain BLAS library. Would the meson machinery be confused by that? In practice, for optimized builds, you'll get all 4 pieces in one package (openblas at least), but I'm always also dealing with the Netlib case where CBLAS is something different from BLAS.
dependency('blas', modules: ['cblas'])
So there's this elaborate machinery that special-cases BLAS as a library that could (not) offer certain sets of API? Well, as long as the dynamic libcblas is used, the base BLAS API should be transparently avaible and those meson tests will work … ? I'll do some testing tomorrow, at least. Alrighty then, Thomas -- Dr. Thomas Orgis HPC @ Universität Hamburg
Am Sun, 3 Dec 2023 19:54:10 +0100 schrieb "Dr. Thomas Orgis" <thomas.orgis@uni-hamburg.de>:
You have to go through a "build frontend" to produce a wheel, which then gets installed/repackaged for your distro.
This is obviously happening in pkgsrc.
I'll do some testing tomorrow, at least.
Well, now is another day. Pkgsrc uses python -m build and I added -Csetup-args=-Dblas=${CBLAS_PC} -Csetup-args=-Dlapack=${LAPACK_PC} which seems to work out fine using cblas.pc and lapack.pc in the case of the netlib install. In fact, most linking is done only to libblas.so instead of libcblas.so, as the linker is smart enough to throw away the unused lib. Alrighty then, Thomas -- Dr. Thomas Orgis HPC @ Universität Hamburg
On Wed, Dec 6, 2023 at 1:32 AM Dr. Thomas Orgis <thomas.orgis@uni-hamburg.de> wrote:
Am Sun, 3 Dec 2023 19:54:10 +0100 schrieb "Dr. Thomas Orgis" <thomas.orgis@uni-hamburg.de>:
You have to go through a "build frontend" to produce a wheel, which then gets installed/repackaged for your distro.
This is obviously happening in pkgsrc.
I'll do some testing tomorrow, at least.
Well, now is another day. Pkgsrc uses python -m build and I added
-Csetup-args=-Dblas=${CBLAS_PC} -Csetup-args=-Dlapack=${LAPACK_PC}
which seems to work out fine using cblas.pc and lapack.pc in the case of the netlib install. In fact, most linking is done only to libblas.so instead of libcblas.so, as the linker is smart enough to throw away the unused lib.
Great, thanks for confirming! Cheers, Ralf
Hapy holidays … but I have an issue still that hopefully can be addressed with the meson blas detection you are upstreaming(?). Am Wed, 6 Dec 2023 18:06:01 +0100 schrieb Ralf Gommers <ralf.gommers@gmail.com>:
Well, now is another day. Pkgsrc uses python -m build and I added
-Csetup-args=-Dblas=${CBLAS_PC} -Csetup-args=-Dlapack=${LAPACK_PC}
which seems to work out fine using cblas.pc and lapack.pc in the case of the netlib install. In fact, most linking is done only to libblas.so instead of libcblas.so, as the linker is smart enough to throw away the unused lib.
Great, thanks for confirming!
This works for numpy and also installs scipy nicely, but this produces a broken scipy install when using netlib reference libraries from pkgsrc. These come as libblas.so liblapack.so (NEEDing libblas.so) libcblas.so (NEEDing libblas.so) libpapacke.so (NEEDing liblapack.so, hence libblas.so) and their respective .pc files. This is the natural order that occus to me when building from netlib upstream. This also means that one could just replace BLAS and put stock LAPACK on top, what optimized BLAS libs usually start out with. Only that they tend to pack all symbols into one common library, which then project builds like numpy rely on. Telling the meson build that BLAS is libcblas works as long as actually CBLAS symbols are used. If not — I presume now, as I didn't yet see the actual build lines that are triggered via the python -m build and meson indirections — the linker might discard the -lcblas and leave symbols unresolved (--as-needed but no --no-undefined). This happens with scipy: $ LANG=C readelf -d /data/pkg/lib/python3.11/site-packages/scipy/sparse/linalg/_dsolve/_superlu.so |grep NEEDED 0x0000000000000001 (NEEDED) Shared library: [libm.so.6] 0x0000000000000001 (NEEDED) Shared library: [libc.so.6] It would link against libopenblas_openmp.so if that had been the CBLAS (and LAPACK) choice and all would be fine, but here, it should link with libcblas.so, or directly to libblas.so, just like our regular install of superlu: $ LANG=C readelf -d /data/pkg/lib/libsuperlu.so|grep NEEDED 0x0000000000000001 (NEEDED) Shared library: [libblas.so.3] 0x0000000000000001 (NEEDED) Shared library: [libm.so.6] 0x0000000000000001 (NEEDED) Shared library: [libc.so.6] Of course, just not vendoring superlu would be one solution for scipy, but I think the deeper issue with the meson BLAS support should be solved: The 4 parts of the BLAS canon (not talking about SCALAPACK etc. yet) need to be handled explicitly. It is confusing, though, as meson prints this: Run-time dependency blas found: YES 3.11.0 Run-time dependency cblas found: YES 3.11.0 Run-time dependency lapack found: YES 3.11.0 It suggests that it looked for and found 3 libraries, but actually, it only cared for -llapack and -lcblas. It needs to find -lblas directly, too (or the cblas package separately, for that matter, not as component of blas). Is that easily fixable from your side? (I'm assuming numpy, scipy and the future stock BLAS support of meson are handled together.) Is this just an oversight on the scipy side and they could link the vendored superly with -lblas without changing the meson machinery? Alrighty then, Thomas -- Dr. Thomas Orgis HPC @ Universität Hamburg
participants (2)
-
Dr. Thomas Orgis -
Ralf Gommers