On Mon, Nov 27, 2023 at 2:10 PM Dr. Thomas Orgis < thomas.orgis@uni-hamburg.de> wrote:
Hi,
I'm involved in packaging NumPy for http://pkgsrc.org/. We install a set of possible BLAS/CBLAS/LAPACK/LAPACKE packages side-by-side in the same prefix. This includes multiple variants of OpenBLAS with regard to multithreading (and indexing). For this purpose, we point software to use the build-time chosen BLAS implementation via BLAS_LIBS and similar variables, or, as seems to be appropriate for the new meson build of NumPy, via .pc files.
A package depends on the generic BLAS library family and central user configuration chooses which one the packages should use during build.
What would be the correct way to force the NumPy build to just use our BLAS choice, avoiding any automatisms that might surprise us?
How agnostic is NumPy regarding the offered BLAS? Does it have to know that it is using OpenMP-parallelized OpenBLAS vs. the serial one (I'd imagine just setting OMP_NUM_THREADS handles paralellism)
The NumPy build does not know anything about this. It will just build, and it will simply call the OpenBLAS functionality - whether those execute under the hood in parallel or not, or with OpenMP or pthreads, is unknown. When a user or downstream library wants to control that parallelism, they can use an environment variable or https://github.com/joblib/threadpoolctl. or MKL, Netlib, … ? By default, the NumPy build will try all of those, in the order given by the `blas-order` and `lapack-order` build options (see `meson_options.txt in the root of the repo). It doesn't scale to have to tell it "openblas" or "netlib",
as there is no universal vocabulary to name the variants, and NumPy doesn't even know openblas_openmp from serial openblas or openblas_pthread (right?).
Basically, I want to do
meson setup -Dcblas_pc=$CBLAS_PC
with CBLAS_PC being the module name of one of
$prefix/lib/pkgconfig/cblas.pc $prefix/lib/pkgconfig/openblas_pthread.pc $prefix/lib/pkgconfig/openblas_openmp.pc $prefix/lib/pkgconfig/openblas64_openmp.pc
so that pkg-config does its thing without the NumPy build guessing around. Is that feasible already? Is it easily supportable with some changes to the build? I dream of a world where package build scripts don't have to add dozens of idiosyncratic lines to detect these libs.
Yes, that is possible. You should be building with a build frontend (pip or pypa/build) and then the invocation will include `-C-Dblas=<your .pc name> -C-Dlapack=<your .pc name>`. See http://scipy.github.io/devdocs/building/blas_lapack.html for more guidance.
I'd like things to work like for CMake's FindBLAS with -DBLA_PREFER_PKGCONFIG and -DBLA_PKGCONFIG_BLAS=$BLAS_PC (see
https://cmake.org/cmake/help/latest/module/FindBLAS.html
since version 3.25).
Can we have that?
Yes, that is implemented since NumPy 1.26.2 and in the main branch.
And: NumPy needs CBLAS … does it also need LAPACKE instead of LAPACK?
No need for LAPACKE.
These are differing libraries, possibly coming in differing binaries, even if your OpenBLAS builds also combine them. So I guess it should be -Dcblas_pc and -Dlapacke_pc, both being possibly identical. A build of the reference Netlib implementation provides four distinct libraries and .pc files:
$prefix/lib/pkgconfig/cblas.pc $prefix/lib/pkgconfig/blas.pc $prefix/lib/pkgconfig/lapacke.pc $prefix/lib/pkgconfig/lapack.pc
We do support installing openblas64 and friends alongside the others and I imagine just setting an ILP64 option and repective symbol suffix (none as of yet, as it's not a settled thing upstream) for the NumPy build if a 64 variant is chosen by the user. I wonder a bit if there are possible pitfalls combining other libraries with Python and indirectly some incompatible BLAS variant via NumPy … but one point of our user choice is that they could ensure that all packages really use the same BLAS.
You have to opt in to ILP64, via a `-Duse-ilp64` flag. It will not work to craft a blas.pc which points at a 64-bit BLAS. Cheers, Ralf
Alrighty then,
Thomas
-- Dr. Thomas Orgis HPC @ Universität Hamburg _______________________________________________ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-leave@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: ralf.gommers@googlemail.com