On Mon, Nov 27, 2023 at 2:10 PM Dr. Thomas Orgis <thomas.orgis@uni-hamburg.de> wrote:
Hi,

I'm involved in packaging NumPy for http://pkgsrc.org/. We install a
set of possible BLAS/CBLAS/LAPACK/LAPACKE packages side-by-side in the
same prefix. This includes multiple variants of OpenBLAS with regard to
multithreading (and indexing). For this purpose, we point software to
use the build-time chosen BLAS implementation via BLAS_LIBS and similar
variables, or, as seems to be appropriate for the new meson build of
NumPy, via .pc files.

A package depends on the generic BLAS library family and central user
configuration chooses which one the packages should use during build.

What would be the correct way to force the NumPy build to just use our
BLAS choice, avoiding any automatisms that might surprise us?

How agnostic is NumPy regarding the offered BLAS? Does it have to know
that it is using OpenMP-parallelized OpenBLAS vs. the serial one (I'd
imagine just setting OMP_NUM_THREADS handles paralellism)

The NumPy build does not know anything about this. It will just build, and it will simply call the OpenBLAS functionality - whether those execute under the hood in parallel or not, or with OpenMP or pthreads, is unknown. When a user or downstream library wants to control that parallelism, they can use an environment variable or https://github.com/joblib/threadpoolctl.

or MKL, Netlib, … ?

By default, the NumPy build will try all of those, in the order given by the `blas-order` and `lapack-order` build options (see `meson_options.txt in the root of the repo).

It doesn't scale to have to tell it "openblas" or "netlib",
as there is no universal vocabulary to name the variants, and NumPy
doesn't even know openblas_openmp from serial openblas or
openblas_pthread (right?).

Basically, I want to do

        meson setup -Dcblas_pc=$CBLAS_PC

with CBLAS_PC being the module name of one of

$prefix/lib/pkgconfig/cblas.pc
$prefix/lib/pkgconfig/openblas_pthread.pc
$prefix/lib/pkgconfig/openblas_openmp.pc
$prefix/lib/pkgconfig/openblas64_openmp.pc

so that pkg-config does its thing without the NumPy build guessing
around. Is that feasible already? Is it easily supportable with some
changes to the build? I dream of a world where package build scripts
don't have to add dozens of idiosyncratic lines to detect these libs.

Yes, that is possible. You should be building with a build frontend (pip or pypa/build) and then the invocation will include `-C-Dblas=<your .pc name>  -C-Dlapack=<your .pc name>`. See http://scipy.github.io/devdocs/building/blas_lapack.html for more guidance.


I'd like things to work like for CMake's FindBLAS with
-DBLA_PREFER_PKGCONFIG and -DBLA_PKGCONFIG_BLAS=$BLAS_PC (see

        https://cmake.org/cmake/help/latest/module/FindBLAS.html

since version 3.25).

Can we have that?

Yes, that is implemented since NumPy 1.26.2 and in the main branch.
 

And: NumPy needs CBLAS … does it also need LAPACKE instead of LAPACK?

No need for LAPACKE.
 
These are differing libraries, possibly coming in differing binaries,
even if your OpenBLAS builds also combine them. So I guess it should be
-Dcblas_pc and -Dlapacke_pc, both being possibly identical. A build of
the reference Netlib implementation provides four distinct libraries
and .pc files:

$prefix/lib/pkgconfig/cblas.pc
$prefix/lib/pkgconfig/blas.pc
$prefix/lib/pkgconfig/lapacke.pc
$prefix/lib/pkgconfig/lapack.pc

We do support installing openblas64 and friends alongside the others
and I imagine just setting an ILP64 option and repective symbol suffix
(none as of yet, as it's not a settled thing upstream) for the NumPy
build if a 64 variant is chosen by the user. I wonder a bit if there
are possible pitfalls combining other libraries with Python and
indirectly some incompatible BLAS variant via NumPy … but one point of
our user choice is that they could ensure that all packages really use
the same BLAS.

You have to opt in to ILP64, via a `-Duse-ilp64` flag. It will not work to craft a blas.pc which points at a 64-bit BLAS.

Cheers,
Ralf

 


Alrighty then,

Thomas

--
Dr. Thomas Orgis
HPC @ Universität Hamburg
_______________________________________________
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-leave@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: ralf.gommers@googlemail.com