[SciPy-User] Statically linking NumPy/SciPy against serial MKL

[Eric Hermes]

Hello,

I am re-sending this message because I don't believe it went through the 
first time.

I have spent a large amount of time in the past attempting to compile 
NumPy and SciPy in such a way that it can be used on clusters that do 
not have access to the Intel compiler and MKL libraries.  Specifically, 
one of the resources that I use utilizes the HTCondor distributed 
computing platform, where little can be guaranteed about the node that 
my jobs end up running on, and consequently any shared libraries that my 
code uses must be submitted along with the job itself.

As such, statically linking MKL libraries into NumPy and SciPy will make 
the code that I use substantially more portable (without recompiling), 
and less difficult to work with.  I have in the past managed to 
statically link in the MKL libraries that NumPy and SciPy need, but only 
by manually re-linking the libraries that these codes create, adding in 
the linker commands that the Intel Link Advisor suggests.

The official way to link against MKL in NumPy is to add an [mkl] section 
to the site.cfg configuration script, and pass the libraries that you 
wish to link against to mkl_libs (e.g. mkl_libs = mkl_intel_lp64, 
mkl_sequential, mkl_core).  I have glanced through the distutils scripts 
that NumPy uses, but I am not particularly familiar with distutils and 
as such I am having a difficult time figuring out how I would go about 
modifying the linker commands that the build script uses away from 
"-lmkl_intel_lp64 -lmkl_sequential -lmkl_core" to "-Wl,--start-group 
$(MKLROOT)/lib/intel64/libmkl_intel_lp64.a 
$(MKLROOT)/lib/intel64/libmkl_sequential.a 
$(MKLROOT)/lib/intel64/libmkl_core.a -Wl,--end-group".

Additionally, it would seem that the distutils script adds libguide and 
libmkl_avx to the list of libraries to link against, which I do not want 
because the former is a threading library and I want a serial version of 
NumPy, and the latter does not have a static version to link against.

Basically, I'm wondering if there isn't an easier way to compile NumPy 
and SciPy such that they are statically linked against the serial MKL 
libraries.  I know for example that adding "-static-intel" to the linker 
command statically links in the Intel compiler libraries, but as far as 
I can tell there is no such argument that does the same for MKL libraries.

It would currently seem to me that my only options are to manually 
relink all NumPy and SciPy libraries after they have been built by 
distutils, or to heavily modify the NumPy build scripts in some fashion 
to correctly compile them statically in the first place.  I would like 
to avoid the former, and I have no idea how I would go about doing the 
latter.

I am cross-posting this message to both the Intel MKL support forums, 
and the SciPy-user mailing list.

Thank you,

Eric Hermes

-- 
Eric Hermes
J.R. Schmidt Group
Chemistry Department
University of Wisconsin - Madison