NumPy distutils limitation?

Hi, Gregor and I are trying to give support for Intel's VML (Vector Mathematical Library) in numexpr. For this, we are trying to make use of the weaponery in NumPy's distutils so as to be able to discover where the MKL (the package that contains VML) is located. The libraries that we need to link with are: mkl_gf_lp64, mkl_gnu_thread, mkl_core *and* iomp5 (Intel's OpenMP library). The problem is that I have installed MKL as part as the Intel compiler for Unix. In this setup, most of the libraries are in one place, namely: /opt/intel/Compiler/11.0/074/mkl/lib/em64t/ However, the OpenMP library is in another directory: /opt/intel/Compiler/11.0/074/lib/intel64 So, I need to specify *two* directories to get the complete set of libraries. My first attempt was setting a site.cfg like: [DEFAULT] #libraries = gfortran [mkl] library_dirs= /opt/intel/Compiler/11.0/074/mkl/lib/em64t/:/opt/intel/Compiler/11.0/074/lib/intel64 include_dirs = /opt/intel/Compiler/11.0/074/mkl/include/ mkl_libs = mkl_gf_lp64, mkl_gnu_thread, mkl_core, iomp5 Unfortunately, distutils complains and says that it cannot find the complete set of libraries: mkl_info: libraries mkl_gf_lp64,mkl_gnu_thread,mkl_core,iomp5 not found in /opt/intel/Compiler/11.0/074/mkl/lib/em64t/ libraries mkl_gf_lp64,mkl_gnu_thread,mkl_core,iomp5 not found in /opt/intel/Compiler/11.0/074/lib/intel64 NOT AVAILABLE After some debugging of the problem, it seems that distutils needs to find *all* the required libraries in *one* single directory. As iomp5 is on a different directory, distutils thinks that the requisites are not fulfilled. I've solved this by requering the iomp5 to be find in the DEFAULT section. Something like: [DEFAULT] library_dirs = /opt/intel/Compiler/11.0/074/lib/intel64 #libraries = gfortran, iomp5 libraries = iomp5 [mkl] library_dirs = /opt/intel/Compiler/11.0/074/mkl/lib/em64t/ include_dirs = /opt/intel/Compiler/11.0/074/mkl/include/ mkl_libs = mkl_gf_lp64, mkl_gnu_thread, mkl_core However, in case one would need to specify several other directories in the DEFAULT section for finding other hypothetical necessary libraries (like gfortran or others), we may run into the same problem than above. My question is, is there an elegant way to handle this problem, or it is a limitation of the current distutils? If the later, it would be nice it that could be solved in a future version, and several libraries can be found in *several* directories. Thanks, -- Francesc Alted

On Fri, Jan 30, 2009 at 14:30, Francesc Alted <faltet@pytables.org> wrote:
My question is, is there an elegant way to handle this problem, or it is a limitation of the current distutils?
Probably the latter.
If the later, it would be nice it that could be solved in a future version, and several libraries can be found in *several* directories.
Patches are welcome. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco

On Sat, Jan 31, 2009 at 5:30 AM, Francesc Alted <faltet@pytables.org> wrote:
Hi,
Gregor and I are trying to give support for Intel's VML (Vector Mathematical Library) in numexpr. For this, we are trying to make use of the weaponery in NumPy's distutils so as to be able to discover where the MKL (the package that contains VML) is located. The libraries that we need to link with are: mkl_gf_lp64, mkl_gnu_thread, mkl_core *and* iomp5 (Intel's OpenMP library).
The problem is that I have installed MKL as part as the Intel compiler for Unix. In this setup, most of the libraries are in one place, namely:
/opt/intel/Compiler/11.0/074/mkl/lib/em64t/
However, the OpenMP library is in another directory:
/opt/intel/Compiler/11.0/074/lib/intel64
So, I need to specify *two* directories to get the complete set of libraries. My first attempt was setting a site.cfg like:
[DEFAULT] #libraries = gfortran
[mkl] library_dirs= /opt/intel/Compiler/11.0/074/mkl/lib/em64t/:/opt/intel/Compiler/11.0/074/lib/intel64 include_dirs = /opt/intel/Compiler/11.0/074/mkl/include/ mkl_libs = mkl_gf_lp64, mkl_gnu_thread, mkl_core, iomp5
Unfortunately, distutils complains and says that it cannot find the complete set of libraries:
mkl_info: libraries mkl_gf_lp64,mkl_gnu_thread,mkl_core,iomp5 not found in /opt/intel/Compiler/11.0/074/mkl/lib/em64t/ libraries mkl_gf_lp64,mkl_gnu_thread,mkl_core,iomp5 not found in /opt/intel/Compiler/11.0/074/lib/intel64 NOT AVAILABLE
After some debugging of the problem, it seems that distutils needs to find *all* the required libraries in *one* single directory.
Yes
My question is, is there an elegant way to handle this problem, or it is a limitation of the current distutils?
The "elegant" solution is to make softlink on unix.
If the later, it would be nice it that could be solved in a future version, and several libraries can be found in *several* directories.
Unfortunately, this would mean rewriting system_info, because this assumption is deeply ingrained in the whole design. Personally, I think the really idea of looking for files for libraries is not the right one - all other tools I know (autoconf, scons, jam, cmake) link code snippet instead. But doing it in distutils won't be fun. David
participants (3)
-
David Cournapeau
-
Francesc Alted
-
Robert Kern