Numpy 1.7.0 with Intel MKL 11.0.2.146
Dear all, I'm at my wits end. I've followed Intel's own instructions<http://software.intel.com/en-us/articles/numpyscipy-with-intel-mkl>on how to compile Numpy with Intel MKL. Everything compiled and linked fine and I've installed it locally in my user folder...There is one nasty problem. When one calls the numpy library to do some computation, it does not use all of the available threads. I have 8 "cores" on my machine and it only uses 4 of them. The MKL_NUM_THREADS environmental variable can be set to tune the number of threads but setting it to 8 does not change anything. Indeed, setting it to 3 does limit the threads to 3....What is going on? As a comparison, the numpy (version 1.4.1, installed from yum, which uses BLAS+ATLAS) uses all 8 threads. I do not get this. You can run this test program python -mtimeit -s'import numpy as np; a = np.random.randn(1e3,1e3)' 'np.dot(a, a)' There is one saving grace, the local numpy built with MKL is much faster than the system's numpy. I hope someone can help me. Searching the internet has been fruitless. Best, Quyen My site.cfg for numpy (1.7.0) [mkl] library_dirs = /opt/intel/mkl/lib/intel64 include_dirs = /opt/intel/mkl/include mkl_libs = mkl_rt lapack_libs = I've edited line 37 of numpy/distutils/intelcompiler.py self.cc_exe = 'icc -O3 -fPIC -fp-model strict -fomit-frame-pointer -openmp -parallel -DMKL_ILP64' Also line 54 of numpy/distutils/fcompiler/intel.py return ['-i8 -xhost -openmp -fp-model strict'] My .bash_profile also contains the lines: source /opt/intel/bin/compilervars.sh intel64 source /opt/intel/mkl/bin/mklvars.sh intel64 The above is needed to set the LD_LIBRARY_PATH so that Python can source the intel dynamic library when numpy is called.
On 3/10/13, QT <rdirective@gmail.com> wrote:
Dear all,
I'm at my wits end. I've followed Intel's own instructions<http://software.intel.com/en-us/articles/numpyscipy-with-intel-mkl>on how to compile Numpy with Intel MKL. Everything compiled and linked fine and I've installed it locally in my user folder...There is one nasty problem. When one calls the numpy library to do some computation, it does not use all of the available threads. I have 8 "cores" on my machine and it only uses 4 of them. The MKL_NUM_THREADS environmental variable can be set to tune the number of threads but setting it to 8 does not change anything. Indeed, setting it to 3 does limit the threads to 3....What is going on?
Does your computer have 8 physical cores, or 4 cores that look like 8 because of hyperthreading? Warren
As a comparison, the numpy (version 1.4.1, installed from yum, which uses BLAS+ATLAS) uses all 8 threads. I do not get this.
You can run this test program
python -mtimeit -s'import numpy as np; a = np.random.randn(1e3,1e3)' 'np.dot(a, a)'
There is one saving grace, the local numpy built with MKL is much faster than the system's numpy.
I hope someone can help me. Searching the internet has been fruitless.
Best, Quyen
My site.cfg for numpy (1.7.0) [mkl] library_dirs = /opt/intel/mkl/lib/intel64 include_dirs = /opt/intel/mkl/include mkl_libs = mkl_rt lapack_libs =
I've edited line 37 of numpy/distutils/intelcompiler.py self.cc_exe = 'icc -O3 -fPIC -fp-model strict -fomit-frame-pointer -openmp -parallel -DMKL_ILP64'
Also line 54 of numpy/distutils/fcompiler/intel.py return ['-i8 -xhost -openmp -fp-model strict']
My .bash_profile also contains the lines: source /opt/intel/bin/compilervars.sh intel64 source /opt/intel/mkl/bin/mklvars.sh intel64
The above is needed to set the LD_LIBRARY_PATH so that Python can source the intel dynamic library when numpy is called.
On 3/10/13, Warren Weckesser <warren.weckesser@gmail.com> wrote:
On 3/10/13, QT <rdirective@gmail.com> wrote:
Dear all,
I'm at my wits end. I've followed Intel's own instructions<http://software.intel.com/en-us/articles/numpyscipy-with-intel-mkl>on how to compile Numpy with Intel MKL. Everything compiled and linked fine and I've installed it locally in my user folder...There is one nasty problem. When one calls the numpy library to do some computation, it does not use all of the available threads. I have 8 "cores" on my machine and it only uses 4 of them. The MKL_NUM_THREADS environmental variable can be set to tune the number of threads but setting it to 8 does not change anything. Indeed, setting it to 3 does limit the threads to 3....What is going on?
Does your computer have 8 physical cores, or 4 cores that look like 8 because of hyperthreading?
Here's why I ask this: http://software.intel.com/en-us/forums/topic/294954
Warren
As a comparison, the numpy (version 1.4.1, installed from yum, which uses BLAS+ATLAS) uses all 8 threads. I do not get this.
You can run this test program
python -mtimeit -s'import numpy as np; a = np.random.randn(1e3,1e3)' 'np.dot(a, a)'
There is one saving grace, the local numpy built with MKL is much faster than the system's numpy.
I hope someone can help me. Searching the internet has been fruitless.
Best, Quyen
My site.cfg for numpy (1.7.0) [mkl] library_dirs = /opt/intel/mkl/lib/intel64 include_dirs = /opt/intel/mkl/include mkl_libs = mkl_rt lapack_libs =
I've edited line 37 of numpy/distutils/intelcompiler.py self.cc_exe = 'icc -O3 -fPIC -fp-model strict -fomit-frame-pointer -openmp -parallel -DMKL_ILP64'
Also line 54 of numpy/distutils/fcompiler/intel.py return ['-i8 -xhost -openmp -fp-model strict']
My .bash_profile also contains the lines: source /opt/intel/bin/compilervars.sh intel64 source /opt/intel/mkl/bin/mklvars.sh intel64
The above is needed to set the LD_LIBRARY_PATH so that Python can source the intel dynamic library when numpy is called.
Dear Warren, It's an Intel i7 950, 4 cores, 8 with hyper-threading. I used MKL 11.0.2.146, but I will read your link. It seems spot on. Best, Quyen On Sun, Mar 10, 2013 at 10:31 PM, Warren Weckesser < warren.weckesser@gmail.com> wrote:
On 3/10/13, Warren Weckesser <warren.weckesser@gmail.com> wrote:
On 3/10/13, QT <rdirective@gmail.com> wrote:
Dear all,
I'm at my wits end. I've followed Intel's own instructions< http://software.intel.com/en-us/articles/numpyscipy-with-intel-mkl>on how to compile Numpy with Intel MKL. Everything compiled and linked fine and I've installed it locally in my user folder...There is one nasty problem. When one calls the numpy library to do some computation, it does not use all of the available threads. I have 8 "cores" on my machine and it only uses 4 of them. The MKL_NUM_THREADS environmental variable can be set to tune the number of threads but setting it to 8 does not change anything. Indeed, setting it to 3 does limit the threads to 3....What is going on?
Does your computer have 8 physical cores, or 4 cores that look like 8 because of hyperthreading?
Here's why I ask this: http://software.intel.com/en-us/forums/topic/294954
Warren
As a comparison, the numpy (version 1.4.1, installed from yum, which
uses
BLAS+ATLAS) uses all 8 threads. I do not get this.
You can run this test program
python -mtimeit -s'import numpy as np; a = np.random.randn(1e3,1e3)' 'np.dot(a, a)'
There is one saving grace, the local numpy built with MKL is much faster than the system's numpy.
I hope someone can help me. Searching the internet has been fruitless.
Best, Quyen
My site.cfg for numpy (1.7.0) [mkl] library_dirs = /opt/intel/mkl/lib/intel64 include_dirs = /opt/intel/mkl/include mkl_libs = mkl_rt lapack_libs =
I've edited line 37 of numpy/distutils/intelcompiler.py self.cc_exe = 'icc -O3 -fPIC -fp-model strict -fomit-frame-pointer -openmp -parallel -DMKL_ILP64'
Also line 54 of numpy/distutils/fcompiler/intel.py return ['-i8 -xhost -openmp -fp-model strict']
My .bash_profile also contains the lines: source /opt/intel/bin/compilervars.sh intel64 source /opt/intel/mkl/bin/mklvars.sh intel64
The above is needed to set the LD_LIBRARY_PATH so that Python can source the intel dynamic library when numpy is called.
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
participants (2)
-
QT -
Warren Weckesser