Performance degradation when installing Numpy and Python not from packaging system?
Hi list, I have observed slowdown if Numpy and Python are installed manually (i.e. not from packaging system). The slowdown can be significant when problem size is large. I wish someone in this mailing list can give me some hints on this issue. I also post performance results and my source code here. My platform is a 12-core dual-socket SMP machine running Ubuntu (Linux kernel 3.2.0). On this platform, I have two sets of installations. One is pre-installed Python 2.7.3 and Numpy 1.6.1 installed with 'apt-get'. The other is Python 2.7.3 and Numpy 1.6.1 both compiled freshly from official tarball release by myself. In order to compare performance, I wrote a piece of code that simply performs matrix-matrix multiplication using one of Numpy's functions. The source code is listed as follows: *if __name__ == "__main__": import numpy, random, time import sys size = int(sys.argv[1]) # For simplicity, we only test square matrix matrix_a = numpy.matrix(numpy.random.randn(size,size)) matrix_b = numpy.matrix(numpy.random.randn(size,size)) start_time = time.time() result = numpy.dot(matrix_a, matrix_b) print '%0.3f ms' % ((time.time() - start_time)*1000.0)* I ran this code with both pre-installed Python and compiled Python. Here is performance numbers I got: Problem size Pre-installed Python Compiled Python/Numpy Slowdown 256x256 14.165 ms 44.538 ms ~3X 512x512 109.330 ms 347.143 ms ~3X 1024x1024 867.936 ms 8975.016 ms ~10X 2048x2048 7755.084 ms 84713.997 ms ~10X Here are commands I used to compile Python and Numpy: a) ~/python2.7.3$ ./configure --prefix=MY_LOCAL_DIR b) ~/python2.7.3$ make c) ~/python2.7.3$ make install e) ~/numpy-1.6.1$ MY_LOCAL_DIR/bin/python setup.py build f) ~/numpy-1.6.1$ MY_LOCAL_DIR/bin/python setup.py install Any ideas? Thanks in advance. Bin Huang
Hi, 08.05.2013 20:47, Bin Huang kirjoitti: [clip]
In order to compare performance, I wrote a piece of code that simply performs matrix-matrix multiplication using one of Numpy's functions. The source code is listed as follows: [clip]
Your benchmark essentially measures the speed of the BLAS library Numpy is linked with. It appears that the one supplied by the system is linked with a high-performance BLAS, whereas the one you compiled yourself is not. You should look at the output of the build command, and look for lines such as -------8<------------- lapack_opt_info: lapack_mkl_info: mkl_info: libraries mkl,vml,guide not found in ['/usr/local/lib', '/usr/lib', '/usr/lib/x86_64-linux-gnu'] NOT AVAILABLE NOT AVAILABLE atlas_threads_info: Setting PTATLAS=ATLAS libraries ptf77blas,ptcblas,atlas not found in /usr/local/lib libraries lapack_atlas not found in /usr/local/lib <class 'numpy.distutils.system_info.atlas_threads_info'> Setting PTATLAS=ATLAS Setting PTATLAS=ATLAS FOUND: language = f77 include_dirs = ['/usr/include/atlas'] library_dirs = ['/usr/lib/atlas-base/atlas', '/usr/lib/atlas-base'] libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas'] define_macros = [('ATLAS_INFO', '"\\"3.8.4\\""')] FOUND: language = f77 include_dirs = ['/usr/include/atlas'] library_dirs = ['/usr/lib/atlas-base/atlas', '/usr/lib/atlas-base'] libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas'] define_macros = [('ATLAS_INFO', '"\\"3.8.4\\""')] -------8<------------- If it says "NOT AVAILABLE" to all the cases, then it does not find a good BLAS/LAPACK library, and uses one that is not optimized for specific processors. See here on how to tell it to use a specific BLAS/LAPACK combination: http://new.scipy.org/scipylib/building/linux.html#generic-instructions -- Pauli Virtanen
participants (2)
-
Bin Huang -
Pauli Virtanen