building NumPy with Intel CC & MKL
I'm trying to build NumPy on an SUSE 10.2 Core 2 Duo system using the Intel compiler and Math Kernel Library, and Python 2.5 I want to see how much faster it is than the SUSE NumPy 1.0.1 rpm that recently became available (huge thanks to whomever it was that created it. Installing NumPy and SciPy has been a lot of work on SUSE systems in the past. Half the time I'd give up in frustration after hours of work with each new distribution, and wait until the next SUSE release and try again). I followed the instructions that are here: http://www.scipy.org/Installing_SciPy/Linux Any distribution with Intel C compiler and MKL I changed the line: cc_exe = 'icc -O2 -g -fomit-frame-pointer -mcpu=pentium4 -mtune=pentium4 -march=pentium4 -msse2 -axWN -Wall' to: cc_exe = 'icc -O2 -g -fomit-frame-pointer -mcpu=pentium4 -mtune=pentium4 -march=pentium4 -msse3 -xP -Wall' But the NumPy compile failed: c2d0:/usr/local/src/numpy-1.0.1 # python setup.py config --compiler=intel build_clib --compiler=intel build_ext --compiler=intel install Running from numpy source directory. Traceback (most recent call last): File "setup.py", line 89, in <module> setup_package() File "setup.py", line 59, in setup_package from numpy.distutils.core import setup File "/usr/local/src/numpy-1.0.1/numpy/distutils/core.py", line 24, in <module> from numpy.distutils.command import build_ext File "/usr/local/src/numpy-1.0.1/numpy/distutils/command/build_ext.py", line 16, in <module> from numpy.distutils.system_info import combine_paths File "/usr/local/src/numpy-1.0.1/numpy/distutils/system_info.py", line 159, in <module> so_ext = distutils.sysconfig.get_config_vars('SO')[0] or '' File "/usr/lib/python2.5/distutils/sysconfig.py", line 493, in get_config_vars func() File "/usr/lib/python2.5/distutils/sysconfig.py", line 352, in _init_posix raise DistutilsPlatformError(my_msg) distutils.errors.DistutilsPlatformError: invalid Python installation: unable to open /usr/lib/python2.5/config/Makefile (No such file or directory) There is no config directory in /usr/lib/python2.5/ These are the subdirectories: /bsddb /compiler /ctypes /disutils /email /encodings /hotshot /lib-dynload /plat-linux2 /site-packages /sqlite3 /wsgiref /xml The only Makefiles under Python2.5 are: ./site-packages/numpy/doc/swig/Makefile ./site-packages/numpy/doc/pyrex/Makefile Help anyone? I hope I don't have to build Python2.5 from source to build NumPy. Thanks, -rex
rex wrote:
distutils.errors.DistutilsPlatformError: invalid Python installation: unable to open /usr/lib/python2.5/config/Makefile (No such file or directory)
There is no config directory in /usr/lib/python2.5/
You need to install the development package for Python. Usually it's named something like python2.5-devel. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco
Robert Kern <robert.kern@gmail.com> [2007-01-23 22:18]:
You need to install the development package for Python. Usually it's named something like python2.5-devel.
Thank you. Did that, and NumPy compiled with a Brazillion warnings, but no errors. Then I did: export LD_RUN_PATH=/opt/lib:/opt/intel/cc/9.1.042/lib:/opt/intel/mkl/8.1/lib/32 (because I used the Intel defaults, and those are the correct paths) But since the SUSE NumPy rpm is also installed, how do I determine which version is loaded when the command: from numpy import * is issued in python? Subjectively, it appears the new version is not being used. I expect a significant speed difference using the Intel compiler and MKL on a Core 2 Duo. Why is this so difficult? Thanks, -rex
rex wrote:
Robert Kern <robert.kern@gmail.com> [2007-01-23 22:18]:
You need to install the development package for Python. Usually it's named something like python2.5-devel.
Thank you. Did that, and NumPy compiled with a Brazillion warnings, but no errors.
Then I did: export LD_RUN_PATH=/opt/lib:/opt/intel/cc/9.1.042/lib:/opt/intel/mkl/8.1/lib/32 (because I used the Intel defaults, and those are the correct paths)
But since the SUSE NumPy rpm is also installed, how do I determine which version is loaded when the command: from numpy import * is issued in python? Subjectively, it appears the new version is not being used. I expect a significant speed difference using the Intel compiler and MKL on a Core 2 Duo.
Why is this so difficult?
It is somewhat difficult to do something somewhat complicated :) In your case, one solution to set the dir where numpy is installed is to use the env variable PYTHONPATH. For example, I don't have root privileges on my workstation, and I installed everything in $HOME/local: python setup.py install --prefix=$HOME/local. Then, numpy is installed in $home/local/lib/python2.4/site-packages, and using this dir as a value for PYTHONPATH works (take care at the python version, replace python2.4 by the version you use) To check which numpy you use, you can simply do a import numpy; print numpy, which should print the full path, cheers, David
David Cournapeau <david@ar.media.kyoto-u.ac.jp> [2007-01-23 23:40]:
rex wrote:
Robert Kern <robert.kern@gmail.com> [2007-01-23 22:18]:
You need to install the development package for Python. Usually it's named something like python2.5-devel.
Thank you. Did that, and NumPy compiled with a Brazillion warnings, but no errors.
Then I did: export LD_RUN_PATH=/opt/lib:/opt/intel/cc/9.1.042/lib:/opt/intel/mkl/8.1/lib/32 (because I used the Intel defaults, and those are the correct paths)
But since the SUSE NumPy rpm is also installed, how do I determine which version is loaded when the command: from numpy import * is issued in python? Subjectively, it appears the new version is not being used. I expect a significant speed difference using the Intel compiler and MKL on a Core 2 Duo.
Why is this so difficult?
It is somewhat difficult to do something somewhat complicated :) In your case, one solution to set the dir where numpy is installed is to use the env variable PYTHONPATH.
But installing software is typically easy for Windows users. IMO, the difficulty of installing Linux applications is a huge barrier to wider adoption of Linux. I started trying Linux in 1994, and stopped using Windows entirely in 1999. I'm old (66), and becoming dumber as my brain shrinks, but I'm still reasonably sharp (or so I like to think). Over the years, NumPy and SciPy have been very difficult to install for me using SUSE (the SUSE developers have different ideas of what paths should be from most of the rest of the world. If I were King I'd lock 'em all in a room and tell them that if they could not agree on a directory structure for Linux in 48 hours, they would all be killed. Impending death tends to focus attention on the problem.) Two of my two closest friends have advanced degrees. One is a PhD in orbital mechanics from UCSD, and the other did everything for a PhD in computer science but complete his thesis. Both have tried Linux repeatedly, but found it to require more effort to install and maintain than they are willing to expend. If people of this caliber are repelled by Linux, I think developers need to wake up and smell the coffee. When people with PhDs in science are turned off by the difficulty the problem needs to be addressed.
To check which numpy you use, you can simply do a import numpy; print numpy, which should print the full path,
import numpy print numpy <module 'numpy' from '/usr/lib/python2.5/site-packages/numpy/__init__.pyc'>
What am I to make of this? Is it the rpm numpy or is it the numpy I built using the Intel compiler and MKL? Thanks for the reply, but I'm still confused. -rex -- "When asked, as I frequently am, why I should concern myself so deeply with the conservation of animal life, I reply that I have been very lucky and that throughout my life the world has given me the most enormous pleasure. But the world is as delicate and as complicated as a spider's web. If you touch one thread you send shudders running through all the other threads. We are not just touching the web we are tearing great holes in it." Gerald Durrell 1925-1995.
rex wrote:
David Cournapeau <david@ar.media.kyoto-u.ac.jp> [2007-01-23 23:40]:
rex wrote:
Robert Kern <robert.kern@gmail.com> [2007-01-23 22:18]:
You need to install the development package for Python. Usually it's named something like python2.5-devel.
Thank you. Did that, and NumPy compiled with a Brazillion warnings, but no errors.
Then I did: export LD_RUN_PATH=/opt/lib:/opt/intel/cc/9.1.042/lib:/opt/intel/mkl/8.1/lib/32 (because I used the Intel defaults, and those are the correct paths)
But since the SUSE NumPy rpm is also installed, how do I determine which version is loaded when the command: from numpy import * is issued in python? Subjectively, it appears the new version is not being used. I expect a significant speed difference using the Intel compiler and MKL on a Core 2 Duo.
Why is this so difficult?
It is somewhat difficult to do something somewhat complicated :) In your case, one solution to set the dir where numpy is installed is to use the env variable PYTHONPATH.
But installing software is typically easy for Windows users. IMO, the difficulty of installing Linux applications is a huge barrier to wider adoption of Linux. I started trying Linux in 1994, and stopped using Windows entirely in 1999. I'm old (66), and becoming dumber as my brain shrinks, but I'm still reasonably sharp (or so I like to think). Over the years, NumPy and SciPy have been very difficult to install for me using SUSE (the SUSE developers have different ideas of what paths should be from most of the rest of the world. If I were King I'd lock 'em all in a room and tell them that if they could not agree on a directory structure for Linux in 48 hours, they would all be killed. Impending death tends to focus attention on the problem.)
Two of my two closest friends have advanced degrees. One is a PhD in orbital mechanics from UCSD, and the other did everything for a PhD in computer science but complete his thesis. Both have tried Linux repeatedly, but found it to require more effort to install and maintain than they are willing to expend. If people of this caliber are repelled by Linux, I think developers need to wake up and smell the coffee. When people with PhDs in science are turned off by the difficulty the problem needs to be addressed.
To check which numpy you use, you can simply do a import numpy; print numpy, which should print the full path,
import numpy print numpy
<module 'numpy' from '/usr/lib/python2.5/site-packages/numpy/__init__.pyc'>
What am I to make of this? Is it the rpm numpy or is it the numpy I built using the Intel compiler and MKL?
Thanks for the reply, but I'm still confused.
-rex
Try import scipy print scipy.__version__ import numpy print numpy.__version__ Also very helpful is the output of numpy.show_config() scipy.show_config() Here I get
import scipy print scipy.__version__ 0.5.3.dev2602
Nils
rex wrote:
import numpy print numpy <module 'numpy' from '/usr/lib/python2.5/site-packages/numpy/__init__.pyc'>
What am I to make of this? Is it the rpm numpy or is it the numpy I built using the Intel compiler and MKL?
Did you change the distutils installation location? See this page for the various ways to do that: http://docs.python.org/inst/alt-install-windows.html If not, then I strongly suspect that you overwrote the numpy that you installed by RPM. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco
Robert Kern <robert.kern@gmail.com> [2007-01-24 03:22]:
rex wrote:
import numpy print numpy <module 'numpy' from '/usr/lib/python2.5/site-packages/numpy/__init__.pyc'>
What am I to make of this? Is it the rpm numpy or is it the numpy I built using the Intel compiler and MKL?
Did you change the distutils installation location? See this page for the various ways to do that:
http://docs.python.org/inst/alt-install-windows.html
If not, then I strongly suspect that you overwrote the numpy that you installed by RPM.
Thanks for the URL. It would not have occurred to me to read "alt-install-windows" since I don't do Windows, and haven't since 1999. I did not change the distutils installation location. How do I determine what is loaded when "import numpy" is issued? That is, how does one distinguish between the RPM and the NumPy built using the Intel compiler and MKL? I suppose I can build NumPy under my user directory and reinstall the RPM, and then run some benchmarks. This is exceedingly frustrating; it's not new -- numpy/scipy are very important to me, but they have been very difficult to install under SUSE for years. I'm very grateful for the selfless contributions of those who have created numpy/scipy, but if I -- a highest honors grad -- have great difficulty making it work, what about others? -rex
Dear rex, I'll try to explain... I hope it's not too basic. Python is searching for its modules along the PYTHONPATH, i.e. a list of directories where it expects to find whatever it needs. This is the same as the Unix shell (or the DOC command.com) is looking in the PATH in order to find programs or shell /batch scripts, or the dynamic loader is using LD_LIBRARY_PATH for finding shared libraries.
import numpy print numpy <module 'numpy' from '/usr/lib/python2.5/site-packages/numpy/__init__.pyc'>
What am I to make of this? Is it the rpm numpy or is it the numpy I built using the Intel compiler and MKL?
This tells from which directory your Python installation actually loaded numpy from: It used the numpy installed in the directory /usr/lib/python2.5/site-packages/numpy By *convention* (as someone already pointed out before), the /usr/lib/python2.5/site-packages is the directory where the original system versions of python packages should be installed. In particular, the rpm version will very likely install it's stuff there. When installing additional python modules or packages via a command like python setup.py install the new packages will also be installed in that system directory. So if you have installed your Intel version of numpy with the above command, you might have overwritten the rpm stuff. There is a way to install in a different place; more on that below. You now probably want to find out if the numpy version in /usr/lib/... is the Intel one or the original rpm one. To do this, you can check if the MKL and Intel libraries are actually loaded by the shared libraries within the numpy installation. You can use the command ldd which shows which shared libraries are loaded by executables or other shared libraries. For example, in my installation, the command ldd <wherever>/python2.5/site-packages/numpy/linalg/lapack_lite.so gives the following output: MEDEA /opt/apps/lib/python2.5/site-packages/numpy>ldd ./linalg/lapack_lite.so linux-gate.so.1 => (0xffffe000) libmkl_lapack32.so => /opt/intel/mkl/8.1/lib/32/libmkl_lapack32.so (0x40124000) libmkl_lapack64.so => /opt/intel/mkl/8.1/lib/32/libmkl_lapack64.so (0x403c8000) libmkl.so => /opt/intel/mkl/8.1/lib/32/libmkl.so (0x40692000) libvml.so => /opt/intel/mkl/8.1/lib/32/libvml.so (0x406f3000) libguide.so => /opt/intel/mkl/8.1/lib/32/libguide.so (0x4072c000) libpthread.so.0 => /lib/tls/libpthread.so.0 (0x40785000) libimf.so => /opt/intel/fc/9.1/lib/libimf.so (0x40797000) libm.so.6 => /lib/tls/libm.so.6 (0x409d5000) libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x409f8000) libirc.so => /opt/intel/fc/9.1/lib/libirc.so (0x40a00000) libc.so.6 => /lib/tls/libc.so.6 (0x40a41000) libdl.so.2 => /lib/libdl.so.2 (0x40b5b000) /lib/ld-linux.so.2 (0x80000000) Note that the MKL libraries are referenced at the beginning - just look at the path names! If the output for your lapack_lite.so also contains references to the MKL libs, you've got the Intel version in /usr/lib/python2.5/.... (and have probably overwritten the rpm version). If you do not get any reference to the MKL stuff, it's still the rpm version which does not use the MKL. Now, let's assume that you have the rpm version in /usr/lib/python2.5/.... Maybe you'll want to reinstall the rpm to be sure that this is the case. You now want to a) install your Intel version in some well-defined place, and b) make sure that your Python picks that version up when importing numpy. To achieve a) one way is to reinstall numpy from the source as before, BUT with python setup.py --prefix=<somewhere> ^^^^^^^^^^^^^^^^^^^^^ <somewhere> is the path to some directory, e.g. python setup.py install --prefix=$HOME The latter would install numpy into the directory $HOME/lib/python2.5/site-packages/numpy Do an ls afterwards to check if numpy really arrived there. Instead of using the environment variable HOME, you can of course also any other directory you like. I'll stick to HOME in the following. For b), we have to tell python that modules are waiting for it to be picked up in $HOME/lib/python2.5/site-packages. You do that by setting the environment variable PYTHONPATH, as was also mentioned in this thread. In our example, you would do (for a bash or ksh) export PYTHONPATH=$HOME/lib/python2.5/site-packages As long as this variable is set and exported (i.e., visible in the environment of every program you start), the next instance of Python you'll start will now begin searching for modules in PYTHONPATH whenever you do an import, and only fall back to the ones in the system wide installation if it doesn't find the required module in PYTHONPATH. So, after having set PYTHONPATH in your environment, start up python and import numpy. Do the 'print numpy' within python again and look at the output. Does it point to the installation directory of your Intel version? Great; you're done. If not, this means that something went wrong. It might be that you had a typo in the export command or the directory name; it might mean that you didn't export the PYTHONPATH before running python; it might be that the installation had failed for some reason. You just have to play around a bit and see what's going on... but it's not difficult. Now that you have two versions of numpy, you can (kind of) switch between them by making use of the PYTHONPATH. If you unset it ('unset PYTHONPATH'), the next python session you are starting in the same shell/window will use the original system version. Setting PYTHONPATH again and having it point to your local site-packages directory activates the stuff you've installed in there. You cannot switch between the two numpy versions in the same session); if you want to try the other, you'll have to start a new python and make sure that the PYTHONPATH is set up appropriately for what you want. In the long run, and if you have decided which version to use, you can export PYTHONPATH in your $HOME/.profile and don't have to do that manually each time (which becomes quite cumbersome after a while, of course). Common practice is probably that you install your favourite versions or builds of python modules in one place (i.e. using $HOME as --prefix), and set PYTHONPATH accordingly. It's not a good idea to overwrite the system wide installations, but again - that's purely a convention, nothing more. Hope this helps a bit... Good luck! Christian.
Did you change the distutils installation location? See this page for the various ways to do that:
http://docs.python.org/inst/alt-install-windows.html
If not, then I strongly suspect that you overwrote the numpy that you installed by RPM.
Thanks for the URL. It would not have occurred to me to read "alt-install-windows" since I don't do Windows, and haven't since 1999.
I did not change the distutils installation location.
How do I determine what is loaded when "import numpy" is issued? That is, how does one distinguish between the RPM and the NumPy built using the Intel compiler and MKL? I suppose I can build NumPy under my user directory and reinstall the RPM, and then run some benchmarks.
This is exceedingly frustrating; it's not new -- numpy/scipy are very important to me, but they have been very difficult to install under SUSE for years.
I'm very grateful for the selfless contributions of those who have created numpy/scipy, but if I -- a highest honors grad -- have great difficulty making it work, what about others?
-rex _______________________________________________ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
rex wrote:
Thanks for the URL. It would not have occurred to me to read "alt-install-windows" since I don't do Windows, and haven't since 1999.
The name of the file is a quirk of the documentation generation program. The table of contents, though, is quite clear about that chapter's generality. http://docs.python.org/inst/inst.html
This is exceedingly frustrating; it's not new -- numpy/scipy are very important to me, but they have been very difficult to install under SUSE for years.
Yes. I would never inflict SuSE on anyone. I encourage you to try Ubuntu. It is mostly sane and especially so when it comes to Python. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco
rex wrote:
David Cournapeau <david@ar.media.kyoto-u.ac.jp> [2007-01-23 23:40]:
But installing software is typically easy for Windows users. I am not sure installing numpy using MKL library is easier on windows :) IMO, the difficulty of installing Linux applications is a huge barrier to wider adoption of Linux. I started trying Linux in 1994, and stopped using Windows entirely in 1999. I'm old (66), and becoming dumber as my brain shrinks, but I'm still reasonably sharp (or so I like to think). Over the years, NumPy and SciPy have been very difficult to install for me using SUSE (the SUSE developers have different ideas of what paths should be from most of the rest of the world. If I were King I'd lock 'em all in a room and tell them that if they could not agree on a directory structure for Linux in 48 hours, they would all be killed. Impending death tends to focus attention on the problem.)
Two of my two closest friends have advanced degrees. One is a PhD in orbital mechanics from UCSD, and the other did everything for a PhD in computer science but complete his thesis. Both have tried Linux repeatedly, but found it to require more effort to install and maintain than they are willing to expend. If people of this caliber are repelled by Linux, I think developers need to wake up and smell the coffee. When people with PhDs in science are turned off by the difficulty the problem needs to be addressed. My parents both have college degree in science, and my mother has a hard time using the mouse, and she is younger than you :)
You are not trying to install a program, but compiling it with a specific set of libraries; I don't see how this is difficult than windows (I would say it is worse because of the lack of decent command line shell in windows). About numpy/scipy and linux: installing software on linux is not difficult; installing software which is not packaged for your distribution is. Installing from sources is easier on linux than on windows, in my experience, and building binaries which work on linux across distributions extremely difficult (and not a fun thing to do). The real problem of numpy/scipy now is that it is not packaged yet by all major distributions, but this should change soon.
import numpy print numpy <module 'numpy' from '/usr/lib/python2.5/site-packages/numpy/__init__.pyc'>
This means that the numpy you use in python is the system-wide path. Either this is the numpy installed by RPM, or the one you build yourself that you overwrote (not something you should do normally). Assuming a standard set of libraries, one way to build numpy and install it in your home directory (and as such avoiding messing up with the system, and you don't need root rights) - uncompress the numpy tarball and go to the numpy directory - run "python setup.py build" - run "python setup.py install --prefix=$HOME/local" to install everything in a local path. Now, when you run python, you need to tell him to look for python modules in your $HOME/local: PYTHONPATH=$HOME/local/lib/python2.4/site-packages/ python or PYTHONPATH=$HOME/local/lib/python2.5/site-packages/ python Depending on your python version (which you can check with python -V) This is not different on linux than on windows, by the way :) cheers, David
What am I to make of this? Is it the rpm numpy or is it the numpy I built using the Intel compiler and MKL?
Thanks for the reply, but I'm still confused.
-rex
participants (5)
-
Christian Marquardt -
David Cournapeau -
Nils Wagner -
rex -
Robert Kern