Re: [Numpy-discussion] Numpy installation problem
On Sat 24/07/10 3:39 PM , David Cournapeau cournape@gmail.com wrote:
On Sun, Jul 25, 2010 at 4:23 AM, Jonathan Tu wrote:
On Jul 24, 2010, at 3:08 PM, Benjamin Root wrote:
On Fri, Jul 23, 2010 at 10:10 AM, Jonathan Tu wrote:
Hi, I am trying to install Numpy on a Linux cluster running RHEL4. I installed a local copy of Python 2.7 because RHEL4 uses Python 2.3.4
for
various internal functionalities. I downloaded the Numpy source code
using
svn co http://svn.scipy.org/svn/numpy/trunk [1] numpy and then I tried to build using python setup.py build This resulted in the following error: gcc: numpy/linalg/lapack_litemodule.c gcc:
numpy/linalg/python_xerbla.c
/usr/bin/g77 -g -Wall -g -Wall -shared build/temp.linux-x86_64-2.7/numpy/linalg/lapack_litemodule.o build/temp.linux-x86_64-2.7/numpy/linalg/python_xerbla.o
-L/usr/lib64/ATLAS
-Lbuild/temp.linux-x86_64-2.7 -llapack -lptf77blas -lptcblas -latlas
-lg2c
-o build/lib.linux-x86_64-2.7/numpy/linalg/lapack_lite.so /usr/bin/ld: /usr/lib64/ATLAS/liblapack.a(dgeev.o): relocation R_X86_64_32 against
`a
local symbol' can not be used when making a shared object; recompile
with
-fPIC /usr/lib64/ATLAS/liblapack.a: could not read symbols: Bad value collect2: ld returned 1 exit status /usr/bin/ld: /usr/lib64/ATLAS/liblapack.a(dgeev.o): relocation R_X86_64_32 against
`a
local symbol' can not be used when making a shared object; recompile
with
-fPIC /usr/lib64/ATLAS/liblapack.a: could not read symbols: Bad value collect2: ld returned 1 exit status error: Command "/usr/bin/g77 -g
-Wall -g
-Wall -shared
build/temp.linux-x86_64-2.7/numpy/linalg/lapack_litemodule.o
build/temp.linux-x86_64-2.7/numpy/linalg/python_xerbla.o
-L/usr/lib64/ATLAS
-Lbuild/temp.linux-x86_64-2.7 -llapack -lptf77blas -lptcblas -latlas
-lg2c
-o build/lib.linux-x86_64-2.7/numpy/linalg/lapack_lite.so" failed with
exit
status 1
Full details of the output are attached in stdout.txt and stderr.txt.
I
thought maybe it was a compiler error so I tried python setup.py build -fcompiler=gnu but this also resulted in errors as well (stdout_2.txt, stderr_2.txt). I just noticed that on both attempts, it is complaining that it can't
find
a Fortran 90 compiler. I'm not sure if I have the right compiler
available.
On this cluster I have the following modules: ------------ /usr/share/Modules/modulefiles ------------ dot module-cvs module-info modules null
use.own
------------ /usr/local/share/Modules/modulefiles ------------ mpich/gcc/1.2.7p1/64 openmpi/gcc-ib/1.2.3/64 mpich/intel/1.2.7dmcrp1/64 openmpi/gcc-ib/1.2.5/64 mpich/intel/1.2.7p1/64 openmpi/intel/1.2.3/64 mpich/pgi-7.1/1.2.7p1/64 openmpi/intel-11.0/1.2.8/64 mpich-debug/gcc/1.2.7p1/64 openmpi/intel-9.1/1.2.8/64 mpich-debug/intel/1.2.7p1/64 openmpi/intel-ib/1.1.5/64 mpich-debug/pgi-7.1/1.2.7p1/64 openmpi/intel-ib/1.2.3/64 mvapich/gcc/0.9.9/64 openmpi/intel-ib/1.2.5/64 mvapich/pgi-7.1/0.9.9/64 openmpi/pgi-7.0/1.2.3/64 openmpi/gcc/1.2.8/64 openmpi/pgi-7.1/1.2.5/64 openmpi/gcc/1.3.0/64 openmpi/pgi-7.1/1.2.8/64 openmpi/gcc-ib/1.1.5/64 openmpi/pgi-8.0/1.2.8/64 ------------ /opt/share/Modules/modulefiles ------------ intel/10.0/64/C/10.0.026 intel/9.1/64/default intel/10.0/64/Fortran/10.0.026 intel-mkl/10/64 intel/10.0/64/Iidb/10.0.026 intel-mkl/10.1/64 intel/10.0/64/default intel-mkl/9/32 intel/11.1/64/11.1.038 intel-mkl/9/64 intel/11.1/64/11.1.072 pgi/7.0/64 intel/9.1/64/C/9.1.045 pgi/7.1/64 intel/9.1/64/Fortran/9.1.040 pgi/8.0/64 intel/9.1/64/Iidb/9.1.045 If anyone has any ideas, they would be greatly appreciated! I am new
to
Linux and am unsure how to fix this problem.
Jonathan Tu
Jonathan,
Looking at your error logs, I suspect that the issue is that the ATLAS libraries that were installed on your system were probably built using
f90,
and then your f77-built objects can't link against ATLAS. Do you have
admin
access to this machine? If possible, try using the yum package
manager to
install f90 (its availability depends on whatever RHEL license you
have,
though...).
If you can install f90, I would then remove the numpy build directory
and
try building it again.
I hope this helps, Ben Root _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion [2]
Hi Ben, I managed to complete my installation. It required setting up the
configure
file to reference my local installation of Python, rather than the
default
/usr/bin/Python. However, I would like to see whether or not I am
using the
optimized ATLAS/LAPACK libraries for my matrix computations.
Run ldd on the concerned extensions: numpy/linalg/lapack_lite.so and numpy/core/_dotblas.so
David
I am unable to find the files lapack_lite.so or _dotblas.so. I used the locate command to look for them. Does this mean my Numpy installation was not build against LAPACK at all? If so, how would I fix this? I'm sorry for the very basic questions, but I am fairly new to Linux and Python.
Jonathan Tu
On Sun, Jul 25, 2010 at 4:50 AM, Jonathan Tu jhtu@princeton.edu wrote:
I am unable to find the files lapack_lite.so or _dotblas.so. I used the locate command to look for them.
If you just installed numpy, locate won't find them as locate uses a database which is usually updated once in a while.
Depending on how you installed numpy, you will find lapack_lite.so in /usr, $HOME/.local or somewhere else (and also in the build directory in the numpy source tree).
_dotblas.so may not be present, but numpy cannot work without a lapack_lite.so.
cheers,
David
On Sat, Jul 24, 2010 at 12:58 PM, David Cournapeau cournape@gmail.com wrote:
On Sun, Jul 25, 2010 at 4:50 AM, Jonathan Tu jhtu@princeton.edu wrote:
I am unable to find the files lapack_lite.so or _dotblas.so. I used the locate command to look for them.
If you just installed numpy, locate won't find them as locate uses a database which is usually updated once in a while.
Depending on how you installed numpy, you will find lapack_lite.so in /usr, $HOME/.local or somewhere else (and also in the build directory in the numpy source tree).
Here's one way to find where numpy is installed:
$ python
import numpy numpy
<module 'numpy' from '/usr/local/lib/python2.6/site-packages/numpy/__init__.pyc'>
So on my computer numpy is installed in /usr/local/lib/python2.6/site-packages
_dotblas.so may not be present, but numpy cannot work without a lapack_lite.so.
On Jul 24, 2010, at 4:04 PM, Keith Goodman wrote:
On Sat, Jul 24, 2010 at 12:58 PM, David Cournapeau cournape@gmail.com wrote:
On Sun, Jul 25, 2010 at 4:50 AM, Jonathan Tu jhtu@princeton.edu wrote:
I am unable to find the files lapack_lite.so or _dotblas.so. I used the locate command to look for them.
If you just installed numpy, locate won't find them as locate uses a database which is usually updated once in a while.
Depending on how you installed numpy, you will find lapack_lite.so in /usr, $HOME/.local or somewhere else (and also in the build directory in the numpy source tree).
Here's one way to find where numpy is installed:
$ python
import numpy numpy
<module 'numpy' from '/usr/local/lib/python2.6/site-packages/numpy/__init__.pyc'>
So on my computer numpy is installed in /usr/local/lib/python2.6/site-packages
_dotblas.so may not be present, but numpy cannot work without a lapack_lite.so.
Thanks for the tip! This worked, and much faster than the manual hunting through directories I'd been trying. Upon running ldd on lapack_lite.so and on _dotblas.so, I got the following (first for lapack_lite, then for _dotblas):
libpthread.so.0 => /lib64/tls/libpthread.so.0 (0x0000002a95792000) libc.so.6 => /lib64/tls/libc.so.6 (0x0000002a958a7000) /lib64/ld-linux-x86-64.so.2 (0x000000552aaaa000)
libpthread.so.0 => /lib64/tls/libpthread.so.0 (0x0000002a95d36000) libc.so.6 => /lib64/tls/libc.so.6 (0x0000002a95e4c000) /lib64/ld-linux-x86-64.so.2 (0x000000552aaaa000)
I am interested in whether my Numpy installation is in fact taking advantage of the LAPACK libraries that are installed on the cluster where I'm running. I know that Numpy can function without such an installation, but I think it is probably faster when taking advantage of optimized libraries. In any case, I know ldd shows library dependencies, but I'm not sure how to make sense of its output, and whether or not it answers my question.
Jonathan Tu
On Sun, Jul 25, 2010 at 5:11 AM, Jonathan Tu jhtu@princeton.edu wrote:
On Jul 24, 2010, at 4:04 PM, Keith Goodman wrote:
On Sat, Jul 24, 2010 at 12:58 PM, David Cournapeau cournape@gmail.com wrote:
On Sun, Jul 25, 2010 at 4:50 AM, Jonathan Tu jhtu@princeton.edu wrote:
I am unable to find the files lapack_lite.so or _dotblas.so. I used the locate command to look for them.
If you just installed numpy, locate won't find them as locate uses a database which is usually updated once in a while.
Depending on how you installed numpy, you will find lapack_lite.so in /usr, $HOME/.local or somewhere else (and also in the build directory in the numpy source tree).
Here's one way to find where numpy is installed:
$ python
import numpy numpy
<module 'numpy' from '/usr/local/lib/python2.6/site-packages/numpy/__init__.pyc'>
So on my computer numpy is installed in /usr/local/lib/python2.6/site-packages
_dotblas.so may not be present, but numpy cannot work without a lapack_lite.so.
Thanks for the tip! This worked, and much faster than the manual hunting through directories I'd been trying. Upon running ldd on lapack_lite.so and on _dotblas.so, I got the following (first for lapack_lite, then for _dotblas):
libpthread.so.0 => /lib64/tls/libpthread.so.0 (0x0000002a95792000) libc.so.6 => /lib64/tls/libc.so.6 (0x0000002a958a7000) /lib64/ld-linux-x86-64.so.2 (0x000000552aaaa000)
libpthread.so.0 => /lib64/tls/libpthread.so.0 (0x0000002a95d36000) libc.so.6 => /lib64/tls/libc.so.6 (0x0000002a95e4c000) /lib64/ld-linux-x86-64.so.2 (0x000000552aaaa000)
I am interested in whether my Numpy installation is in fact taking advantage of the LAPACK libraries that are installed on the cluster where I'm running. I know that Numpy can function without such an installation, but I think it is probably faster when taking advantage of optimized libraries. In any case, I know ldd shows library dependencies, but I'm not sure how to make sense of its output, and whether or not it answers my question.
It shows you don't use either atlas or even simple lapack. If your install used atlas, the atlas libraries would show up on those .so with ldd,
David
On Sun, Jul 25, 2010 at 5:13 AM, David Cournapeau cournape@gmail.com wrote:
On Sun, Jul 25, 2010 at 5:11 AM, Jonathan Tu jhtu@princeton.edu wrote:
On Jul 24, 2010, at 4:04 PM, Keith Goodman wrote:
On Sat, Jul 24, 2010 at 12:58 PM, David Cournapeau cournape@gmail.com wrote:
On Sun, Jul 25, 2010 at 4:50 AM, Jonathan Tu jhtu@princeton.edu wrote:
I am unable to find the files lapack_lite.so or _dotblas.so. I used the locate command to look for them.
If you just installed numpy, locate won't find them as locate uses a database which is usually updated once in a while.
Depending on how you installed numpy, you will find lapack_lite.so in /usr, $HOME/.local or somewhere else (and also in the build directory in the numpy source tree).
Here's one way to find where numpy is installed:
$ python
import numpy numpy
<module 'numpy' from '/usr/local/lib/python2.6/site-packages/numpy/__init__.pyc'>
So on my computer numpy is installed in /usr/local/lib/python2.6/site-packages
_dotblas.so may not be present, but numpy cannot work without a lapack_lite.so.
Thanks for the tip! This worked, and much faster than the manual hunting through directories I'd been trying. Upon running ldd on lapack_lite.so and on _dotblas.so, I got the following (first for lapack_lite, then for _dotblas):
libpthread.so.0 => /lib64/tls/libpthread.so.0 (0x0000002a95792000) libc.so.6 => /lib64/tls/libc.so.6 (0x0000002a958a7000) /lib64/ld-linux-x86-64.so.2 (0x000000552aaaa000)
libpthread.so.0 => /lib64/tls/libpthread.so.0 (0x0000002a95d36000) libc.so.6 => /lib64/tls/libc.so.6 (0x0000002a95e4c000) /lib64/ld-linux-x86-64.so.2 (0x000000552aaaa000)
I am interested in whether my Numpy installation is in fact taking advantage of the LAPACK libraries that are installed on the cluster where I'm running. I know that Numpy can function without such an installation, but I think it is probably faster when taking advantage of optimized libraries. In any case, I know ldd shows library dependencies, but I'm not sure how to make sense of its output, and whether or not it answers my question.
It shows you don't use either atlas or even simple lapack. If your install used atlas, the atlas libraries would show up on those .so with ldd
Here is what I would do for RHEL4: - use python 2.6, not 2.7, because the released numpy does not support 2.7 yet. - build atlas 3.8.3 and lapack 3.1.1 by yourself: this is your core issue here. Both lapack and atlas should be built with the -fPIC flag on x86_64 arch (modify the make.inc in lapack to include -fPIC in OPT and NOOPT flags, and add the option -Fa alg -fPIC when calling atlas configure script)
David
Here is what I would do for RHEL4:
- use python 2.6, not 2.7, because the released numpy does not support 2.7 yet.
- build atlas 3.8.3 and lapack 3.1.1 by yourself: this is your core
issue here. Both lapack and atlas should be built with the -fPIC flag on x86_64 arch (modify the make.inc in lapack to include -fPIC in OPT and NOOPT flags, and add the option -Fa alg -fPIC when calling atlas configure script)
David
I have built lapack and atlas per your instructions (using http://www.scipy.org/Installing_SciPy/Linux for reference). I think this has been done successfully. My first go at building Numpy was not successful, and I was wondering if perhaps I did a few things wrong.
At one point, the website I referenced above suggests that I can make shared libraries by doing cd lib #to where the atlas libraries are make shared # for sequential libraries make ptshared # for threaded libraries
I initially did both of these things, which I'm not sure is ok. The above website also says "our atlas libraries should now be in the lib subdirectory of the current directory. You should copy them to some-place that you can tell site.cfg about." So I copied the libraries using cp to /home/jhtu/local/lib.
The final thing I had to do was modify my site.cfg. I had a few questions about this. First of all, for my default paths, do I need to do library_dirs = /usr/local/bin:/home/jhtu/local/lib or just library_dirs = /home/jhtu/local/lib
Finally, to specifiy the lapack libraries, there is a section like [blas_opt] libraries = f77blas,... but also another choice [blas_opt] libraries=ptf77blas,... I think this is related to the make shared vs make ptshared question above, but this is something I know nothing about. I think these are the only things that could be causing my problems.
Jonathan Tu
On Jul 24, 2010, at 5:46 PM, Jonathan Tu wrote:
Here is what I would do for RHEL4:
- use python 2.6, not 2.7, because the released numpy does not support 2.7 yet.
- build atlas 3.8.3 and lapack 3.1.1 by yourself: this is your core
issue here. Both lapack and atlas should be built with the -fPIC flag on x86_64 arch (modify the make.inc in lapack to include -fPIC in OPT and NOOPT flags, and add the option -Fa alg -fPIC when calling atlas configure script)
David
I have built lapack and atlas per your instructions (using http://www.scipy.org/Installing_SciPy/Linux for reference). I think this has been done successfully. My first go at building Numpy was not successful, and I was wondering if perhaps I did a few things wrong.
At one point, the website I referenced above suggests that I can make shared libraries by doing cd lib #to where the atlas libraries are make shared # for sequential libraries make ptshared # for threaded libraries
I initially did both of these things, which I'm not sure is ok. The above website also says "our atlas libraries should now be in the lib subdirectory of the current directory. You should copy them to some-place that you can tell site.cfg about." So I copied the libraries using cp to /home/jhtu/local/lib.
The final thing I had to do was modify my site.cfg. I had a few questions about this. First of all, for my default paths, do I need to do library_dirs = /usr/local/bin:/home/jhtu/local/lib or just library_dirs = /home/jhtu/local/lib
Finally, to specifiy the lapack libraries, there is a section like [blas_opt] libraries = f77blas,... but also another choice [blas_opt] libraries=ptf77blas,... I think this is related to the make shared vs make ptshared question above, but this is something I know nothing about. I think these are the only things that could be causing my problems.
Also, I recently tried this all again, only doing make shared, and only using the list of libraries without "pt." In both cases, when I ran Python interactively and tried to load the numpy module, I got the following:
import numpy
Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/home/jhtu/local/lib/python2.6/site-packages/numpy/__init__.py", line 132, in <module> import add_newdocs File "/home/jhtu/local/lib/python2.6/site-packages/numpy/add_newdocs.py", line 9, in <module> from lib import add_newdoc File "/home/jhtu/local/lib/python2.6/site-packages/numpy/lib/__init__.py", line 13, in <module> from polynomial import * File "/home/jhtu/local/lib/python2.6/site-packages/numpy/lib/polynomial.py", line 17, in <module> from numpy.linalg import eigvals, lstsq File "/home/jhtu/local/lib/python2.6/site-packages/numpy/linalg/__init__.py", line 47, in <module> from linalg import * File "/home/jhtu/local/lib/python2.6/site-packages/numpy/linalg/linalg.py", line 22, in <module> from numpy.linalg import lapack_lite ImportError: libptf77blas.so: cannot open shared object file: No such file or directory
On Sun, Jul 25, 2010 at 6:50 AM, Jonathan Tu jhtu@princeton.edu wrote:
Also, I recently tried this all again, only doing make shared, and only using the list of libraries without "pt." In both cases, when I ran Python interactively and tried to load the numpy module, I got the following:
import numpy
Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/home/jhtu/local/lib/python2.6/site-packages/numpy/__init__.py", line 132, in <module> import add_newdocs File "/home/jhtu/local/lib/python2.6/site-packages/numpy/add_newdocs.py", line 9, in <module> from lib import add_newdoc File "/home/jhtu/local/lib/python2.6/site-packages/numpy/lib/__init__.py", line 13, in <module> from polynomial import * File "/home/jhtu/local/lib/python2.6/site-packages/numpy/lib/polynomial.py", line 17, in <module> from numpy.linalg import eigvals, lstsq File "/home/jhtu/local/lib/python2.6/site-packages/numpy/linalg/__init__.py", line 47, in <module> from linalg import * File "/home/jhtu/local/lib/python2.6/site-packages/numpy/linalg/linalg.py", line 22, in <module> from numpy.linalg import lapack_lite ImportError: libptf77blas.so: cannot open shared object file: No such file or directory
If you install shared libraries into a directory which is not looked in by default by ld, you should add it to the LD_LIBRARY_PATH, e.g.:
LD_LIBRARY_PATH=$HOME/local/lib:$LD_LIBRARY_PATH python -c "import numpy"
David
If you install shared libraries into a directory which is not looked in by default by ld, you should add it to the LD_LIBRARY_PATH, e.g.:
LD_LIBRARY_PATH=$HOME/local/lib:$LD_LIBRARY_PATH python -c "import numpy"
David
Ok, that makes sense to me. I tried to see what the current value is by doing echo $LD_LIBRARY_PATH but it told me that the this is an "Undefined variable." That seems strange to me. I am running on a RHEL4 cluster with a C-shell. I tried to do setenv LD_LIBRARY_PATH /home/jhtu/local/lib:${LD_LIBRARY_PATH} and of course got the same error about an undefined variable.
I'm not sure what this means.
On Sun, Jul 25, 2010 at 7:00 AM, Jonathan Tu jhtu@princeton.edu wrote:
If you install shared libraries into a directory which is not looked in by default by ld, you should add it to the LD_LIBRARY_PATH, e.g.:
LD_LIBRARY_PATH=$HOME/local/lib:$LD_LIBRARY_PATH python -c "import numpy"
David
Ok, that makes sense to me. I tried to see what the current value is by doing echo $LD_LIBRARY_PATH
It is very possible that you don't have it defined by default on your platform. The undefined variable may be a peculiarity of the cshell - the instructions I gave you assumed a bourne shell or compatible.
David
On Jul 24, 2010, at 6:09 PM, David Cournapeau wrote:
On Sun, Jul 25, 2010 at 7:00 AM, Jonathan Tu jhtu@princeton.edu wrote:
If you install shared libraries into a directory which is not looked in by default by ld, you should add it to the LD_LIBRARY_PATH, e.g.:
LD_LIBRARY_PATH=$HOME/local/lib:$LD_LIBRARY_PATH python -c "import numpy"
David
Ok, that makes sense to me. I tried to see what the current value is by doing echo $LD_LIBRARY_PATH
It is very possible that you don't have it defined by default on your platform. The undefined variable may be a peculiarity of the cshell - the instructions I gave you assumed a bourne shell or compatible.
David
Ok, I assumed as much and proceeded by simply setting the environment variable. I did not add the part where you said python -c "import numpy"
What does that part do? It turns out that by fixing my library path, numpy now imports. I assume that my installation of h5py doesn't need to be redone just because I reinstalled Numpy?
Jonathan Tu
On Sat, Jul 24, 2010 at 3:12 PM, Jonathan Tu jhtu@princeton.edu wrote:
On Jul 24, 2010, at 6:09 PM, David Cournapeau wrote:
On Sun, Jul 25, 2010 at 7:00 AM, Jonathan Tu jhtu@princeton.edu wrote:
If you install shared libraries into a directory which is not looked in by default by ld, you should add it to the LD_LIBRARY_PATH, e.g.:
LD_LIBRARY_PATH=$HOME/local/lib:$LD_LIBRARY_PATH python -c "import numpy"
David
Ok, that makes sense to me. I tried to see what the current value is by doing echo $LD_LIBRARY_PATH
It is very possible that you don't have it defined by default on your platform. The undefined variable may be a peculiarity of the cshell - the instructions I gave you assumed a bourne shell or compatible.
David
Ok, I assumed as much and proceeded by simply setting the environment variable. I did not add the part where you said python -c "import numpy"
What does that part do? It turns out that by fixing my library path, numpy now imports. I assume that my installation of h5py doesn't need to be redone just because I reinstalled Numpy?
Try:
$ python
import h5py.tests h5py.tests.runtests()
....................................................................................................................................................................................................................................................... ---------------------------------------------------------------------- Ran 247 tests in 1.180s
OK <unittest._TextTestResult run=247 errors=0 failures=0>
On Sun, Jul 25, 2010 at 7:12 AM, Jonathan Tu jhtu@princeton.edu wrote:
What does that part do? It turns out that by fixing my library path, numpy now imports. I assume that my installation of h5py doesn't need to be redone just because I reinstalled Numpy?
Actually, you may have to if you installed h5py against the numpy built from the trunk. I don't think the trunk ABI is forward compatible with 1.4.1,
David
On Jul 24, 2010, at 6:21 PM, David Cournapeau wrote:
On Sun, Jul 25, 2010 at 7:12 AM, Jonathan Tu jhtu@princeton.edu wrote:
What does that part do? It turns out that by fixing my library path, numpy now imports. I assume that my installation of h5py doesn't need to be redone just because I reinstalled Numpy?
Actually, you may have to if you installed h5py against the numpy built from the trunk. I don't think the trunk ABI is forward compatible with 1.4.1,
David
Ok, so I reinstalled numpy as we've discussed, and now I've reinstalled h5py too. When I test numpy, I get the following
import numpy numpy.test() Running unit tests for numpy Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/home/jhtu/local/lib/python2.6/site-packages/numpy/testing/nosetester.py", line 326, in test self._show_system_info() File "/home/jhtu/local/lib/python2.6/site-packages/numpy/testing/nosetester.py", line 187, in _show_system_info nose = import_nose() File "/home/jhtu/local/lib/python2.6/site-packages/numpy/testing/nosetester.py", line 69, in import_nose raise ImportError(msg) ImportError: Need nose >= 0.10.0 for tests - see http://somethingaboutorange.com/mrl/projects/nose
Note that when I try test(level=1) it tells me this is an unexpected keyword argument. (The level=1 was suggested by the scipy website.) Though this test seems to fail, the simple math tests I've tried seem to work ok.
Then I tried to test h5py: import h5py import h5py.tests h5py.tests.runtests()
====================================================================== ERROR: test_data (h5py.tests.test_filters.TestFilters) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/jhtu/local/lib/python2.6/site-packages/h5py/tests/test_filters.py", line 124, in test_data test_dset(s, t, compression=c, shuffle=False) File "/home/jhtu/local/lib/python2.6/site-packages/h5py/tests/test_filters.py", line 117, in test_dset dset[...] = arr File "/home/jhtu/local/lib/python2.6/site-packages/h5py/highlevel.py", line 1288, in __setitem__ self.id.write(mspace, fspace, val, mtype) File "h5d.pyx", line 231, in h5py.h5d.DatasetID.write (h5py/h5d.c:2336) File "_proxy.pyx", line 115, in h5py._proxy.dset_rw (h5py/_proxy.c:1143) TypeError: Not a dataset (Invalid arguments to routine: Inappropriate type)
====================================================================== FAIL: (File) Backing store for core driver ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/jhtu/local/lib/python2.6/site-packages/h5py/tests/high/test_file.py", line 135, in test_backing self.assert_('g' in self.f) AssertionError
====================================================================== FAIL: (File) Append ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/jhtu/local/lib/python2.6/site-packages/h5py/tests/high/test_file.py", line 52, in test_a self.assert_('g' in self.f) AssertionError
====================================================================== FAIL: (File) Readwrite ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/jhtu/local/lib/python2.6/site-packages/h5py/tests/high/test_file.py", line 72, in test_rp self.assert_('g' in self.f) AssertionError
====================================================================== FAIL: test_open_close (h5py.tests.test_h5g.TestH5G) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/jhtu/local/lib/python2.6/site-packages/h5py/tests/test_h5g.py", line 39, in test_open_close self.assert_(self.is_grp(grp)) AssertionError
---------------------------------------------------------------------- Ran 247 tests in 1.307s
FAILED (failures=4, errors=1) <unittest._TextTestResult run=247 errors=1 failures=4>
But again, I seem to be able to open hdf5 files and look into them ok, though sometimes the files randomly close, i.e.
f = h5py.File( '/path/to/file.h5','r') ... f <Closed hdf5 file> (or something like this)
Should I be worried about these errors, or are things working ok? Someone had once mentioned to me that the numpy tests are prone to failure.
Jonathan Tu
On Sat, Jul 24, 2010 at 3:39 PM, Jonathan Tu jhtu@princeton.edu wrote:
On Jul 24, 2010, at 6:21 PM, David Cournapeau wrote:
On Sun, Jul 25, 2010 at 7:12 AM, Jonathan Tu jhtu@princeton.edu wrote:
What does that part do? It turns out that by fixing my library path, numpy now imports. I assume that my installation of h5py doesn't need to be redone just because I reinstalled Numpy?
Actually, you may have to if you installed h5py against the numpy built from the trunk. I don't think the trunk ABI is forward compatible with 1.4.1,
David
Ok, so I reinstalled numpy as we've discussed, and now I've reinstalled h5py too. When I test numpy, I get the following
import numpy numpy.test() Running unit tests for numpy Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/home/jhtu/local/lib/python2.6/site-packages/numpy/testing/nosetester.py", line 326, in test self._show_system_info() File "/home/jhtu/local/lib/python2.6/site-packages/numpy/testing/nosetester.py", line 187, in _show_system_info nose = import_nose() File "/home/jhtu/local/lib/python2.6/site-packages/numpy/testing/nosetester.py", line 69, in import_nose raise ImportError(msg) ImportError: Need nose >= 0.10.0 for tests - see http://somethingaboutorange.com/mrl/projects/nose
Note that when I try test(level=1) it tells me this is an unexpected keyword argument. (The level=1 was suggested by the scipy website.) Though this test seems to fail, the simple math tests I've tried seem to work ok.
Then I tried to test h5py: import h5py import h5py.tests h5py.tests.runtests()
====================================================================== ERROR: test_data (h5py.tests.test_filters.TestFilters)
Traceback (most recent call last): File "/home/jhtu/local/lib/python2.6/site-packages/h5py/tests/test_filters.py", line 124, in test_data test_dset(s, t, compression=c, shuffle=False) File "/home/jhtu/local/lib/python2.6/site-packages/h5py/tests/test_filters.py", line 117, in test_dset dset[...] = arr File "/home/jhtu/local/lib/python2.6/site-packages/h5py/highlevel.py", line 1288, in __setitem__ self.id.write(mspace, fspace, val, mtype) File "h5d.pyx", line 231, in h5py.h5d.DatasetID.write (h5py/h5d.c:2336) File "_proxy.pyx", line 115, in h5py._proxy.dset_rw (h5py/_proxy.c:1143) TypeError: Not a dataset (Invalid arguments to routine: Inappropriate type)
====================================================================== FAIL: (File) Backing store for core driver
Traceback (most recent call last): File "/home/jhtu/local/lib/python2.6/site-packages/h5py/tests/high/test_file.py", line 135, in test_backing self.assert_('g' in self.f) AssertionError
====================================================================== FAIL: (File) Append
Traceback (most recent call last): File "/home/jhtu/local/lib/python2.6/site-packages/h5py/tests/high/test_file.py", line 52, in test_a self.assert_('g' in self.f) AssertionError
====================================================================== FAIL: (File) Readwrite
Traceback (most recent call last): File "/home/jhtu/local/lib/python2.6/site-packages/h5py/tests/high/test_file.py", line 72, in test_rp self.assert_('g' in self.f) AssertionError
====================================================================== FAIL: test_open_close (h5py.tests.test_h5g.TestH5G)
Traceback (most recent call last): File "/home/jhtu/local/lib/python2.6/site-packages/h5py/tests/test_h5g.py", line 39, in test_open_close self.assert_(self.is_grp(grp)) AssertionError
Ran 247 tests in 1.307s
FAILED (failures=4, errors=1) <unittest._TextTestResult run=247 errors=1 failures=4>
But again, I seem to be able to open hdf5 files and look into them ok, though sometimes the files randomly close, i.e.
f = h5py.File( '/path/to/file.h5','r') ... f <Closed hdf5 file> (or something like this)
Should I be worried about these errors, or are things working ok? Someone had once mentioned to me that the numpy tests are prone to failure.
Yes, you should be worried. To run the numpy tests you need to install nose. That's what your output said:
ImportError: Need nose >= 0.10.0 for tests - see http://somethingaboutorange.com/mrl/projects/nose
For h5py, you might want to try the h5py list. Did you remove the build directory? Because I don't know much about installation, I remove everything: the installed package, the build directory.
Yes, you should be worried. To run the numpy tests you need to install
nose. That's what your output said:
ImportError: Need nose >= 0.10.0 for tests - see http://somethingaboutorange.com/mrl/projects/nose
For h5py, you might want to try the h5py list. Did you remove the build directory? Because I don't know much about installation, I remove everything: the installed package, the build directory.
Thanks for the tip on nose. I installed it, and now I see that I have 5 numpy failures:
====================================================================== FAIL: test_lapack (test_build.TestF77Mismatch) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/jhtu/local/lib/python2.6/site-packages/numpy/testing/decorators.py", line 146, in skipper_func return f(*args, **kwargs) File "/home/jhtu/local/lib/python2.6/site-packages/numpy/linalg/tests/test_build.py", line 48, in test_lapack information.""") AssertionError: Both g77 and gfortran runtimes linked in lapack_lite ! This is likely to cause random crashes and wrong results. See numpy INSTALL.txt for more information.
====================================================================== FAIL: test_cdouble (test_linalg.TestEigh) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/jhtu/local/lib/python2.6/site-packages/numpy/linalg/tests/test_linalg.py", line 221, in test_cdouble self.do(a) File "/home/jhtu/local/lib/python2.6/site-packages/numpy/linalg/tests/test_linalg.py", line 259, in do assert_almost_equal(ev, evalues) File "/home/jhtu/local/lib/python2.6/site-packages/numpy/linalg/tests/test_linalg.py", line 23, in assert_almost_equal old_assert_almost_equal(a, b, decimal=decimal, **kw) File "/home/jhtu/local/lib/python2.6/site-packages/numpy/testing/utils.py", line 437, in assert_almost_equal "DESIRED: %s\n" % (str(actual), str(desired))) AssertionError: Items are not equal: ACTUAL: [-4.48116078 3.37176039] DESIRED: [-2.60555128+0.j 4.60555128+0.j]
====================================================================== FAIL: test_csingle (test_linalg.TestEigh) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/jhtu/local/lib/python2.6/site-packages/numpy/linalg/tests/test_linalg.py", line 217, in test_csingle self.do(a) File "/home/jhtu/local/lib/python2.6/site-packages/numpy/linalg/tests/test_linalg.py", line 259, in do assert_almost_equal(ev, evalues) File "/home/jhtu/local/lib/python2.6/site-packages/numpy/linalg/tests/test_linalg.py", line 23, in assert_almost_equal old_assert_almost_equal(a, b, decimal=decimal, **kw) File "/home/jhtu/local/lib/python2.6/site-packages/numpy/testing/utils.py", line 437, in assert_almost_equal "DESIRED: %s\n" % (str(actual), str(desired))) AssertionError: Items are not equal: ACTUAL: [-4.48116064 3.37176037] DESIRED: [-2.60555124+0.j 4.60555124+0.j]
====================================================================== FAIL: test_cdouble (test_linalg.TestEigvalsh) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/jhtu/local/lib/python2.6/site-packages/numpy/linalg/tests/test_linalg.py", line 221, in test_cdouble self.do(a) File "/home/jhtu/local/lib/python2.6/site-packages/numpy/linalg/tests/test_linalg.py", line 249, in do assert_almost_equal(ev, evalues) File "/home/jhtu/local/lib/python2.6/site-packages/numpy/linalg/tests/test_linalg.py", line 23, in assert_almost_equal old_assert_almost_equal(a, b, decimal=decimal, **kw) File "/home/jhtu/local/lib/python2.6/site-packages/numpy/testing/utils.py", line 437, in assert_almost_equal "DESIRED: %s\n" % (str(actual), str(desired))) AssertionError: Items are not equal: ACTUAL: [-4.48116078+0.j 3.37176039+0.j] DESIRED: [-2.60555128+0.j 4.60555128+0.j]
====================================================================== FAIL: test_csingle (test_linalg.TestEigvalsh) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/jhtu/local/lib/python2.6/site-packages/numpy/linalg/tests/test_linalg.py", line 217, in test_csingle self.do(a) File "/home/jhtu/local/lib/python2.6/site-packages/numpy/linalg/tests/test_linalg.py", line 249, in do assert_almost_equal(ev, evalues) File "/home/jhtu/local/lib/python2.6/site-packages/numpy/linalg/tests/test_linalg.py", line 23, in assert_almost_equal old_assert_almost_equal(a, b, decimal=decimal, **kw) File "/home/jhtu/local/lib/python2.6/site-packages/numpy/testing/utils.py", line 437, in assert_almost_equal "DESIRED: %s\n" % (str(actual), str(desired))) AssertionError: Items are not equal: ACTUAL: [-4.48116064+0.j 3.37176037+0.j] DESIRED: [-2.60555124+0.j 4.60555124+0.j]
---------------------------------------------------------------------- Ran 2483 tests in 15.250s
FAILED (KNOWNFAIL=4, failures=5) <nose.result.TextTestResult run=2483 errors=0 failures=5>
The first seems to be linked to g77 vs gfortran. I used g77 to install lapack. I thought that when I installed numpy, it would use the g77 compiler by default...but maybe this is not the case? Perhaps it also caused the other 4 errors?
And thanks for the tip, I will try the h5py list for those tests.
Jonathan Tu
It shows you don't use either atlas or even simple lapack. If your install used atlas, the atlas libraries would show up on those .so with ldd,
David
Hmm. Ok. What I did for Numpy installation was check out the source code using svn. Then I did python setup.py build where I changed the site.cfg file so that the [Default] category also contains the path to my local Python 2.6.5 installation, which is /home/jhtu/local.
Then I did make install and I think there was probably a prefix type option where I specified that I wanted to install it locally.
How do I specify that I want to build against ATLAS/LAPACK? I'm certain that they are installed on this cluster.
Jonathan Tu
On Sun, Jul 25, 2010 at 5:18 AM, Jonathan Tu jhtu@princeton.edu wrote:
It shows you don't use either atlas or even simple lapack. If your install used atlas, the atlas libraries would show up on those .so with ldd,
David
Hmm. Ok. What I did for Numpy installation was check out the source code using svn.
Use a released version instead (1.4.1 is the last one as we speak). Although we try to keep a relatively robust trunk, it does not have the QA put in the releases.
Then I did python setup.py build where I changed the site.cfg file so that the [Default] category also contains the path to my local Python 2.6.5 installation, which is /home/jhtu/local.
If you install the atlas libraries in /home/jhtu/local/lib, you need something like:
[atlas] library_dirs = /home/jhtu/local/lib
in your site.cfg
Then I did make install and I think there was probably a prefix type option where I specified that I wanted to install it locally.
I guess you meant python setup.py install ? there is no makefile in numpy
How do I specify that I want to build against ATLAS/LAPACK? I'm certain that they are installed on this cluster.
They are, but they are broken, as showed on the logs you posted. The atlas libraries packaged by RHEL 4 (and even 5 IIRC) are hopelessly broken, you need to build atlas by yourself - preferably into an rpm if you want to deploy on a cluster if you have admin rights.
David
participants (3)
-
David Cournapeau -
Jonathan Tu -
Keith Goodman