On Sat, Jul 24, 2010 at 12:58 PM, David Cournapeau wrote:

On Sun, Jul 25, 2010 at 4:50 AM, Jonathan Tu wrote:

...

I am unable to find the files lapack_lite.so or _dotblas.so.  I used the locate command to look for them.

If you just installed numpy, locate won't find them as locate uses a database which is usually updated once in a while.

Depending on how you installed numpy, you will find lapack_lite.so in /usr, $HOME/.local or somewhere else (and also in the build directory in the numpy source tree).

Here's one way to find where numpy is installed: $ python

...

...

import numpy numpy

So on my computer numpy is installed in /usr/local/lib/python2.6/site-packages

_dotblas.so may not be present, but numpy cannot work without a lapack_lite.so.

On Sun, Jul 25, 2010 at 5:11 AM, Jonathan Tu wrote:

On Jul 24, 2010, at 4:04 PM, Keith Goodman wrote:

...

On Sat, Jul 24, 2010 at 12:58 PM, David Cournapeau wrote:

...

On Sun, Jul 25, 2010 at 4:50 AM, Jonathan Tu wrote:

...

I am unable to find the files lapack_lite.so or _dotblas.so.  I used the locate command to look for them.

If you just installed numpy, locate won't find them as locate uses a database which is usually updated once in a while.

Depending on how you installed numpy, you will find lapack_lite.so in /usr, $HOME/.local or somewhere else (and also in the build directory in the numpy source tree).

Here's one way to find where numpy is installed:

$ python

...

...

...

import numpy numpy

So on my computer numpy is installed in /usr/local/lib/python2.6/site-packages

...

_dotblas.so may not be present, but numpy cannot work without a lapack_lite.so.

Thanks for the tip!  This worked, and much faster than the manual hunting through directories I'd been trying.  Upon running ldd on lapack_lite.so and on _dotblas.so, I got the following (first for lapack_lite, then for _dotblas):

       libpthread.so.0 => /lib64/tls/libpthread.so.0 (0x0000002a95792000)        libc.so.6 => /lib64/tls/libc.so.6 (0x0000002a958a7000)        /lib64/ld-linux-x86-64.so.2 (0x000000552aaaa000)

       libpthread.so.0 => /lib64/tls/libpthread.so.0 (0x0000002a95d36000)        libc.so.6 => /lib64/tls/libc.so.6 (0x0000002a95e4c000)        /lib64/ld-linux-x86-64.so.2 (0x000000552aaaa000)

I am interested in whether my Numpy installation is in fact taking advantage of the LAPACK libraries that are installed on the cluster where I'm running.  I know that Numpy can function without such an installation, but I think it is probably faster when taking advantage of optimized libraries.  In any case, I know ldd shows library dependencies, but I'm not sure how to make sense of its output, and whether or not it answers my question.

It shows you don't use either atlas or even simple lapack. If your install used atlas, the atlas libraries would show up on those .so with ldd, David

Here is what I would do for RHEL4: - use python 2.6, not 2.7, because the released numpy does not support 2.7 yet. - build atlas 3.8.3 and lapack 3.1.1 by yourself: this is your core issue here. Both lapack and atlas should be built with the -fPIC flag on x86_64 arch (modify the make.inc in lapack to include -fPIC in OPT and NOOPT flags, and add the option -Fa alg -fPIC when calling atlas configure script)

David

I have built lapack and atlas per your instructions (using http://www.scipy.org/Installing_SciPy/Linux for reference). I think this has been done successfully. My first go at building Numpy was not successful, and I was wondering if perhaps I did a few things wrong. At one point, the website I referenced above suggests that I can make shared libraries by doing cd lib #to where the atlas libraries are make shared # for sequential libraries make ptshared # for threaded libraries I initially did both of these things, which I'm not sure is ok. The above website also says "our atlas libraries should now be in the lib subdirectory of the current directory. You should copy them to some-place that you can tell site.cfg about." So I copied the libraries using cp to /home/jhtu/local/lib. The final thing I had to do was modify my site.cfg. I had a few questions about this. First of all, for my default paths, do I need to do library_dirs = /usr/local/bin:/home/jhtu/local/lib or just library_dirs = /home/jhtu/local/lib Finally, to specifiy the lapack libraries, there is a section like [blas_opt] libraries = f77blas,... but also another choice [blas_opt] libraries=ptf77blas,... I think this is related to the make shared vs make ptshared question above, but this is something I know nothing about. I think these are the only things that could be causing my problems. Jonathan Tu

On Sun, Jul 25, 2010 at 6:50 AM, Jonathan Tu wrote:

...

...

...

import numpy Traceback (most recent call last):   File "<stdin>", line 1, in <module>   File "/home/jhtu/local/lib/python2.6/site-packages/numpy/__init__.py",

Also, I recently tried this all again, only doing make shared, and only using the list of libraries without "pt."  In both cases, when I ran Python interactively and tried to load the numpy module, I got the following: line 132, in <module>     import add_newdocs   File "/home/jhtu/local/lib/python2.6/site-packages/numpy/add_newdocs.py", line 9, in <module>     from lib import add_newdoc   File "/home/jhtu/local/lib/python2.6/site-packages/numpy/lib/__init__.py", line 13, in <module>     from polynomial import *   File "/home/jhtu/local/lib/python2.6/site-packages/numpy/lib/polynomial.py", line 17, in <module>     from numpy.linalg import eigvals, lstsq   File "/home/jhtu/local/lib/python2.6/site-packages/numpy/linalg/__init__.py", line 47, in <module>     from linalg import *   File "/home/jhtu/local/lib/python2.6/site-packages/numpy/linalg/linalg.py", line 22, in <module>     from numpy.linalg import lapack_lite ImportError: libptf77blas.so: cannot open shared object file: No such file or directory

If you install shared libraries into a directory which is not looked in by default by ld, you should add it to the LD_LIBRARY_PATH, e.g.: LD_LIBRARY_PATH=$HOME/local/lib:$LD_LIBRARY_PATH python -c "import numpy" David

On Sun, Jul 25, 2010 at 7:00 AM, Jonathan Tu wrote:

...

If you install shared libraries into a directory which is not looked in by default by ld, you should add it to the LD_LIBRARY_PATH, e.g.:

LD_LIBRARY_PATH=$HOME/local/lib:$LD_LIBRARY_PATH python -c "import numpy"

David

Ok, that makes sense to me.  I tried to see what the current value is by doing        echo $LD_LIBRARY_PATH

It is very possible that you don't have it defined by default on your platform. The undefined variable may be a peculiarity of the cshell - the instructions I gave you assumed a bourne shell or compatible. David

On Sat, Jul 24, 2010 at 3:12 PM, Jonathan Tu wrote:

On Jul 24, 2010, at 6:09 PM, David Cournapeau wrote:

...

On Sun, Jul 25, 2010 at 7:00 AM, Jonathan Tu wrote:

...

...

If you install shared libraries into a directory which is not looked in by default by ld, you should add it to the LD_LIBRARY_PATH, e.g.:

LD_LIBRARY_PATH=$HOME/local/lib:$LD_LIBRARY_PATH python -c "import numpy"

David

Ok, that makes sense to me.  I tried to see what the current value is by doing        echo $LD_LIBRARY_PATH

It is very possible that you don't have it defined by default on your platform. The undefined variable may be a peculiarity of the cshell - the instructions I gave you assumed a bourne shell or compatible.

David

Ok, I assumed as much and proceeded by simply setting the environment variable.  I did not add the part where you said        python -c "import numpy"

What does that part do?  It turns out that by fixing my library path, numpy now imports.  I assume that my installation of h5py doesn't need to be redone just because I reinstalled Numpy?

Try: $ python

...

...

import h5py.tests h5py.tests.runtests() .......................................................................................................................................................................................................................................................

Ran 247 tests in 1.180s OK

On Jul 24, 2010, at 6:21 PM, David Cournapeau wrote:

On Sun, Jul 25, 2010 at 7:12 AM, Jonathan Tu wrote:

...

What does that part do? It turns out that by fixing my library path, numpy now imports. I assume that my installation of h5py doesn't need to be redone just because I reinstalled Numpy?

Actually, you may have to if you installed h5py against the numpy built from the trunk. I don't think the trunk ABI is forward compatible with 1.4.1,

David

Ok, so I reinstalled numpy as we've discussed, and now I've reinstalled h5py too. When I test numpy, I get the following import numpy numpy.test() Running unit tests for numpy Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/home/jhtu/local/lib/python2.6/site-packages/numpy/testing/nosetester.py", line 326, in test self._show_system_info() File "/home/jhtu/local/lib/python2.6/site-packages/numpy/testing/nosetester.py", line 187, in _show_system_info nose = import_nose() File "/home/jhtu/local/lib/python2.6/site-packages/numpy/testing/nosetester.py", line 69, in import_nose raise ImportError(msg) ImportError: Need nose >= 0.10.0 for tests - see http://somethingaboutorange.com/mrl/projects/nose Note that when I try test(level=1) it tells me this is an unexpected keyword argument. (The level=1 was suggested by the scipy website.) Though this test seems to fail, the simple math tests I've tried seem to work ok. Then I tried to test h5py: import h5py import h5py.tests h5py.tests.runtests() ====================================================================== ERROR: test_data (h5py.tests.test_filters.TestFilters) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/jhtu/local/lib/python2.6/site-packages/h5py/tests/test_filters.py", line 124, in test_data test_dset(s, t, compression=c, shuffle=False) File "/home/jhtu/local/lib/python2.6/site-packages/h5py/tests/test_filters.py", line 117, in test_dset dset[...] = arr File "/home/jhtu/local/lib/python2.6/site-packages/h5py/highlevel.py", line 1288, in __setitem__ self.id.write(mspace, fspace, val, mtype) File "h5d.pyx", line 231, in h5py.h5d.DatasetID.write (h5py/h5d.c:2336) File "_proxy.pyx", line 115, in h5py._proxy.dset_rw (h5py/_proxy.c:1143) TypeError: Not a dataset (Invalid arguments to routine: Inappropriate type) ====================================================================== FAIL: (File) Backing store for core driver ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/jhtu/local/lib/python2.6/site-packages/h5py/tests/high/test_file.py", line 135, in test_backing self.assert_('g' in self.f) AssertionError ====================================================================== FAIL: (File) Append ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/jhtu/local/lib/python2.6/site-packages/h5py/tests/high/test_file.py", line 52, in test_a self.assert_('g' in self.f) AssertionError ====================================================================== FAIL: (File) Readwrite ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/jhtu/local/lib/python2.6/site-packages/h5py/tests/high/test_file.py", line 72, in test_rp self.assert_('g' in self.f) AssertionError ====================================================================== FAIL: test_open_close (h5py.tests.test_h5g.TestH5G) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/jhtu/local/lib/python2.6/site-packages/h5py/tests/test_h5g.py", line 39, in test_open_close self.assert_(self.is_grp(grp)) AssertionError ---------------------------------------------------------------------- Ran 247 tests in 1.307s FAILED (failures=4, errors=1) But again, I seem to be able to open hdf5 files and look into them ok, though sometimes the files randomly close, i.e. f = h5py.File( '/path/to/file.h5','r') ... f <Closed hdf5 file> (or something like this) Should I be worried about these errors, or are things working ok? Someone had once mentioned to me that the numpy tests are prone to failure. Jonathan Tu

...

Yes, you should be worried. To run the numpy tests you need to install nose. That's what your output said:

ImportError: Need nose >= 0.10.0 for tests - see http://somethingaboutorange.com/mrl/projects/nose

For h5py, you might want to try the h5py list. Did you remove the build directory? Because I don't know much about installation, I remove everything: the installed package, the build directory.

Thanks for the tip on nose. I installed it, and now I see that I have 5 numpy failures: ====================================================================== FAIL: test_lapack (test_build.TestF77Mismatch) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/jhtu/local/lib/python2.6/site-packages/numpy/testing/decorators.py", line 146, in skipper_func return f(*args, **kwargs) File "/home/jhtu/local/lib/python2.6/site-packages/numpy/linalg/tests/test_build.py", line 48, in test_lapack information.""") AssertionError: Both g77 and gfortran runtimes linked in lapack_lite ! This is likely to cause random crashes and wrong results. See numpy INSTALL.txt for more information. ====================================================================== FAIL: test_cdouble (test_linalg.TestEigh) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/jhtu/local/lib/python2.6/site-packages/numpy/linalg/tests/test_linalg.py", line 221, in test_cdouble self.do(a) File "/home/jhtu/local/lib/python2.6/site-packages/numpy/linalg/tests/test_linalg.py", line 259, in do assert_almost_equal(ev, evalues) File "/home/jhtu/local/lib/python2.6/site-packages/numpy/linalg/tests/test_linalg.py", line 23, in assert_almost_equal old_assert_almost_equal(a, b, decimal=decimal, **kw) File "/home/jhtu/local/lib/python2.6/site-packages/numpy/testing/utils.py", line 437, in assert_almost_equal "DESIRED: %s\n" % (str(actual), str(desired))) AssertionError: Items are not equal: ACTUAL: [-4.48116078 3.37176039] DESIRED: [-2.60555128+0.j 4.60555128+0.j] ====================================================================== FAIL: test_csingle (test_linalg.TestEigh) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/jhtu/local/lib/python2.6/site-packages/numpy/linalg/tests/test_linalg.py", line 217, in test_csingle self.do(a) File "/home/jhtu/local/lib/python2.6/site-packages/numpy/linalg/tests/test_linalg.py", line 259, in do assert_almost_equal(ev, evalues) File "/home/jhtu/local/lib/python2.6/site-packages/numpy/linalg/tests/test_linalg.py", line 23, in assert_almost_equal old_assert_almost_equal(a, b, decimal=decimal, **kw) File "/home/jhtu/local/lib/python2.6/site-packages/numpy/testing/utils.py", line 437, in assert_almost_equal "DESIRED: %s\n" % (str(actual), str(desired))) AssertionError: Items are not equal: ACTUAL: [-4.48116064 3.37176037] DESIRED: [-2.60555124+0.j 4.60555124+0.j] ====================================================================== FAIL: test_cdouble (test_linalg.TestEigvalsh) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/jhtu/local/lib/python2.6/site-packages/numpy/linalg/tests/test_linalg.py", line 221, in test_cdouble self.do(a) File "/home/jhtu/local/lib/python2.6/site-packages/numpy/linalg/tests/test_linalg.py", line 249, in do assert_almost_equal(ev, evalues) File "/home/jhtu/local/lib/python2.6/site-packages/numpy/linalg/tests/test_linalg.py", line 23, in assert_almost_equal old_assert_almost_equal(a, b, decimal=decimal, **kw) File "/home/jhtu/local/lib/python2.6/site-packages/numpy/testing/utils.py", line 437, in assert_almost_equal "DESIRED: %s\n" % (str(actual), str(desired))) AssertionError: Items are not equal: ACTUAL: [-4.48116078+0.j 3.37176039+0.j] DESIRED: [-2.60555128+0.j 4.60555128+0.j] ====================================================================== FAIL: test_csingle (test_linalg.TestEigvalsh) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/jhtu/local/lib/python2.6/site-packages/numpy/linalg/tests/test_linalg.py", line 217, in test_csingle self.do(a) File "/home/jhtu/local/lib/python2.6/site-packages/numpy/linalg/tests/test_linalg.py", line 249, in do assert_almost_equal(ev, evalues) File "/home/jhtu/local/lib/python2.6/site-packages/numpy/linalg/tests/test_linalg.py", line 23, in assert_almost_equal old_assert_almost_equal(a, b, decimal=decimal, **kw) File "/home/jhtu/local/lib/python2.6/site-packages/numpy/testing/utils.py", line 437, in assert_almost_equal "DESIRED: %s\n" % (str(actual), str(desired))) AssertionError: Items are not equal: ACTUAL: [-4.48116064+0.j 3.37176037+0.j] DESIRED: [-2.60555124+0.j 4.60555124+0.j] ---------------------------------------------------------------------- Ran 2483 tests in 15.250s FAILED (KNOWNFAIL=4, failures=5) The first seems to be linked to g77 vs gfortran. I used g77 to install lapack. I thought that when I installed numpy, it would use the g77 compiler by default...but maybe this is not the case? Perhaps it also caused the other 4 errors? And thanks for the tip, I will try the h5py list for those tests. Jonathan Tu

On Sun, Jul 25, 2010 at 5:18 AM, Jonathan Tu wrote:

...

It shows you don't use either atlas or even simple lapack. If your install used atlas, the atlas libraries would show up on those .so with ldd,

David

Hmm.  Ok.  What I did for Numpy installation was check out the source code using svn.

Use a released version instead (1.4.1 is the last one as we speak). Although we try to keep a relatively robust trunk, it does not have the QA put in the releases.

 Then I did        python setup.py build where I changed the site.cfg file so that the [Default] category also contains the path to my local Python 2.6.5 installation, which is /home/jhtu/local.

If you install the atlas libraries in /home/jhtu/local/lib, you need something like: [atlas] library_dirs = /home/jhtu/local/lib in your site.cfg

Then I did        make install and I think there was probably a prefix type option where I specified that I wanted to install it locally.

I guess you meant python setup.py install ? there is no makefile in numpy

How do I specify that I want to build against ATLAS/LAPACK?  I'm certain that they are installed on this cluster.

They are, but they are broken, as showed on the logs you posted. The atlas libraries packaged by RHEL 4 (and even 5 IIRC) are hopelessly broken, you need to build atlas by yourself - preferably into an rpm if you want to deploy on a cluster if you have admin rights. David