linalg.eigh hangs only after importing sparse module
Hello, The following simple code hangs only when sparse has been imported: ``` from scipy import sparse # <<<<<<< BUG import numpy as np from scipy import linalg N = 1000 np.random.seed(42) X = np.random.random((N, N)) print X.mean() v, Q = linalg.eigh(X) print v.mean() ``` Do you think this may be related to other arpack/umfpack/etc. known failures ? Please let us know how can we help fix this issue. Thanks for your help. Regards, -- Nicolas Pinto http://web.mit.edu/pinto
On Mon, 19 Mar 2012 19:24:33 +0100 Nicolas Pinto <nicolas.pinto@gmail.com> wrote:
Hello,
The following simple code hangs only when sparse has been imported:
``` from scipy import sparse # <<<<<<< BUG import numpy as np from scipy import linalg
N = 1000 np.random.seed(42) X = np.random.random((N, N)) print X.mean() v, Q = linalg.eigh(X) print v.mean() ```
Do you think this may be related to other arpack/umfpack/etc. known failures ?
Please let us know how can we help fix this issue.
Thanks for your help.
Regards,
Your matrix X is not symmetric.
X-X.T array([[ 0. , 0.76558138, 0.47028826, ..., 0.0515565 , 0.19001774, 0.33171462], [-0.76558138, 0. , 0.62596704, ..., -0.0230795 , -0.90677174, 0.12238354], [-0.47028826, -0.62596704, 0. , ..., -0.38459427, 0.28527075, 0.04568694], ..., [-0.0515565 , 0.0230795 , 0.38459427, ..., 0. , 0.57859577, -0.24268277], [-0.19001774, 0.90677174, -0.28527075, ..., -0.57859577, 0. , 0.52747713], [-0.33171462, -0.12238354, -0.04568694, ..., 0.24268277, -0.52747713, 0. ]])
eigh assumes a symmetric or hermitian matrix. Nils
Your matrix X is not symmetric.
This is not the problem. (Even if that were the problem, it wouldn't cause a hang -- docs say "no error will be reported but results will be wrong".) In fact, the same hang happens on the installation which originally had this problem if you replace X with X = np.dot(X, X.T) so that the matrix is symmetric.
Hi, 19.03.2012 19:24, Nicolas Pinto kirjoitti:
The following simple code hangs only when sparse has been imported: [clip]
It does not hang for me. So, first things first: - which platform? - which binaries? - which LAPACK?
Do you think this may be related to other arpack/umfpack/etc. known failures ?
Please let us know how can we help fix this issue.
ARPACK et al are probably not related, because they are not imported by ``from scipy import sparse``. A more likely candidate is the SWIG-wrapped `sparsetools` package: it is known to also cause some other weirdness: http://projects.scipy.org/scipy/ticket/1314 This unfortunately seems pretty difficult to debug. One thing I could imagine doing is minimizing the problem, by first stripping everything away from `scipy.sparse` except the sparsetools module, and then stripping down the sparsetools code until the failing part is found. -- Pauli Virtanen
19.03.2012 22:09, Pauli Virtanen kirjoitti: [clip]
This unfortunately seems pretty difficult to debug. One thing I could imagine doing is minimizing the problem, by first stripping everything away from `scipy.sparse` except the sparsetools module, and then stripping down the sparsetools code until the failing part is found.
Another possibility is that the problem comes just from the c++ runtime. There's another c++ module in Scipy, `scipy.interpolate._interpolate` -- could you check if importing it also causes the same issue? -- Pauli Virtanen
Thanks for the answers. Sorry for the late answer, I've been out of the country.
Another possibility is that the problem comes just from the c++ runtime. There's another c++ module in Scipy, `scipy.interpolate._interpolate` -- could you check if importing it also causes the same issue?
You are right, the same issue happens with `from scipy.interpolate import _interpolate`. Any advice on how to debug/fix from here? Thanks. N
Nicolas Pinto <nicolas.pinto <at> gmail.com> writes: [clip]
You are right, the same issue happens with `from scipy.interpolate import _interpolate`. Any advice on how to debug/fix from here?
I think you should also verify this by copying `_interpolate.so` outside Scipy and importing it --- namely, `from scipy.interpolate import ...` import also `scipy.sparse` so you cannot pinpoint the problem to `_interpolate`. -- Pauli Virtanen
I think you should also verify this by copying `_interpolate.so` outside Scipy and importing it --- namely, `from scipy.interpolate import ...` import also `scipy.sparse` so you cannot pinpoint the problem to `_interpolate`.
Good point. I can still reproduce the bug by copying `_interpolate.so`.
-- Pauli Virtanen
_______________________________________________ SciPy-User mailing list SciPy-User@scipy.org http://mail.scipy.org/mailman/listinfo/scipy-user
-- Nicolas Pinto http://web.mit.edu/pinto
29.03.2012 17:28, Nicolas Pinto kirjoitti:
I think you should also verify this by copying `_interpolate.so` outside Scipy and importing it --- namely, `from scipy.interpolate import ...` import also `scipy.sparse` so you cannot pinpoint the problem to `_interpolate`.
Good point. I can still reproduce the bug by copying `_interpolate.so`.
To get further, the following information is needed: - which platform? - which binaries? - which LAPACK? I'm assuming you're on 64-bit Windows 7. If so, I don't have good clues on how to fix or debug the issue. However, if it's really the C++ runtime that is causing the problems, then compiling Numpy/Scipy with a different compiler could fix the problem. -- Pauli Virtanen
To get further, the following information is needed:
- which platform?
x86_64 Gentoo Linux, with gcc-4.5.3
- which binaries?
what do you mean ?
- which LAPACK?
atlas-3.8.0
on how to fix or debug the issue. However, if it's really the C++ runtime that is causing the problems, then compiling Numpy/Scipy with a different compiler could fix the problem.
I can try to compile with another gcc version. Thanks for your help. N
-- Pauli Virtanen
_______________________________________________ SciPy-User mailing list SciPy-User@scipy.org http://mail.scipy.org/mailman/listinfo/scipy-user
-- Nicolas Pinto http://web.mit.edu/pinto
I tried (in a virtualenv) to compile with gcc-4.2.4, gcc-4.3.6 and gcc-4.4.6 and all failed. Any suggestion on what to try next? Thanks again. N On Tue, Apr 3, 2012 at 8:21 PM, Nicolas Pinto <nicolas.pinto@gmail.com> wrote:
To get further, the following information is needed:
- which platform?
x86_64 Gentoo Linux, with gcc-4.5.3
- which binaries?
what do you mean ?
- which LAPACK?
atlas-3.8.0
on how to fix or debug the issue. However, if it's really the C++ runtime that is causing the problems, then compiling Numpy/Scipy with a different compiler could fix the problem.
I can try to compile with another gcc version.
Thanks for your help.
N
-- Pauli Virtanen
_______________________________________________ SciPy-User mailing list SciPy-User@scipy.org http://mail.scipy.org/mailman/listinfo/scipy-user
-- Nicolas Pinto http://web.mit.edu/pinto
-- Nicolas Pinto http://web.mit.edu/pinto
05.04.2012 20:33, Nicolas Pinto kirjoitti:
I tried (in a virtualenv) to compile with gcc-4.2.4, gcc-4.3.6 and gcc-4.4.6 and all failed. Any suggestion on what to try next?
Suggestions - Check with different versions of ATLAS and Lapack. - Run the program under valgrind, with python-specific suppressions enabled. Python's source distribution has a suppression file for Valgrind. - Run the program in gdb, first having compiled everything with debug symbols. (OPT="-ggdb" FOPT="-ggdb" python setup.py build ...) Check if the values passed down to the LAPACK routine are OK. Unfortunately, these take some work. Pauli
Update: the issue was related to atlas-3.8.0, upgrading to 3.8.2 or to the 3.9.* series fixed the issue. HTH n On Thu, Apr 5, 2012 at 3:00 PM, Pauli Virtanen <pav@iki.fi> wrote:
05.04.2012 20:33, Nicolas Pinto kirjoitti:
I tried (in a virtualenv) to compile with gcc-4.2.4, gcc-4.3.6 and gcc-4.4.6 and all failed. Any suggestion on what to try next?
Suggestions
- Check with different versions of ATLAS and Lapack.
- Run the program under valgrind, with python-specific suppressions enabled. Python's source distribution has a suppression file for Valgrind.
- Run the program in gdb, first having compiled everything with debug symbols. (OPT="-ggdb" FOPT="-ggdb" python setup.py build ...)
Check if the values passed down to the LAPACK routine are OK.
Unfortunately, these take some work.
Pauli
_______________________________________________ SciPy-User mailing list SciPy-User@scipy.org http://mail.scipy.org/mailman/listinfo/scipy-user
-- Nicolas Pinto http://web.mit.edu/pinto
participants (4)
-
Dan Yamins -
Nicolas Pinto -
Nils Wagner -
Pauli Virtanen