scipy import problem in apache-mod_wsgi environment
Hello, list! First of all, this will be a long message, due to the complexity of our environment. So, please be patient with my question. I am trying to run a simple django application in a cluster environment. And, my application hangs while it imports scipy.linalg, and both scipy and apache do not write out error messages. When I run my application in my local python shell, it imports scipy.linalg. But, somehow it does not when it is run by apache. So, after reading this message, please share any ideas about how to debug this problem or new solutions to address this issue or deploy my application. Now, let me explain our current setup. 1. OS -- The server is a compute cluster where each node runs centos 6 that was installed from a clean version of centos6 minimal. 2. Apache -- Apache 2.2 is also manually installed from one of default linux repository. To be specific, it is installed from source code together with httpd-dev. 3. Python -- Python 2.7.2 is also installed from source code across all nodes in the cluster. Its source code is downloaded from python.org's ftp. 4. Python packages: nose, numpy, scipy -- Nose 1.1.2 was downloaded from pypi.python.org and installed from its source code. -- numpy 1.6.1 was downloaded and installed from a linux repository. When building numpy, gnu95 fortran complier was used. -- To install scipy, we installed atlas-3.8.4, lapack-3.3.1, and blas from their source code. ----- atlas was from sourceforge's 3.8.4 stable version. To compile altas, gcc was used. ----- lapack and blas was obtained from netlib.org's repository. To compile the package of lapack and blas, gforan was used. ----- Finally, after exporting paths to blas, lapack, and atlas, scipy-0.9.0 was installed from its source code. scipy was obtained from sourceforge.net's repository. All of the above were installed in the same way across all nodes in our cluster. Since I am the only user of the cluster who needs to run python web applications, I installed python virtualenv package in my local directory. Within my virtual environment, django-1.3 and pysal-1.2 (our own package) were installed. To deploy my web applications, we used mod_wsgi. mod-wsgi was compiled with python-2.7.2 and loaded into apache-2.2. My application is attached. Basically, it is 'hello world' application that tests if numpy, scipy, and pysal can be imported. In the attached file, lines 4-9 are just adding paths to django and pysal so that apache knows where to find these packages. Also, to let apache know where to find atlas-related packages, the path to those packages was added to the LD_LIBRARY_PATH environment variable in the /etc/sysconfig/httpd file. When I first ran my application, it just hung and wrote no message. So, across scipy.linalg modules, I added print out statements to figure out at which point the import was broken. Here is the messages I got when I imported scipy.linalg in my local python shell. 1. ######################## 2. starting linalg.__init__ 3. pre __init__.__doc__ 4. pre __init__.__version__ 5. pre __init__.misc 6. pre __init__.basic 7. ####################### 8. Starting basic 9. pre basic.flinalg 10. pre basic.lapack 11. pre basic.misc 12. pre basic.scipy.linalg 13. pre basic.decomp_svd 14. pre __init__.decomp 15. ################ 16. starting decomp 17. pre decomp.array et al. 18. pre decomp.calc_lwork 19. pre decomp.LinAlgError 20. pre decomp.get_lapack_funcs 21. pre decomp.get_blas_funcs 22. #################### 23. Starting blas 24. pre blas.scipy.linalg.fblas 25. pre blas.scipy.linalg.cblas 26. pre __init__.decomp_lu 27. pre __init__.decomp_cholesky 28. pre __init__.decomp_qr 29. ################# 30. Starting special_matrices 31. pre special_matrices.math 32. pre special_matrices.np 33. pre __init__.decomp_svd 34. pre __init__.decomp_schur 35. ################## 36. starting schur... 37. pre decomp_schur.misc 38. pre decomp_schur.LinAlgError 39. pre decomp_schur.get_lapack_funcs 40. pre decomp_schur.eigvals:1320454147.23Fri Nov 4 17:49:07 2011 41. schur testing 42. pre __init__.matfuncs 43. ##################### 44. Starting matfuncs 45. pre matfuncs. asarray et al 46. pre matfuncs.matrix 47. pre matfuncs.np 48. pre matfuncs.misc 49. pre matfuncs.basic 50. pre matfuncs.special_matrices 51. pre matfuncs.decomp 52. pre matfuncs.decomp_svd 53. pre matfuncs.decomp_schur 54. pre __init__.blas 55. pre __init__.special_matrices When scipy.linalg is successfully imported, I should get these messages. But, when my web application tried to import scipy.linalg, the output messages stop at line 41. At line 41, decomp_schur.py tries to import decomp.py. Since decomp.py was already imported at line 16, scipy ignores it and continues to import other modules in my local shell. But, somehow, in apache-mod_wsgi environment, scipy failed to ignore or reload decomp.py and seems to kill my web application. This is really odd, because python does not give any message about this error and neither does apache. apache just hangs without sending out any response. Since lapack and blas functions were imported successfully, the problem seems not related to path setup. If anyone in the list has any insights into or experience into this kind of symptom, please share your insights and experience. In particular, debugging techniques or less-known installation/compilation problems would be helpful. I feel like I am at a dead end. So, please help me. Thanks for reading this post. I will look forward to yo -- Myung-Hwa Hwang GeoDa Center School of Geographical Sciences and Urban Planning Arizona State University mhwang4@gmail.com or Myunghwa.Hwang@asu.edu
Hello, list! First of all, this will be a long message, due to the complexity of our environment. So, please be patient with my question. I am trying to run a simple django application in a cluster environment. And, my application hangs while it imports scipy.linalg, and both scipy and apache do not write out error messages. When I run my application in my local python shell, it imports scipy.linalg. But, somehow it does not when it is run by apache. So, after reading this message, please share any ideas about how to debug this problem or new solutions to address this issue or deploy my application. Now, let me explain our current setup. 1. OS -- The server is a compute cluster where each node runs centos 6 that was installed from a clean version of centos6 minimal. 2. Apache -- Apache 2.2 was also manually installed from one of default linux repository. To be specific, it was installed from its source code together with httpd-dev. 3. Python -- Python 2.7.2 was also installed from its source code across all nodes in the cluster. Its source code was downloaded from python.org's ftp. 4. Python packages: nose, numpy, scipy -- Nose 1.1.2 was downloaded from pypi.python.org and installed from its source code. -- numpy 1.6.1 was downloaded and installed from a linux repository. When building numpy, gnu95 fortran complier was used. -- To install scipy, we installed atlas-3.8.4, lapack-3.3.1, and blas from their source code. ----- atlas was from sourceforge's 3.8.4 stable version. To compile altas, gcc was used. ----- lapack and blas was obtained from netlib.org's repository. To compile the package of lapack and blas, gforan was used. ----- Finally, after exporting paths to blas, lapack, and atlas, scipy-0.9.0 was installed from its source code. scipy was obtained from sourceforge.net's repository. A note that contains the above information about software installation is attached. All of the above were installed in the same way across all nodes in our cluster. Since I am the only user of the cluster who needs to run python web applications, I installed python virtualenv package in my local directory. Within my virtual environment, django-1.3 and pysal-1.2 (our own package) were installed. To deploy my web applications, we used mod_wsgi. mod-wsgi was compiled with python-2.7.2 and loaded into apache-2.2. My application is attached. Basically, it is a 'hello world' application that tests if numpy, scipy, and pysal can be imported. In the attached file, lines 4-9 are just adding paths to django and pysal so that apache knows where to find these packages. Also, to let apache know where to find atlas-related packages, the path to those packages was added to the LD_LIBRARY_PATH environment variable in the /etc/sysconfig/httpd file. When I first ran my application, it just hung and wrote no message. So, across scipy.linalg modules, I added print out statements to figure out at which point the import was broken. Here is the messages I got when I imported scipy.linalg in my local python shell. 1. ######################## 2. starting linalg.__init__ 3. pre __init__.__doc__ 4. pre __init__.__version__ 5. pre __init__.misc 6. pre __init__.basic 7. ####################### 8. Starting basic 9. pre basic.flinalg 10. pre basic.lapack 11. pre basic.misc 12. pre basic.scipy.linalg 13. pre basic.decomp_svd 14. pre __init__.decomp 15. ################ 16. starting decomp 17. pre decomp.array et al. 18. pre decomp.calc_lwork 19. pre decomp.LinAlgError 20. pre decomp.get_lapack_funcs 21. pre decomp.get_blas_funcs 22. #################### 23. Starting blas 24. pre blas.scipy.linalg.fblas 25. pre blas.scipy.linalg.cblas 26. pre __init__.decomp_lu 27. pre __init__.decomp_cholesky 28. pre __init__.decomp_qr 29. ################# 30. Starting special_matrices 31. pre special_matrices.math 32. pre special_matrices.np 33. pre __init__.decomp_svd 34. pre __init__.decomp_schur 35. ################## 36. starting schur... 37. pre decomp_schur.misc 38. pre decomp_schur.LinAlgError 39. pre decomp_schur.get_lapack_funcs 40. pre decomp_schur.eigvals:1320454147.23Fri Nov 4 17:49:07 2011 41. schur testing 42. pre __init__.matfuncs 43. ##################### 44. Starting matfuncs 45. pre matfuncs. asarray et al 46. pre matfuncs.matrix 47. pre matfuncs.np 48. pre matfuncs.misc 49. pre matfuncs.basic 50. pre matfuncs.special_matrices 51. pre matfuncs.decomp 52. pre matfuncs.decomp_svd 53. pre matfuncs.decomp_schur 54. pre __init__.blas 55. pre __init__.special_matrices When scipy.linalg is successfully imported, I should get these messages. But, when my web application tried to import scipy.linalg, the output messages stop at line 41. At line 41, decomp_schur.py tries to import decomp.py. Since decomp.py was already imported at line 16, scipy ignores it and continues to import other modules in my local shell. But, somehow, in apache-mod_wsgi environment, scipy failed to ignore or reload decomp.py and seems to kill my web application. This is really odd, because python does not give any message about this error and neither does apache. apache just hangs without sending out any response. Since lapack and blas functions were imported successfully, the problem seems not related to path setup. If anyone in the list has any insights into or experience into this kind of symptom, please share your insights and experience. In particular, debugging techniques or less-known installation/compilation problems would be helpful. I feel like I am at a dead end. So, please help me. Thanks for reading this post. I will look forward to your responses. -- Myung-Hwa Hwang -- Myung-Hwa Hwang GeoDa Center School of Geographical Sciences and Urban Planning Arizona State University mhwang4@gmail.com or Myunghwa.Hwang@asu.edu -- Myung-Hwa Hwang GeoDa Center School of Geographical Sciences and Urban Planning Arizona State University mhwang4@gmail.com or Myunghwa.Hwang@asu.edu
I would try putting print statements inside "decomp_schur.py" since that is the module that you said is causing problems. Print out the contents of the dictionary sys.modules just before the import of decomp in "decomp_schur.py". Is 'decomp' in the dictionary? What happens if you comment-out the import of decomp in "decomp_schur.py" ? -- Cameron Hayne macdev@hayne.net On 5-Nov-11, at 3:59 PM, Myunghwa Hwang wrote:
I am trying to run a simple django application in a cluster environment. And, my application hangs while it imports scipy.linalg, and both scipy and apache do not write out error messages. When I run my application in my local python shell, it imports scipy.linalg. But, somehow it does not when it is run by apache. So, after reading this message, please share any ideas about how to debug this problem or new solutions to address this issue or deploy my application.
Now, let me explain our current setup. 1. OS -- The server is a compute cluster where each node runs centos 6 that was installed from a clean version of centos6 minimal. 2. Apache -- Apache 2.2 was also manually installed from one of default linux repository. To be specific, it was installed from its source code together with httpd-dev. 3. Python -- Python 2.7.2 was also installed from its source code across all nodes in the cluster. Its source code was downloaded from python.org's ftp. 4. Python packages: nose, numpy, scipy -- Nose 1.1.2 was downloaded from pypi.python.org and installed from its source code. -- numpy 1.6.1 was downloaded and installed from a linux repository. When building numpy, gnu95 fortran complier was used. -- To install scipy, we installed atlas-3.8.4, lapack-3.3.1, and blas from their source code. ----- atlas was from sourceforge's 3.8.4 stable version. To compile altas, gcc was used. ----- lapack and blas was obtained from netlib.org's repository. To compile the package of lapack and blas, gforan was used. ----- Finally, after exporting paths to blas, lapack, and atlas, scipy-0.9.0 was installed from its source code. scipy was obtained from sourceforge.net's repository. A note that contains the above information about software installation is attached.
All of the above were installed in the same way across all nodes in our cluster. Since I am the only user of the cluster who needs to run python web applications, I installed python virtualenv package in my local directory. Within my virtual environment, django-1.3 and pysal-1.2 (our own package) were installed. To deploy my web applications, we used mod_wsgi. mod-wsgi was compiled with python-2.7.2 and loaded into apache-2.2. My application is attached. Basically, it is a 'hello world' application that tests if numpy, scipy, and pysal can be imported. In the attached file, lines 4-9 are just adding paths to django and pysal so that apache knows where to find these packages. Also, to let apache know where to find atlas-related packages, the path to those packages was added to the LD_LIBRARY_PATH environment variable in the /etc/sysconfig/httpd file.
When I first ran my application, it just hung and wrote no message. So, across scipy.linalg modules, I added print out statements to figure out at which point the import was broken. Here is the messages I got when I imported scipy.linalg in my local python shell. • ######################## • starting linalg.__init__ • pre __init__.__doc__ • pre __init__.__version__ • pre __init__.misc • pre __init__.basic • ####################### • Starting basic • pre basic.flinalg • pre basic.lapack • pre basic.misc • pre basic.scipy.linalg • pre basic.decomp_svd • pre __init__.decomp • ################ • starting decomp • pre decomp.array et al. • pre decomp.calc_lwork • pre decomp.LinAlgError • pre decomp.get_lapack_funcs • pre decomp.get_blas_funcs • #################### • Starting blas • pre blas.scipy.linalg.fblas • pre blas.scipy.linalg.cblas • pre __init__.decomp_lu • pre __init__.decomp_cholesky • pre __init__.decomp_qr • ################# • Starting special_matrices • pre special_matrices.math • pre special_matrices.np • pre __init__.decomp_svd • pre __init__.decomp_schur • ################## • starting schur... • pre decomp_schur.misc • pre decomp_schur.LinAlgError • pre decomp_schur.get_lapack_funcs • pre decomp_schur.eigvals:1320454147.23Fri Nov 4 17:49:07 2011 • schur testing • pre __init__.matfuncs • ##################### • Starting matfuncs • pre matfuncs. asarray et al • pre matfuncs.matrix • pre matfuncs.np • pre matfuncs.misc • pre matfuncs.basic • pre matfuncs.special_matrices • pre matfuncs.decomp • pre matfuncs.decomp_svd • pre matfuncs.decomp_schur • pre __init__.blas • pre __init__.special_matrices When scipy.linalg is successfully imported, I should get these messages. But, when my web application tried to import scipy.linalg, the output messages stop at line 41. At line 41, decomp_schur.py tries to import decomp.py. Since decomp.py was already imported at line 16, scipy ignores it and continues to import other modules in my local shell. But, somehow, in apache-mod_wsgi environment, scipy failed to ignore or reload decomp.py and seems to kill my web application. This is really odd, because python does not give any message about this error and neither does apache. apache just hangs without sending out any response. Since lapack and blas functions were imported successfully, the problem seems not related to path setup.
If anyone in the list has any insights into or experience into this kind of symptom, please share your insights and experience. In particular, debugging techniques or less-known installation/compilation problems would be helpful. I feel like I am at a dead end. So, please help me.
Thanks for reading this post. I will look forward to your responses.
-- Myung-Hwa Hwang
-- Myung-Hwa Hwang GeoDa Center School of Geographical Sciences and Urban Planning Arizona State University mhwang4@gmail.com or Myunghwa.Hwang@asu.edu
Hi, Hayne! Thanks for your answer. After trying out what you suggested (that is, commenting out the import of decomp), I found out the import of decomp was not the problem. In decomp_schur, there are two lines checking something related to rounding errors specific to a single machine as follows: eps = np.finfo(float).eps feps = numpy.finfo(single).eps When scipy reaches the above lines, my application hangs. I found a web document where the author encountered the same problem with these lines but in different contexts: http://stackoverflow.com/questions/7592565/when-embedding-cpython-in-java-wh... The discussion in the web document is not applicable to my problem. Also, the np.finfo statements seem to exist in multiple modules of scipy. Without addressing all related modules manually, would it be any other solutions? Thanks! --Myung-Hwa On Sat, Nov 5, 2011 at 1:14 PM, <hayne@sympatico.ca> wrote:
I would try putting print statements inside "decomp_schur.py" since that is the module that you said is causing problems. Print out the contents of the dictionary sys.modules just before the import of decomp in "decomp_schur.py". Is 'decomp' in the dictionary? What happens if you comment-out the import of decomp in "decomp_schur.py" ? -- Cameron Hayne macdev@hayne.net
On 5-Nov-11, at 3:59 PM, Myunghwa Hwang wrote:
I am trying to run a simple django application in a cluster environment. And, my application hangs while it imports scipy.linalg, and both scipy and apache do not write out error messages. When I run my application in my local python shell, it imports scipy.linalg. But, somehow it does not when it is run by apache. So, after reading this message, please share any ideas about how to debug this problem or new solutions to address this issue or deploy my application.
Now, let me explain our current setup. 1. OS -- The server is a compute cluster where each node runs centos 6 that was installed from a clean version of centos6 minimal.2. Apache
-- Apache 2.2 was also manually installed from one of default linux repository. To be specific, it was installed from its source code together with httpd-dev. 3. Python -- Python 2.7.2 was also installed from its source code across all nodes in the cluster. Its source code was downloaded from python.org's ftp. 4. Python packages: nose, numpy, scipy -- Nose 1.1.2 was downloaded from pypi.python.org and installed from its source code. -- numpy 1.6.1 was downloaded and installed from a linux repository. When building numpy, gnu95 fortran complier was used. -- To install scipy, we installed atlas-3.8.4, lapack-3.3.1, and blas from their source code.----- atlas was from sourceforge's 3.8.4 stable version. To compile altas, gcc was used.
----- lapack and blas was obtained from netlib.org's repository. To compile the package of lapack and blas, gforan was used. ----- Finally, after exporting paths to blas, lapack, and atlas, scipy-0.9.0 was installed from its source code. scipy was obtained from sourceforge.net's repository. A note that contains the above information about software installation is attached.
All of the above were installed in the same way across all nodes in our cluster. Since I am the only user of the cluster who needs to run python web applications, I installed python virtualenv package in my local directory. Within my virtual environment, django-1.3 and pysal-1.2 (our own package) were installed. To deploy my web applications, we used mod_wsgi. mod-wsgi was compiled with python-2.7.2 and loaded into apache-2.2. My application is attached. Basically, it is a 'hello world' application that tests if numpy, scipy, and pysal can be imported. In the attached file, lines 4-9 are just adding paths to django and pysal so that apache knows where to find these packages. Also, to let apache know where to find atlas-related packages, the path to those packages was added to the LD_LIBRARY_PATH environment variable in the /etc/sysconfig/httpd file.
When I first ran my application, it just hung and wrote no message. So, across scipy.linalg modules, I added print out statements to figure out at which point the import was broken. Here is the messages I got when I imported scipy.linalg in my local python shell. • ######################## • starting linalg.__init__ • pre __init__.__doc__ • pre __init__.__version__ • pre __init__.misc • pre __init__.basic • ####################### • Starting basic • pre basic.flinalg • pre basic.lapack • pre basic.misc • pre basic.scipy.linalg • pre basic.decomp_svd • pre __init__.decomp • ################ • starting decomp • pre decomp.array et al. • pre decomp.calc_lwork • pre decomp.LinAlgError • pre decomp.get_lapack_funcs • pre decomp.get_blas_funcs • #################### • Starting blas • pre blas.scipy.linalg.fblas • pre blas.scipy.linalg.cblas • pre __init__.decomp_lu • pre __init__.decomp_cholesky • pre __init__.decomp_qr • ################# • Starting special_matrices • pre special_matrices.math • pre special_matrices.np • pre __init__.decomp_svd • pre __init__.decomp_schur • ################## • starting schur... • pre decomp_schur.misc • pre decomp_schur.LinAlgError • pre decomp_schur.get_lapack_funcs • pre decomp_schur.eigvals:**1320454147.23Fri Nov 4 17:49:07 2011 • schur testing • pre __init__.matfuncs • ##################### • Starting matfuncs • pre matfuncs. asarray et al • pre matfuncs.matrix • pre matfuncs.np • pre matfuncs.misc • pre matfuncs.basic • pre matfuncs.special_matrices • pre matfuncs.decomp • pre matfuncs.decomp_svd • pre matfuncs.decomp_schur • pre __init__.blas • pre __init__.special_matrices When scipy.linalg is successfully imported, I should get these messages. But, when my web application tried to import scipy.linalg, the output messages stop at line 41. At line 41, decomp_schur.py tries to import decomp.py. Since decomp.py was already imported at line 16, scipy ignores it and continues to import other modules in my local shell. But, somehow, in apache-mod_wsgi environment, scipy failed to ignore or reload decomp.py and seems to kill my web application. This is really odd, because python does not give any message about this error and neither does apache. apache just hangs without sending out any response. Since lapack and blas functions were imported successfully, the problem seems not related to path setup.
If anyone in the list has any insights into or experience into this kind of symptom, please share your insights and experience. In particular, debugging techniques or less-known installation/compilation problems would be helpful. I feel like I am at a dead end. So, please help me.
Thanks for reading this post. I will look forward to your responses.
-- Myung-Hwa Hwang
-- Myung-Hwa Hwang GeoDa Center School of Geographical Sciences and Urban Planning Arizona State University mhwang4@gmail.com or Myunghwa.Hwang@asu.edu
-- Myung-Hwa Hwang GeoDa Center School of Geographical Sciences and Urban Planning Arizona State University mhwang4@gmail.com or Myunghwa.Hwang@asu.edu
On Mon, Nov 7, 2011 at 5:43 AM, Myunghwa Hwang <mhwang4@gmail.com> wrote:
Hi, Hayne!
Thanks for your answer. After trying out what you suggested (that is, commenting out the import of decomp), I found out the import of decomp was not the problem. In decomp_schur, there are two lines checking something related to rounding errors specific to a single machine as follows: eps = np.finfo(float).eps feps = numpy.finfo(single).eps
If you execute just the above lines in your application instead of importing scipy, does it hang too?
Ralf
When scipy reaches the above lines, my application hangs. I found a web document where the author encountered the same problem with these lines but in different contexts:
http://stackoverflow.com/questions/7592565/when-embedding-cpython-in-java-wh...
The discussion in the web document is not applicable to my problem. Also, the np.finfo statements seem to exist in multiple modules of scipy. Without addressing all related modules manually, would it be any other solutions?
Thanks!
--Myung-Hwa
On Sat, Nov 5, 2011 at 1:14 PM, <hayne@sympatico.ca> wrote:
I would try putting print statements inside "decomp_schur.py" since that is the module that you said is causing problems. Print out the contents of the dictionary sys.modules just before the import of decomp in "decomp_schur.py". Is 'decomp' in the dictionary? What happens if you comment-out the import of decomp in "decomp_schur.py" ? -- Cameron Hayne macdev@hayne.net
On 5-Nov-11, at 3:59 PM, Myunghwa Hwang wrote:
I am trying to run a simple django application in a cluster environment. And, my application hangs while it imports scipy.linalg, and both scipy and apache do not write out error messages. When I run my application in my local python shell, it imports scipy.linalg. But, somehow it does not when it is run by apache. So, after reading this message, please share any ideas about how to debug this problem or new solutions to address this issue or deploy my application.
Now, let me explain our current setup. 1. OS -- The server is a compute cluster where each node runs centos 6 that was installed from a clean version of centos6 minimal.2. Apache
-- Apache 2.2 was also manually installed from one of default linux repository. To be specific, it was installed from its source code together with httpd-dev. 3. Python -- Python 2.7.2 was also installed from its source code across all nodes in the cluster. Its source code was downloaded from python.org's ftp. 4. Python packages: nose, numpy, scipy -- Nose 1.1.2 was downloaded from pypi.python.org and installed from its source code. -- numpy 1.6.1 was downloaded and installed from a linux repository. When building numpy, gnu95 fortran complier was used. -- To install scipy, we installed atlas-3.8.4, lapack-3.3.1, and blas from their source code.----- atlas was from sourceforge's 3.8.4 stable version. To compile altas, gcc was used.
----- lapack and blas was obtained from netlib.org's repository. To compile the package of lapack and blas, gforan was used. ----- Finally, after exporting paths to blas, lapack, and atlas, scipy-0.9.0 was installed from its source code. scipy was obtained from sourceforge.net's repository. A note that contains the above information about software installation is attached.
All of the above were installed in the same way across all nodes in our cluster. Since I am the only user of the cluster who needs to run python web applications, I installed python virtualenv package in my local directory. Within my virtual environment, django-1.3 and pysal-1.2 (our own package) were installed. To deploy my web applications, we used mod_wsgi. mod-wsgi was compiled with python-2.7.2 and loaded into apache-2.2. My application is attached. Basically, it is a 'hello world' application that tests if numpy, scipy, and pysal can be imported. In the attached file, lines 4-9 are just adding paths to django and pysal so that apache knows where to find these packages. Also, to let apache know where to find atlas-related packages, the path to those packages was added to the LD_LIBRARY_PATH environment variable in the /etc/sysconfig/httpd file.
When I first ran my application, it just hung and wrote no message. So, across scipy.linalg modules, I added print out statements to figure out at which point the import was broken. Here is the messages I got when I imported scipy.linalg in my local python shell. • ######################## • starting linalg.__init__ • pre __init__.__doc__ • pre __init__.__version__ • pre __init__.misc • pre __init__.basic • ####################### • Starting basic • pre basic.flinalg • pre basic.lapack • pre basic.misc • pre basic.scipy.linalg • pre basic.decomp_svd • pre __init__.decomp • ################ • starting decomp • pre decomp.array et al. • pre decomp.calc_lwork • pre decomp.LinAlgError • pre decomp.get_lapack_funcs • pre decomp.get_blas_funcs • #################### • Starting blas • pre blas.scipy.linalg.fblas • pre blas.scipy.linalg.cblas • pre __init__.decomp_lu • pre __init__.decomp_cholesky • pre __init__.decomp_qr • ################# • Starting special_matrices • pre special_matrices.math • pre special_matrices.np • pre __init__.decomp_svd • pre __init__.decomp_schur • ################## • starting schur... • pre decomp_schur.misc • pre decomp_schur.LinAlgError • pre decomp_schur.get_lapack_funcs • pre decomp_schur.eigvals:**1320454147.23Fri Nov 4 17:49:07 2011 • schur testing • pre __init__.matfuncs • ##################### • Starting matfuncs • pre matfuncs. asarray et al • pre matfuncs.matrix • pre matfuncs.np • pre matfuncs.misc • pre matfuncs.basic • pre matfuncs.special_matrices • pre matfuncs.decomp • pre matfuncs.decomp_svd • pre matfuncs.decomp_schur • pre __init__.blas • pre __init__.special_matrices When scipy.linalg is successfully imported, I should get these messages. But, when my web application tried to import scipy.linalg, the output messages stop at line 41. At line 41, decomp_schur.py tries to import decomp.py. Since decomp.py was already imported at line 16, scipy ignores it and continues to import other modules in my local shell. But, somehow, in apache-mod_wsgi environment, scipy failed to ignore or reload decomp.py and seems to kill my web application. This is really odd, because python does not give any message about this error and neither does apache. apache just hangs without sending out any response. Since lapack and blas functions were imported successfully, the problem seems not related to path setup.
If anyone in the list has any insights into or experience into this kind of symptom, please share your insights and experience. In particular, debugging techniques or less-known installation/compilation problems would be helpful. I feel like I am at a dead end. So, please help me.
Thanks for reading this post. I will look forward to your responses.
-- Myung-Hwa Hwang
-- Myung-Hwa Hwang GeoDa Center School of Geographical Sciences and Urban Planning Arizona State University mhwang4@gmail.com or Myunghwa.Hwang@asu.edu
-- Myung-Hwa Hwang GeoDa Center School of Geographical Sciences and Urban Planning Arizona State University mhwang4@gmail.com or Myunghwa.Hwang@asu.edu
_______________________________________________ SciPy-User mailing list SciPy-User@scipy.org http://mail.scipy.org/mailman/listinfo/scipy-user
On Mon, Nov 7, 2011 at 11:47 AM, Ralf Gommers <ralf.gommers@googlemail.com>wrote:
On Mon, Nov 7, 2011 at 5:43 AM, Myunghwa Hwang <mhwang4@gmail.com> wrote:
Hi, Hayne!
Thanks for your answer. After trying out what you suggested (that is, commenting out the import of decomp), I found out the import of decomp was not the problem. In decomp_schur, there are two lines checking something related to rounding errors specific to a single machine as follows: eps = np.finfo(float).eps feps = numpy.finfo(single).eps
If you execute just the above lines in your application instead of importing scipy, does it hang too?
The current finfo is kind of a mess and could use a rewrite. If we truly stick to ieee it could also be reduced to a set of known tables, with maybe an exception thrown in for some of the PPC. I don't think it works for PPC at the moment in any case. Myunghwa, what hardware are you running on? <snip> Chuck
On 11/08/2011 11:58 AM, Charles R Harris wrote:
On Mon, Nov 7, 2011 at 11:47 AM, Ralf Gommers <ralf.gommers@googlemail.com <mailto:ralf.gommers@googlemail.com>> wrote:
On Mon, Nov 7, 2011 at 5:43 AM, Myunghwa Hwang <mhwang4@gmail.com <mailto:mhwang4@gmail.com>> wrote:
Hi, Hayne!
Thanks for your answer. After trying out what you suggested (that is, commenting out the import of decomp), I found out the import of decomp was not the problem. In decomp_schur, there are two lines checking something related to rounding errors specific to a single machine as follows: eps = np.finfo(float).eps feps = numpy.finfo(single).eps
If you execute just the above lines in your application instead of importing scipy, does it hang too?
The current finfo is kind of a mess and could use a rewrite. If we truly stick to ieee it could also be reduced to a set of known tables, with maybe an exception thrown in for some of the PPC. I don't think it works for PPC at the moment in any case.
Myunghwa, what hardware are you running on?
<snip>
Chuck
_______________________________________________ SciPy-User mailing list SciPy-User@scipy.org http://mail.scipy.org/mailman/listinfo/scipy-user
Also, what is the state of selinux in your centos 6 server? There is also this rather common ctypes bug that you should see in the Apache error log file (which is really important to have access to). http://stackoverflow.com/questions/3762566/occasional-ctypes-error-importing... http://bugs.python.org/issue5504 Bruce
Hi, Hayne, Charles, and Bruce! Thanks for your response. I was out of town and got into other projects. For Hayne's question, when I included only the two lines including the finfo function, my application hung. For Charles's question, all computer nodes in our server use centos 6 linux system. For hardware specifics, I know only they are all 64-bit systems. For Bruce's question, I am aware of the selinux problem. It was not the issue. Actually, my problem seemed related to how mod_wsgi handles multiple python processes. My problem was solved by including an apache directive related to mod_wsgi. Particularly, I added the following line into my apache configuration file: WSGIApplicationGroup %{GLOBAL} Here is the explanation of this apache directive: The application group name will be set to the empty string. Any WSGI applications in the global application group will always be executed within the context of the first interpreter created by Python when it is initialised. Forcing a WSGI application to run within the first interpreter can be necessary when a third party C extension module for Python has used the simplified threading API for manipulation of the Python GIL and thus will not run correctly within any additional sub interpreters created by Python. Still, I am not quite sure about the real source that caused my problem. But, it is gone now. This is all thanks to you and scipy mailing list. Again, I really appreciate your help. Myunghwa On Mon, Nov 7, 2011 at 11:47 AM, Ralf Gommers <ralf.gommers@googlemail.com>wrote:
On Mon, Nov 7, 2011 at 5:43 AM, Myunghwa Hwang <mhwang4@gmail.com> wrote:
Hi, Hayne!
Thanks for your answer. After trying out what you suggested (that is, commenting out the import of decomp), I found out the import of decomp was not the problem. In decomp_schur, there are two lines checking something related to rounding errors specific to a single machine as follows: eps = np.finfo(float).eps feps = numpy.finfo(single).eps
If you execute just the above lines in your application instead of importing scipy, does it hang too?
Ralf
When scipy reaches the above lines, my application hangs. I found a web document where the author encountered the same problem with these lines but in different contexts:
http://stackoverflow.com/questions/7592565/when-embedding-cpython-in-java-wh...
The discussion in the web document is not applicable to my problem. Also, the np.finfo statements seem to exist in multiple modules of scipy. Without addressing all related modules manually, would it be any other solutions?
Thanks!
--Myung-Hwa
On Sat, Nov 5, 2011 at 1:14 PM, <hayne@sympatico.ca> wrote:
I would try putting print statements inside "decomp_schur.py" since that is the module that you said is causing problems. Print out the contents of the dictionary sys.modules just before the import of decomp in "decomp_schur.py". Is 'decomp' in the dictionary? What happens if you comment-out the import of decomp in "decomp_schur.py" ? -- Cameron Hayne macdev@hayne.net
On 5-Nov-11, at 3:59 PM, Myunghwa Hwang wrote:
I am trying to run a simple django application in a cluster environment. And, my application hangs while it imports scipy.linalg, and both scipy and apache do not write out error messages. When I run my application in my local python shell, it imports scipy.linalg. But, somehow it does not when it is run by apache. So, after reading this message, please share any ideas about how to debug this problem or new solutions to address this issue or deploy my application.
Now, let me explain our current setup. 1. OS -- The server is a compute cluster where each node runs centos 6 that was installed from a clean version of centos6 minimal.2. Apache
-- Apache 2.2 was also manually installed from one of default linux repository. To be specific, it was installed from its source code together with httpd-dev. 3. Python -- Python 2.7.2 was also installed from its source code across all nodes in the cluster. Its source code was downloaded from python.org's ftp. 4. Python packages: nose, numpy, scipy -- Nose 1.1.2 was downloaded from pypi.python.org and installed from its source code. -- numpy 1.6.1 was downloaded and installed from a linux repository. When building numpy, gnu95 fortran complier was used. -- To install scipy, we installed atlas-3.8.4, lapack-3.3.1, and blas from their source code.----- atlas was from sourceforge's 3.8.4 stable version. To compile altas, gcc was used.
----- lapack and blas was obtained from netlib.org's repository. To compile the package of lapack and blas, gforan was used. ----- Finally, after exporting paths to blas, lapack, and atlas, scipy-0.9.0 was installed from its source code. scipy was obtained from sourceforge.net's repository. A note that contains the above information about software installation is attached.
All of the above were installed in the same way across all nodes in our cluster. Since I am the only user of the cluster who needs to run python web applications, I installed python virtualenv package in my local directory. Within my virtual environment, django-1.3 and pysal-1.2 (our own package) were installed. To deploy my web applications, we used mod_wsgi. mod-wsgi was compiled with python-2.7.2 and loaded into apache-2.2. My application is attached. Basically, it is a 'hello world' application that tests if numpy, scipy, and pysal can be imported. In the attached file, lines 4-9 are just adding paths to django and pysal so that apache knows where to find these packages. Also, to let apache know where to find atlas-related packages, the path to those packages was added to the LD_LIBRARY_PATH environment variable in the /etc/sysconfig/httpd file.
When I first ran my application, it just hung and wrote no message. So, across scipy.linalg modules, I added print out statements to figure out at which point the import was broken. Here is the messages I got when I imported scipy.linalg in my local python shell. • ######################## • starting linalg.__init__ • pre __init__.__doc__ • pre __init__.__version__ • pre __init__.misc • pre __init__.basic • ####################### • Starting basic • pre basic.flinalg • pre basic.lapack • pre basic.misc • pre basic.scipy.linalg • pre basic.decomp_svd • pre __init__.decomp • ################ • starting decomp • pre decomp.array et al. • pre decomp.calc_lwork • pre decomp.LinAlgError • pre decomp.get_lapack_funcs • pre decomp.get_blas_funcs • #################### • Starting blas • pre blas.scipy.linalg.fblas • pre blas.scipy.linalg.cblas • pre __init__.decomp_lu • pre __init__.decomp_cholesky • pre __init__.decomp_qr • ################# • Starting special_matrices • pre special_matrices.math • pre special_matrices.np • pre __init__.decomp_svd • pre __init__.decomp_schur • ################## • starting schur... • pre decomp_schur.misc • pre decomp_schur.LinAlgError • pre decomp_schur.get_lapack_funcs • pre decomp_schur.eigvals:**1320454147.23Fri Nov 4 17:49:07 2011 • schur testing • pre __init__.matfuncs • ##################### • Starting matfuncs • pre matfuncs. asarray et al • pre matfuncs.matrix • pre matfuncs.np • pre matfuncs.misc • pre matfuncs.basic • pre matfuncs.special_matrices • pre matfuncs.decomp • pre matfuncs.decomp_svd • pre matfuncs.decomp_schur • pre __init__.blas • pre __init__.special_matrices When scipy.linalg is successfully imported, I should get these messages. But, when my web application tried to import scipy.linalg, the output messages stop at line 41. At line 41, decomp_schur.py tries to import decomp.py. Since decomp.py was already imported at line 16, scipy ignores it and continues to import other modules in my local shell. But, somehow, in apache-mod_wsgi environment, scipy failed to ignore or reload decomp.py and seems to kill my web application. This is really odd, because python does not give any message about this error and neither does apache. apache just hangs without sending out any response. Since lapack and blas functions were imported successfully, the problem seems not related to path setup.
If anyone in the list has any insights into or experience into this kind of symptom, please share your insights and experience. In particular, debugging techniques or less-known installation/compilation problems would be helpful. I feel like I am at a dead end. So, please help me.
Thanks for reading this post. I will look forward to your responses.
-- Myung-Hwa Hwang
-- Myung-Hwa Hwang GeoDa Center School of Geographical Sciences and Urban Planning Arizona State University mhwang4@gmail.com or Myunghwa.Hwang@asu.edu
-- Myung-Hwa Hwang GeoDa Center School of Geographical Sciences and Urban Planning Arizona State University mhwang4@gmail.com or Myunghwa.Hwang@asu.edu
_______________________________________________ SciPy-User mailing list SciPy-User@scipy.org http://mail.scipy.org/mailman/listinfo/scipy-user
_______________________________________________ SciPy-User mailing list SciPy-User@scipy.org http://mail.scipy.org/mailman/listinfo/scipy-user
-- Myung-Hwa Hwang GeoDa Center School of Geographical Sciences and Urban Planning Arizona State University mhwang4@gmail.com or Myunghwa.Hwang@asu.edu
On Sat, Nov 5, 2011 at 8:52 PM, Myunghwa Hwang <mhwang4@gmail.com> wrote:
Hello, list!
First of all, this will be a long message, due to the complexity of our environment. So, please be patient with my question.
I am trying to run a simple django application in a cluster environment. And, my application hangs while it imports scipy.linalg, and both scipy and apache do not write out error messages. When I run my application in my local python shell, it imports scipy.linalg. But, somehow it does not when it is run by apache. So, after reading this message, please share any ideas about how to debug this problem or new solutions to address this issue or deploy my application.
Now, let me explain our current setup. 1. OS -- The server is a compute cluster where each node runs centos 6 that was installed from a clean version of centos6 minimal. 2. Apache -- Apache 2.2 is also manually installed from one of default linux repository. To be specific, it is installed from source code together with httpd-dev. 3. Python -- Python 2.7.2 is also installed from source code across all nodes in the cluster. Its source code is downloaded from python.org's ftp. 4. Python packages: nose, numpy, scipy -- Nose 1.1.2 was downloaded from pypi.python.org and installed from its source code. -- numpy 1.6.1 was downloaded and installed from a linux repository. When building numpy, gnu95 fortran complier was used. -- To install scipy, we installed atlas-3.8.4, lapack-3.3.1, and blas from their source code. ----- atlas was from sourceforge's 3.8.4 stable version. To compile altas, gcc was used. ----- lapack and blas was obtained from netlib.org's repository. To compile the package of lapack and blas, gforan was used. ----- Finally, after exporting paths to blas, lapack, and atlas, scipy-0.9.0 was installed from its source code. scipy was obtained from sourceforge.net's repository.
All of the above were installed in the same way across all nodes in our cluster. Since I am the only user of the cluster who needs to run python web applications, I installed python virtualenv package in my local directory. Within my virtual environment, django-1.3 and pysal-1.2 (our own package) were installed. To deploy my web applications, we used mod_wsgi. mod-wsgi was compiled with python-2.7.2 and loaded into apache-2.2. My application is attached. Basically, it is 'hello world' application that tests if numpy, scipy, and pysal can be imported. In the attached file, lines 4-9 are just adding paths to django and pysal so that apache knows where to find these packages. Also, to let apache know where to find atlas-related packages, the path to those packages was added to the LD_LIBRARY_PATH environment variable in the /etc/sysconfig/httpd file.
When I first ran my application, it just hung and wrote no message. So, across scipy.linalg modules, I added print out statements to figure out at which point the import was broken. Here is the messages I got when I imported scipy.linalg in my local python shell.
1. ######################## 2. starting linalg.__init__ 3. pre __init__.__doc__ 4. pre __init__.__version__ 5. pre __init__.misc 6. pre __init__.basic 7. ####################### 8. Starting basic 9. pre basic.flinalg 10. pre basic.lapack 11. pre basic.misc 12. pre basic.scipy.linalg 13. pre basic.decomp_svd 14. pre __init__.decomp 15. ################ 16. starting decomp 17. pre decomp.array et al. 18. pre decomp.calc_lwork 19. pre decomp.LinAlgError 20. pre decomp.get_lapack_funcs 21. pre decomp.get_blas_funcs 22. #################### 23. Starting blas 24. pre blas.scipy.linalg.fblas 25. pre blas.scipy.linalg.cblas 26. pre __init__.decomp_lu 27. pre __init__.decomp_cholesky 28. pre __init__.decomp_qr 29. ################# 30. Starting special_matrices 31. pre special_matrices.math 32. pre special_matrices.np 33. pre __init__.decomp_svd 34. pre __init__.decomp_schur 35. ################## 36. starting schur... 37. pre decomp_schur.misc 38. pre decomp_schur.LinAlgError 39. pre decomp_schur.get_lapack_funcs 40. pre decomp_schur.eigvals:1320454147.23Fri Nov 4 17:49:07 2011 41. schur testing 42. pre __init__.matfuncs 43. ##################### 44. Starting matfuncs 45. pre matfuncs. asarray et al 46. pre matfuncs.matrix 47. pre matfuncs.np 48. pre matfuncs.misc 49. pre matfuncs.basic 50. pre matfuncs.special_matrices 51. pre matfuncs.decomp 52. pre matfuncs.decomp_svd 53. pre matfuncs.decomp_schur 54. pre __init__.blas 55. pre __init__.special_matrices
When scipy.linalg is successfully imported, I should get these messages. But, when my web application tried to import scipy.linalg, the output messages stop at line 41. At line 41, decomp_schur.py tries to import decomp.py. Since decomp.py was already imported at line 16, scipy ignores it and continues to import other modules in my local shell. But, somehow, in apache-mod_wsgi environment, scipy failed to ignore or reload decomp.py and seems to kill my web application. This is really odd, because python does not give any message about this error and neither does apache. apache just hangs without sending out any response. Since lapack and blas functions were imported successfully, the problem seems not related to path setup.
If anyone in the list has any insights into or experience into this kind of symptom, please share your insights and experience. In particular, debugging techniques or less-known installation/compilation problems would be helpful. I feel like I am at a dead end. So, please help me.
Thanks for reading this post. I will look forward to yo
Looking at linalg/__init__.py the register_func calls are an obvious candidate for causing this strange issue. of the few functions left in decomp_schur.py, norm() and dot() have both gone through register_func, which messes with the call stack. Could you comment out all calls to that function and see if that helps? Replace functions that then become unavailable with numpy ones of the same name where needed. Cheers, Ralf
-- Myung-Hwa Hwang GeoDa Center School of Geographical Sciences and Urban Planning Arizona State University mhwang4@gmail.com or Myunghwa.Hwang@asu.edu
_______________________________________________ SciPy-User mailing list SciPy-User@scipy.org http://mail.scipy.org/mailman/listinfo/scipy-user
On Sun, Nov 6, 2011 at 3:49 PM, Ralf Gommers <ralf.gommers@googlemail.com> wrote:
On Sat, Nov 5, 2011 at 8:52 PM, Myunghwa Hwang <mhwang4@gmail.com> wrote:
Hello, list! First of all, this will be a long message, due to the complexity of our environment. So, please be patient with my question. I am trying to run a simple django application in a cluster environment. And, my application hangs while it imports scipy.linalg, and both scipy and apache do not write out error messages. When I run my application in my local python shell, it imports scipy.linalg. But, somehow it does not when it is run by apache. So, after reading this message, please share any ideas about how to debug this problem or new solutions to address this issue or deploy my application. Now, let me explain our current setup. 1. OS -- The server is a compute cluster where each node runs centos 6 that was installed from a clean version of centos6 minimal. 2. Apache -- Apache 2.2 is also manually installed from one of default linux repository. To be specific, it is installed from source code together with httpd-dev. 3. Python -- Python 2.7.2 is also installed from source code across all nodes in the cluster. Its source code is downloaded from python.org's ftp. 4. Python packages: nose, numpy, scipy -- Nose 1.1.2 was downloaded from pypi.python.org and installed from its source code. -- numpy 1.6.1 was downloaded and installed from a linux repository. When building numpy, gnu95 fortran complier was used. -- To install scipy, we installed atlas-3.8.4, lapack-3.3.1, and blas from their source code. ----- atlas was from sourceforge's 3.8.4 stable version. To compile altas, gcc was used. ----- lapack and blas was obtained from netlib.org's repository. To compile the package of lapack and blas, gforan was used. ----- Finally, after exporting paths to blas, lapack, and atlas, scipy-0.9.0 was installed from its source code. scipy was obtained from sourceforge.net's repository. All of the above were installed in the same way across all nodes in our cluster. Since I am the only user of the cluster who needs to run python web applications, I installed python virtualenv package in my local directory. Within my virtual environment, django-1.3 and pysal-1.2 (our own package) were installed. To deploy my web applications, we used mod_wsgi. mod-wsgi was compiled with python-2.7.2 and loaded into apache-2.2. My application is attached. Basically, it is 'hello world' application that tests if numpy, scipy, and pysal can be imported. In the attached file, lines 4-9 are just adding paths to django and pysal so that apache knows where to find these packages. Also, to let apache know where to find atlas-related packages, the path to those packages was added to the LD_LIBRARY_PATH environment variable in the /etc/sysconfig/httpd file. When I first ran my application, it just hung and wrote no message. So, across scipy.linalg modules, I added print out statements to figure out at which point the import was broken. Here is the messages I got when I imported scipy.linalg in my local python shell.
######################## starting linalg.__init__ pre __init__.__doc__ pre __init__.__version__ pre __init__.misc pre __init__.basic ####################### Starting basic pre basic.flinalg pre basic.lapack pre basic.misc pre basic.scipy.linalg pre basic.decomp_svd pre __init__.decomp ################ starting decomp pre decomp.array et al. pre decomp.calc_lwork pre decomp.LinAlgError pre decomp.get_lapack_funcs pre decomp.get_blas_funcs #################### Starting blas pre blas.scipy.linalg.fblas pre blas.scipy.linalg.cblas pre __init__.decomp_lu pre __init__.decomp_cholesky pre __init__.decomp_qr ################# Starting special_matrices pre special_matrices.math pre special_matrices.np pre __init__.decomp_svd pre __init__.decomp_schur ################## starting schur... pre decomp_schur.misc pre decomp_schur.LinAlgError pre decomp_schur.get_lapack_funcs pre decomp_schur.eigvals:1320454147.23Fri Nov 4 17:49:07 2011 schur testing pre __init__.matfuncs ##################### Starting matfuncs pre matfuncs. asarray et al pre matfuncs.matrix pre matfuncs.np pre matfuncs.misc pre matfuncs.basic pre matfuncs.special_matrices pre matfuncs.decomp pre matfuncs.decomp_svd pre matfuncs.decomp_schur pre __init__.blas pre __init__.special_matrices
When scipy.linalg is successfully imported, I should get these messages. But, when my web application tried to import scipy.linalg, the output messages stop at line 41. At line 41, decomp_schur.py tries to import decomp.py. Since decomp.py was already imported at line 16, scipy ignores it and continues to import other modules in my local shell. But, somehow, in apache-mod_wsgi environment, scipy failed to ignore or reload decomp.py and seems to kill my web application. This is really odd, because python does not give any message about this error and neither does apache. apache just hangs without sending out any response. Since lapack and blas functions were imported successfully, the problem seems not related to path setup. If anyone in the list has any insights into or experience into this kind of symptom, please share your insights and experience. In particular, debugging techniques or less-known installation/compilation problems would be helpful. I feel like I am at a dead end. So, please help me. Thanks for reading this post. I will look forward to yo
Looking at linalg/__init__.py the register_func calls are an obvious candidate for causing this strange issue. of the few functions left in decomp_schur.py, norm() and dot() have both gone through register_func, which messes with the call stack. Could you comment out all calls to that function and see if that helps? Replace functions that then become unavailable with numpy ones of the same name where needed.
Cheers, Ralf
-- Myung-Hwa Hwang GeoDa Center School of Geographical Sciences and Urban Planning Arizona State University mhwang4@gmail.com or Myunghwa.Hwang@asu.edu
_______________________________________________ SciPy-User mailing list SciPy-User@scipy.org http://mail.scipy.org/mailman/listinfo/scipy-user
Also, if you provide some installation instruction, I might try to run it. But I do wonder if this is due to your virtual environment and associated paths once called via Apache. Bruce
Ralf and Bruce, Thanks for your answer. I just found that the problem area is the following lines: eps = np.finfo(float).eps feps = np.finfo(single).eps When scipy hits these lines, my app hangs. Do you have any ideas about why these lines raise the issue? Thanks, Myung-Hwa On Sun, Nov 6, 2011 at 2:49 PM, Ralf Gommers <ralf.gommers@googlemail.com>wrote:
On Sat, Nov 5, 2011 at 8:52 PM, Myunghwa Hwang <mhwang4@gmail.com> wrote:
Hello, list!
First of all, this will be a long message, due to the complexity of our environment. So, please be patient with my question.
I am trying to run a simple django application in a cluster environment. And, my application hangs while it imports scipy.linalg, and both scipy and apache do not write out error messages. When I run my application in my local python shell, it imports scipy.linalg. But, somehow it does not when it is run by apache. So, after reading this message, please share any ideas about how to debug this problem or new solutions to address this issue or deploy my application.
Now, let me explain our current setup. 1. OS -- The server is a compute cluster where each node runs centos 6 that was installed from a clean version of centos6 minimal. 2. Apache -- Apache 2.2 is also manually installed from one of default linux repository. To be specific, it is installed from source code together with httpd-dev. 3. Python -- Python 2.7.2 is also installed from source code across all nodes in the cluster. Its source code is downloaded from python.org's ftp. 4. Python packages: nose, numpy, scipy -- Nose 1.1.2 was downloaded from pypi.python.org and installed from its source code. -- numpy 1.6.1 was downloaded and installed from a linux repository. When building numpy, gnu95 fortran complier was used. -- To install scipy, we installed atlas-3.8.4, lapack-3.3.1, and blas from their source code. ----- atlas was from sourceforge's 3.8.4 stable version. To compile altas, gcc was used. ----- lapack and blas was obtained from netlib.org's repository. To compile the package of lapack and blas, gforan was used. ----- Finally, after exporting paths to blas, lapack, and atlas, scipy-0.9.0 was installed from its source code. scipy was obtained from sourceforge.net's repository.
All of the above were installed in the same way across all nodes in our cluster. Since I am the only user of the cluster who needs to run python web applications, I installed python virtualenv package in my local directory. Within my virtual environment, django-1.3 and pysal-1.2 (our own package) were installed. To deploy my web applications, we used mod_wsgi. mod-wsgi was compiled with python-2.7.2 and loaded into apache-2.2. My application is attached. Basically, it is 'hello world' application that tests if numpy, scipy, and pysal can be imported. In the attached file, lines 4-9 are just adding paths to django and pysal so that apache knows where to find these packages. Also, to let apache know where to find atlas-related packages, the path to those packages was added to the LD_LIBRARY_PATH environment variable in the /etc/sysconfig/httpd file.
When I first ran my application, it just hung and wrote no message. So, across scipy.linalg modules, I added print out statements to figure out at which point the import was broken. Here is the messages I got when I imported scipy.linalg in my local python shell.
1. ######################## 2. starting linalg.__init__ 3. pre __init__.__doc__ 4. pre __init__.__version__ 5. pre __init__.misc 6. pre __init__.basic 7. ####################### 8. Starting basic 9. pre basic.flinalg 10. pre basic.lapack 11. pre basic.misc 12. pre basic.scipy.linalg 13. pre basic.decomp_svd 14. pre __init__.decomp 15. ################ 16. starting decomp 17. pre decomp.array et al. 18. pre decomp.calc_lwork 19. pre decomp.LinAlgError 20. pre decomp.get_lapack_funcs 21. pre decomp.get_blas_funcs 22. #################### 23. Starting blas 24. pre blas.scipy.linalg.fblas 25. pre blas.scipy.linalg.cblas 26. pre __init__.decomp_lu 27. pre __init__.decomp_cholesky 28. pre __init__.decomp_qr 29. ################# 30. Starting special_matrices 31. pre special_matrices.math 32. pre special_matrices.np 33. pre __init__.decomp_svd 34. pre __init__.decomp_schur 35. ################## 36. starting schur... 37. pre decomp_schur.misc 38. pre decomp_schur.LinAlgError 39. pre decomp_schur.get_lapack_funcs 40. pre decomp_schur.eigvals:1320454147.23Fri Nov 4 17:49:07 2011 41. schur testing 42. pre __init__.matfuncs 43. ##################### 44. Starting matfuncs 45. pre matfuncs. asarray et al 46. pre matfuncs.matrix 47. pre matfuncs.np 48. pre matfuncs.misc 49. pre matfuncs.basic 50. pre matfuncs.special_matrices 51. pre matfuncs.decomp 52. pre matfuncs.decomp_svd 53. pre matfuncs.decomp_schur 54. pre __init__.blas 55. pre __init__.special_matrices
When scipy.linalg is successfully imported, I should get these messages. But, when my web application tried to import scipy.linalg, the output messages stop at line 41. At line 41, decomp_schur.py tries to import decomp.py. Since decomp.py was already imported at line 16, scipy ignores it and continues to import other modules in my local shell. But, somehow, in apache-mod_wsgi environment, scipy failed to ignore or reload decomp.py and seems to kill my web application. This is really odd, because python does not give any message about this error and neither does apache. apache just hangs without sending out any response. Since lapack and blas functions were imported successfully, the problem seems not related to path setup.
If anyone in the list has any insights into or experience into this kind of symptom, please share your insights and experience. In particular, debugging techniques or less-known installation/compilation problems would be helpful. I feel like I am at a dead end. So, please help me.
Thanks for reading this post. I will look forward to yo
Looking at linalg/__init__.py the register_func calls are an obvious candidate for causing this strange issue. of the few functions left in decomp_schur.py, norm() and dot() have both gone through register_func, which messes with the call stack. Could you comment out all calls to that function and see if that helps? Replace functions that then become unavailable with numpy ones of the same name where needed.
Cheers, Ralf
-- Myung-Hwa Hwang GeoDa Center School of Geographical Sciences and Urban Planning Arizona State University mhwang4@gmail.com or Myunghwa.Hwang@asu.edu
_______________________________________________ SciPy-User mailing list SciPy-User@scipy.org http://mail.scipy.org/mailman/listinfo/scipy-user
_______________________________________________ SciPy-User mailing list SciPy-User@scipy.org http://mail.scipy.org/mailman/listinfo/scipy-user
-- Myung-Hwa Hwang GeoDa Center School of Geographical Sciences and Urban Planning Arizona State University mhwang4@gmail.com or Myunghwa.Hwang@asu.edu
participants (5)
-
Bruce Southey -
Charles R Harris -
hayne＠sympatico.ca -
Myunghwa Hwang -
Ralf Gommers