[SciPy-User] scipy import problem in apache-mod_wsgi environment

[hayne at sympatico.ca]

I would try putting print statements inside "decomp_schur.py" since  
that is the module that you said is causing problems.
Print out the contents of the dictionary sys.modules just before the  
import of decomp in "decomp_schur.py". Is 'decomp' in the dictionary?
What happens if you comment-out the import of decomp in  
"decomp_schur.py"  ?
--
Cameron Hayne
macdev at hayne.net


On 5-Nov-11, at 3:59 PM, Myunghwa Hwang wrote:
> I am trying to run a simple django application in a cluster  
> environment.
> And, my application hangs while it imports scipy.linalg, and both  
> scipy and apache do not write out error messages.
> When I run my application in my local python shell, it imports  
> scipy.linalg. But, somehow it does not when it is run by apache.
> So, after reading this message, please share any ideas about how to  
> debug this problem or new solutions to address this issue or deploy  
> my application.
>
> Now, let me explain our current setup.
> 1. OS
> -- The server is a compute cluster where each node runs centos 6  
> that was installed from a clean version of centos6 minimal. 
> 2. Apache
> -- Apache 2.2 was also manually installed from one of default linux  
> repository. 
>   To be specific, it was installed from its source code together  
> with httpd-dev.
> 3. Python
> -- Python 2.7.2 was also installed from its source code across all  
> nodes in the cluster. Its source code was downloaded from  
> python.org's ftp.
> 4. Python packages: nose, numpy, scipy
> -- Nose 1.1.2 was downloaded from pypi.python.org and installed from  
> its source code.
> -- numpy 1.6.1 was downloaded and installed from a linux repository.  
> When building numpy, gnu95 fortran complier was used.
> -- To install scipy, we installed atlas-3.8.4, lapack-3.3.1, and  
> blas from their source code. 
> ----- atlas was from sourceforge's 3.8.4 stable version. To compile  
> altas, gcc was used.
> ----- lapack and blas was obtained from netlib.org's repository. To  
> compile the package of lapack and blas, gforan was used.
> ----- Finally, after exporting paths to blas, lapack, and atlas,  
> scipy-0.9.0 was installed from its source code.
>       scipy was obtained from sourceforge.net's repository.
> A note that contains the above information about software  
> installation is attached.
>
> All of the above were installed in the same way across all nodes in  
> our cluster.
> Since I am the only user of the cluster who needs to run python web  
> applications,
> I installed python virtualenv package in my local directory.
> Within my virtual environment, django-1.3 and pysal-1.2 (our own  
> package) were installed.
> To deploy my web applications, we used mod_wsgi.
> mod-wsgi was compiled with python-2.7.2 and loaded into apache-2.2.
> My application is attached. Basically, it is a 'hello world'  
> application that tests if numpy, scipy, and pysal can be imported.
> In the attached file, lines 4-9 are just adding paths to django and  
> pysal so that apache knows where to find these packages.
> Also, to let apache know where to find atlas-related packages, the  
> path to those packages was added to the LD_LIBRARY_PATH environment  
> variable in the /etc/sysconfig/httpd file.
>
> When I first ran my application, it just hung and wrote no message.
> So, across scipy.linalg modules, I added print out statements to  
> figure out at which point the import was broken.
> Here is the messages I got when I imported scipy.linalg in my local  
> python shell.
> 	• ########################
> 	• starting linalg.__init__
> 	• pre __init__.__doc__
> 	• pre __init__.__version__
> 	• pre __init__.misc
> 	• pre __init__.basic
> 	• #######################
> 	• Starting basic
> 	• pre basic.flinalg
> 	• pre basic.lapack
> 	• pre basic.misc
> 	• pre basic.scipy.linalg
> 	• pre basic.decomp_svd
> 	• pre __init__.decomp
> 	• ################
> 	• starting decomp
> 	• pre decomp.array et al.
> 	• pre decomp.calc_lwork
> 	• pre decomp.LinAlgError
> 	• pre decomp.get_lapack_funcs
> 	• pre decomp.get_blas_funcs
> 	• ####################
> 	• Starting blas
> 	• pre blas.scipy.linalg.fblas
> 	• pre blas.scipy.linalg.cblas
> 	• pre __init__.decomp_lu
> 	• pre __init__.decomp_cholesky
> 	• pre __init__.decomp_qr
> 	• #################
> 	• Starting special_matrices
> 	• pre special_matrices.math
> 	• pre special_matrices.np
> 	• pre __init__.decomp_svd
> 	• pre __init__.decomp_schur
> 	• ##################
> 	• starting schur...
> 	• pre decomp_schur.misc
> 	• pre decomp_schur.LinAlgError
> 	• pre decomp_schur.get_lapack_funcs
> 	• pre decomp_schur.eigvals:1320454147.23Fri Nov  4 17:49:07 2011
> 	• schur testing
> 	• pre __init__.matfuncs
> 	• #####################
> 	• Starting matfuncs
> 	• pre matfuncs. asarray et al
> 	• pre matfuncs.matrix
> 	• pre matfuncs.np
> 	• pre matfuncs.misc
> 	• pre matfuncs.basic
> 	• pre matfuncs.special_matrices
> 	• pre matfuncs.decomp
> 	• pre matfuncs.decomp_svd
> 	• pre matfuncs.decomp_schur
> 	• pre __init__.blas
> 	• pre __init__.special_matrices
> When scipy.linalg is successfully imported, I should get these  
> messages.
> But, when my web application tried to import scipy.linalg, the  
> output messages stop at line 41.
> At line 41, decomp_schur.py tries to import decomp.py. Since  
> decomp.py was already imported at line 16, scipy ignores it and  
> continues to import other modules in my local shell.
> But, somehow, in apache-mod_wsgi environment, scipy failed to ignore  
> or reload decomp.py and seems to kill my web application.
> This is really odd, because python does not give any message about  
> this error and neither does apache. apache just hangs without  
> sending out any response.
> Since lapack and blas functions were imported successfully, the  
> problem seems not related to path setup.
>
> If anyone in the list has any insights into or experience into this  
> kind of symptom,
> please share your insights and experience. In particular, debugging  
> techniques or less-known installation/compilation problems would be  
> helpful.
> I feel like I am at a dead end. So, please help me.
>
> Thanks for reading this post.
> I will look forward to your responses.
>
> -- Myung-Hwa Hwang
>
> -- 
> Myung-Hwa Hwang
> GeoDa Center
> School of Geographical Sciences and Urban Planning
> Arizona State University
> mhwang4 at gmail.com or Myunghwa.Hwang at asu.edu