Mailman 3 September 2010 - SciPy-User

problems configuring scipy (0.8.0) from numpy (1.5.0) in a itanium cluster
by Miguel Pan Fidalgo Sept. 27, 2010

Sept. 27, 2010

Technical information ++++++++++++++++++++++ $ python -c 'import sys;print sys.version' 2.4.2 (#1, Apr 13 2007, 16:14:04) [GCC 4.1.0 (SUSE Linux)] $ module load numpy/1.5.0 $ module list Currently Loaded Modulefiles: 1) icc/11.1.056 3) mkl/10.2.2 5) nose/0.11.2 7) numpy/1.5.0 2) ifort/11.1.056 4) fftw/3.2.2 6) SuiteSparse/3.4.0 $ python -c 'import numpy;print numpy.__version__' 1.5.0 $ uname -a Linux cn002 2.6.16.53-0.8_SFS2.3_0-default #1 SMP Mon Nov 5 14:15:47 CET 2007 ia64 ia64 ia64 GNU/Linux numpy-1.5.0 +++++++++++++++++ I have installed numpy-1.5.0 in our Itanium cluster. This is my site.cfg file: $ cat /opt/cesga/numpy-1.5.0/src_numpy-1.5.0/site.cfg [DEFAULT] library_dirs = /opt/cesga/fftw-3.2.2/lib:/opt/cesga/intel/intel11.0/Compiler/11.1/056/mkl/lib/64:/opt/cesga/SuiteSparse-3.4.0/lib include_dirs = /opt/cesga/fftw-3.2.2/include:/opt/cesga/intel/intel11.0/Compiler/11.1/056/mkl/include:/opt/cesga/SuiteSparse-3.4.0/include # UMFPACK # ------- [amd] amd_libs = amd [umfpack] umfpack_libs = umfpack # FFT libraries # ------------- [fftw] libraries = fftw3 # MKL # ---- [mkl] library_dirs = /opt/cesga/intel/intel11.0/Compiler/11.1/056/mkl/lib/64 lapack_libs = mkl_lapack mkl_libs = mkl_solver_lp64, mkl_intel_lp64, mkl_intel_thread, mkl_i2p, mkl_core These are the hacks: $ diff system_info.py.original system_info.py 818c818 < _lib_mkl = ['mkl','vml','guide'] --- > _lib_mkl = ['mkl','mkl_lapack','guide'] $ diff intelccompiler.py.original intelccompiler.py 28a29 > cc_exe=cc_exe + ' -liomp5 -lpthread -fPIC' scipy-0.8.0 +++++++++++++++ When I try to install scipy-0.8.0, the configuration process fails: $ tar zxvf scipy-0.8.0.tar.gz $ mv scipy-0.8.0 src $ cp -va run.sh src/ $ cd src/ $ cat run.sh #!/bin/bash module load numpy/1.5.0 python setup.py config_fc --fcompiler=intele #install --prefix=/sfs/home/cesga/mpan/scipy-0.8.0 $ ./run.sh Warning: No configuration returned, assuming unavailable.blas_opt_info: blas_mkl_info: FOUND: libraries = ['mkl_solver_lp64', 'mkl_intel_lp64', 'mkl_intel_thread', 'mkl_i2p', 'mkl_core', 'pthread'] library_dirs = ['/opt/cesga/intel/intel11.0/Compiler/11.1/056/mkl/lib/64'] define_macros = [('SCIPY_MKL_H', None)] include_dirs = ['/opt/cesga/fftw-3.2.2/include', '/opt/cesga/intel/intel11.0/Compiler/11.1/056/mkl/include', '/opt/cesga/SuiteSparse-3.4.0/include'] FOUND: libraries = ['mkl_solver_lp64', 'mkl_intel_lp64', 'mkl_intel_thread', 'mkl_i2p', 'mkl_core', 'pthread'] library_dirs = ['/opt/cesga/intel/intel11.0/Compiler/11.1/056/mkl/lib/64'] define_macros = [('SCIPY_MKL_H', None)] include_dirs = ['/opt/cesga/fftw-3.2.2/include', '/opt/cesga/intel/intel11.0/Compiler/11.1/056/mkl/include', '/opt/cesga/SuiteSparse-3.4.0/include'] lapack_opt_info: lapack_mkl_info: mkl_info: FOUND: libraries = ['mkl_solver_lp64', 'mkl_intel_lp64', 'mkl_intel_thread', 'mkl_i2p', 'mkl_core', 'pthread'] library_dirs = ['/opt/cesga/intel/intel11.0/Compiler/11.1/056/mkl/lib/64'] define_macros = [('SCIPY_MKL_H', None)] include_dirs = ['/opt/cesga/fftw-3.2.2/include', '/opt/cesga/intel/intel11.0/Compiler/11.1/056/mkl/include', '/opt/cesga/SuiteSparse-3.4.0/include'] FOUND: libraries = ['mkl_lapack', 'mkl_solver_lp64', 'mkl_intel_lp64', 'mkl_intel_thread', 'mkl_i2p', 'mkl_core', 'pthread'] library_dirs = ['/opt/cesga/intel/intel11.0/Compiler/11.1/056/mkl/lib/64'] define_macros = [('SCIPY_MKL_H', None)] include_dirs = ['/opt/cesga/fftw-3.2.2/include', '/opt/cesga/intel/intel11.0/Compiler/11.1/056/mkl/include', '/opt/cesga/SuiteSparse-3.4.0/include'] FOUND: libraries = ['mkl_lapack', 'mkl_solver_lp64', 'mkl_intel_lp64', 'mkl_intel_thread', 'mkl_i2p', 'mkl_core', 'pthread'] library_dirs = ['/opt/cesga/intel/intel11.0/Compiler/11.1/056/mkl/lib/64'] define_macros = [('SCIPY_MKL_H', None)] include_dirs = ['/opt/cesga/fftw-3.2.2/include', '/opt/cesga/intel/intel11.0/Compiler/11.1/056/mkl/include', '/opt/cesga/SuiteSparse-3.4.0/include'] umfpack_info: libraries umfpack not found in /opt/cesga/fftw-3.2.2/lib libraries umfpack not found in /opt/cesga/intel/intel11.0/Compiler/11.1/056/mkl/lib/64 amd_info: libraries amd not found in /opt/cesga/fftw-3.2.2/lib libraries amd not found in /opt/cesga/intel/intel11.0/Compiler/11.1/056/mkl/lib/64 FOUND: libraries = ['amd'] library_dirs = ['/opt/cesga/SuiteSparse-3.4.0/lib'] swig_opts = ['-I/opt/cesga/SuiteSparse-3.4.0/include'] define_macros = [('SCIPY_AMD_H', None)] include_dirs = ['/opt/cesga/SuiteSparse-3.4.0/include'] FOUND: libraries = ['umfpack', 'amd'] library_dirs = ['/opt/cesga/SuiteSparse-3.4.0/lib'] swig_opts = ['-I/opt/cesga/SuiteSparse-3.4.0/include', '- I/opt/cesga/SuiteSparse-3.4.0/include'] define_macros = [('SCIPY_UMFPACK_H', None), ('SCIPY_AMD_H', None)] include_dirs = ['/opt/cesga/SuiteSparse-3.4.0/include'] running config_fc unifing config_fc, config, build_clib, build_ext, build commands --fcompiler options the warning (first line) implies that BLAS primitives have to be recompiled. At the end the installation ends properly... this is not the point, but scipy doesn't use MKL BLAS primitives. BUT according to numpy... blas_mkl and blas_opt should be the same (like lapack_mkl and lapack_opt): $ module load numpy/1.5.0 $ python Python 2.4.2 (#1, Apr 13 2007, 16:14:04) [GCC 4.1.0 (SUSE Linux)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from numpy.distutils.system_info import get_info >>> get_info('blas_mkl') {'libraries': ['mkl_solver_lp64', 'mkl_intel_lp64', 'mkl_intel_thread', 'mkl_i2p', 'mkl_core', 'pthread'], 'library_dirs': ['/opt/cesga/intel/intel11.0/Compiler/11.1/056/mkl/lib/64'], 'define_macros': [('SCIPY_MKL_H', None)], 'include_dirs': ['/opt/cesga/fftw-3.2.2/include', '/opt/cesga/intel/intel11.0/Compiler/11.1/056/mkl/include', '/opt/cesga/SuiteSparse-3.4.0/include']} >>> get_info('blas_opt') {'libraries': ['mkl_solver_lp64', 'mkl_intel_lp64', 'mkl_intel_thread', 'mkl_i2p', 'mkl_core', 'pthread'], 'library_dirs': ['/opt/cesga/intel/intel11.0/Compiler/11.1/056/mkl/lib/64'], 'define_macros': [('SCIPY_MKL_H', None)], 'include_dirs': ['/opt/cesga/fftw-3.2.2/include', '/opt/cesga/intel/intel11.0/Compiler/11.1/056/mkl/include', '/opt/cesga/SuiteSparse-3.4.0/include']} >>> There is no need to recompile BLAS!!!! any idea or suggestion ? WHAT I'm doing wrong ? -- Atentamente, Miguel Pan Fidalgo (Applications Technician) mail: mpan at cesga.es web: http://www.cesga.es Avda. de Vigo s/n 15705, Santiago de Compostela Telf.: +34 981 569810 - Fax: 981 594616 -------------------------------------------------------------------------

1 0

scikits.cuda - initial public release
by Lev Givon Sept. 26, 2010

Sept. 26, 2010

I recently released a scikit containing wrappers for functions in CUBLAS, CUFFT, and the free-of-charge (i.e., non-premium) version of the CULA library [1], as well as some higher-level numpy-like functions that make use of them. The wrappers are ctypes-based [2] and do not attempt to be exhaustive (primarily because of the research needs of myself and my colleagues). The current release of the scikit is available here: http://pypi.python.org/pypi/scikits.cuda/ Feedback welcome. L.G. [1] http://www.culatools.com [2] and make use of some the ideas in PARRET: http://www.mathcs.emory.edu/~yfan/PARRET/doc/index.html

3 5

Re: [SciPy-User] Installation of SciPy 0.8.0 for Python 2.7 on MacOS 10.6
by Hans-Martin v. Gaudecker Sept. 26, 2010

Sept. 26, 2010

On 25 Sep 2010, at 08:13, Ralf Gommers <ralf.gommers(a)googlemail.com> wrote: > Which Python are you using? Setting "-arch x86_64" should only be done if > you use a home-built 64-bit capable Python or Apple Python, not for the > Python from python.org. Okay, I guess I thought that since python.org now provides a 64-bit version alongside the 32-bit one, this was the way to go when it failed without the flags... > Are you using numpy 1.5.0? If so, try deleting 'ppc64' from > distutils/fcompiler/gnu.py, that may fix the architecture errors. Also, > don't use "sudo" to install. Many thanks, I would not have been able to find that out myself... For the record, changing line 257 of that file from: for arch in ["ppc", "i686", "x86_64", "ppc64"]: to: for arch in ["i686", "x86_64"]: did the job for me, together with the standard "python setup.py install". All tests pass: >>> scipy.test() Ran 4405 tests in 54.433s OK (KNOWNFAIL=19, SKIP=39) <nose.result.TextTestResult run=4405 errors=0 failures=0> If anybody is interested: I could not get it to work using numscons. On the MacBook Pro, the "Exception: Could not find F77 BLAS, needed for integrate package:" error showed up again. On the Mac Pro, it compiled, but the architecture error creeped back up. Cheerio, HM

2 1

64 bit on Mac?
by Keflavich Sept. 26, 2010

Sept. 26, 2010

Hi folks, I'd really like to work with some of the large images I have; I work on a galactic plane survey that has lots of images with ~10^8 floating point pixels. I'm also using a mac. Unfortunately I can't get a 64-bit version of python compiled and google has been unhelpful in resolving the problem. Is there a workaround to get 64 bit numpy working without 64 bit python installed? I doubt that, but if anyone has tips I'd appreciate them. On a related note, I also use a linux machine with 64 bit python installed, but it crashes with large arrays giving a segfault where my mac spits out the "ValueError: dimensions too large." error. Any ideas there? Thanks, Adam

10 17

scipy install issues
by Stephen Webb Sept. 26, 2010

Sept. 26, 2010

I recently had to rebuild my entire Python library from scratch after doing some very foolish things. After installing SciPy on my Mac (OS X 10.6) using the Fink scipy-26 version, I had no errors after the install (I had to use mv /usr/library /usr/library.moved to get around a gcc issue while installing) I still cannot import scipy through python26. I can import numpy without a problem. For reference: uname -a : Darwin Macintosh-4.local 10.4.0 Darwin Kernel Version 10.4.0: Fri Apr 23 18:28:53 PDT 2010; root:xnu-1504.7.4~1/RELEASE_I386 i386 Any help would be greatly appreciated. Best, Stephen Webb sdavis.webb(a)gmail.com

2 1

Finding Multiple Zeros of a Function within Interval
by Lorenzo Isella Sept. 25, 2010

Sept. 25, 2010

Dear All, I am looking for an algorithm able to find the (possibly more than one) zeros of a nonlinear function f(x,y,z) for x in [a,b] (maybe bad notation: let us say that y and z act as parameters since the only variable I can really tune is x) which does not require f(a)f(b)<0. I had a look at the optimization module in scipy, but it seems to be that most of the functions would return the first zero they can find and then stop. Any suggestion is really appreciated. Cheers Lorenzo

2 1

leastsq not converging to optimal solution, fmin won't accept parameters
by Joses Ho Sept. 25, 2010

Sept. 25, 2010

Hi, I am trying to fit to my qPCR data an exponential function where each successive y-value depends on the previous v-value in the following fashion: y[n] = y[n-1] + k ln (1 + (y[n-1]/k) + b where y[n] = current y-value being evaluated and y[n-1] = previous y-value I therefore have three values to fit - y0 (starting value of y), k, and b. I define a residuals function where I compute 20 values of y, given an initial estimate of y0, k and b, and then subtract these from my 20 measured values of y. I have tried to use scipy.optimize.leastsq on the above residuals function, but it doesn't seem to be converging on the correct values of y0,k and b. I have tried scipy.optimize.fmin on the same function, but I get the following error: ...fsim[0] = func(x0) ...ValueError: setting an array element with a sequence. which seems to me that it refuses to accept the x0 array I give to it, consisting of [y0,k,b]. Basically, I am trying to implement the recent method of Boggy and Woolf (PLoS 2010) [doi:10.1371/ journal.pone.0012355] in Python. I am using SciPy 0.8.0b1 and NumPy 1.4.0 on Python 2.6.5 of the Enthought Python Distribution 6.2-2. Any help here would be very much appreciated! Thanks Joses --- This email is confidential and may be privileged. If you are not the intended recipient, please delete it and notify the sender immediately. Do not copy or use it for any purpose, or disclose its contents to any other person.

2 1

scipy.linalg.lstsq
by Tim Valenta Sept. 25, 2010

Sept. 25, 2010

Hello all, Longtime Python user here. I'm a computer science guy, went through mostly calculus and discrete math courses-- never much in pure statistics. Which leads me into my dilemma: I'm doing least-square linear regression on a scatter plot of points, where several inputs lead to one output. That is, it's not just one system of x,y coordinates, but instead several parallel systems something more like (x1,y) (x2, y) (x3, y), upwards to (xn, y). I'm attempting to get my company off of a severely broken calculation engine, but throwing my data at linalg.lstsq or stats.linregress isn't just a magic one-step fix to my desired solution. Basically, the company ends up using data from Microsoft Office 2003 Excel, which can do hardly-configurable regression calculation on a matrix of data, each row formatted like y, x1, x2, x3 (to follow my example given above). It spits out all kinds of variables and data, and the company draws on the calculated 'coefficients' to make a pie graph, the coefficients taken as percentages. The Excel coefficient numbers come out as something in the range of (-1,1). Their goal is to take the 2 or 3 largest of the coefficients and look at the corresponding xn value (that is, x1, x2, or x3) to decide which of the x values is most influential in the resulting y. So first: linalg.lstsq gives me back four variables, none of which, but `rank`, do I completely understand. The first returned value, `x`, claims to be a vector, but I'm a little lost on that. If it's a vector, I assume it's projecting from the origin point? But that seems too easy, and likely incorrect, but I've got no `intercept` value to observe. Is this capable of giving me the result I'm trying to chase? Second: stats.linregress gives me much more familiar data, but seems to only compute one (x,y) plot at a time, leading me to conclude that I should do 3 separate computations: linregress with all (x1,y) data points, then again with (x2,y) points, and then again with (x3,y) points. Using this method, I get slopes, intercepts, r-values, etc. Is it then up to me to minimize the r^2 value? Simply my lack of exposure to stats leaves me unsure of this step. Third: Is the procedure my company uses totally flawed? That is, if I want to determine which of x1, x2, and x3 is most influential on y, across all samples, is there a more direct calculation that yields the correct conclusion? Thanks in advance-- I'm eager to wrap my head around this! I just need some direction as I continue to play with scipy. Tim

3 2

need assistance
by AHILAN KANAGASUNDARAM Sept. 25, 2010

Sept. 25, 2010

Hi, I'm working on python and scipy. I need to create block diagonal matrix, When I tried to use below function, it says that there is not block module >>> a=numpy.ones([2,2]) >>> b=numpy.random.rand(3,3) >>> import block >>> c=block.block_diag(a,b) >>> print c How can I fix this problem. Is there any alternative way to create block diagonal matrix. I look forward to hearing from you. Thanks K.Ahilan

2 1

Installation of SciPy 0.8.0 for Python 2.7 on MacOS 10.6
by Hans-Martin v. Gaudecker Sept. 24, 2010

Sept. 24, 2010

Hi, I am trying to compile SciPy for use with python.org Python 2.7 on Snow Leopard, but I can't seem to get the architectures to work together correctly (examples all use numscons, the install didn't work with the standard procedure, either). Same behaviour on Core2 Duo MacBook Pro and a Mac Pro with Quad-Core Xeon. Setting the flags to 64-bit as described on the SciPy website leads to the build to break (F77 BLAS not found). The build runs through if I leave them out, but then many tests fail with [...] no suitable image found. Did find [...] mach-o, but wrong architecture. I am lost -- any help would be greatly appreciated. More details below. Best wishes, Hans-Martin gfortran is 4.2.3 for SL from r.resarch.att.com: $ gfortran -v Using built-in specs. Target: i686-apple-darwin8 Configured with: /Builds/unix/gcc/gcc-4.2/configure --prefix=/usr/local --mandir=/share/man --program-transform-name=/^[cg][^.-]*$/s/$/-4.2/ --build=i686-apple-darwin8 --host=i686-apple-darwin8 --target=i686-apple-darwin8 --enable-languages=fortran Thread model: posix gcc version 4.2.3 Error for: sudo LDFLAGS="-arch x86_64" FFLAGS="-arch x86_64" python setupscons.py scons --jobs=4 install Mkdir("build/scons/scipy/integrate") Checking if gfortran needs dummy main - Failed ! Exception: Could not find F77 BLAS, needed for integrate package: File "~/Downloads/scipy-0.8.0/scipy/integrate/SConstruct", line 2: GetInitEnvironment(ARGUMENTS).DistutilsSConscript('SConscript') File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numscons/core/numpyenv.py", line 135: build_dir = '$build_dir', src_dir = '$src_dir') File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numscons/scons-local/scons-local-1.2.0/SCons/Script/SConscript.py", line 553: return apply(_SConscript, [self.fs,] + files, subst_kw) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numscons/scons-local/scons-local-1.2.0/SCons/Script/SConscript.py", line 262: exec _file_ in call_stack[-1].globals File "~/Downloads/scipy-0.8.0/build/scons/scipy/integrate/SConscript", line 15: raise Exception("Could not find F77 BLAS, needed for integrate package") error: Error while executing scons command. See above for more information. If you think it is a problem in numscons, you can also try executing the scons command with --log-level option for more detailed output of what numscons is doing, for example --log-level=0; the lowest the level is, the more detailed the output it. Typical test output after "sudo python setupscons.py scons --jobs=4 install" seemed to work fine: ====================================================================== ERROR: Failure: ImportError (dlopen(/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so, 2): no suitable image found. Did find: /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so: mach-o, but wrong architecture) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose-0.11.4-py2.7.egg/nose/loader.py", line 382, in loadTestsFromName addr.filename, addr.module) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose-0.11.4-py2.7.egg/nose/importer.py", line 39, in importFromPath return self.importFromDir(dir_path, fqname) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose-0.11.4-py2.7.egg/nose/importer.py", line 86, in importFromDir mod = load_module(part_fqname, fh, filename, desc) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/__init__.py", line 9, in <module> import vq, hierarchy File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/hierarchy.py", line 199, in <module> import _hierarchy_wrap ImportError: dlopen(/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so, 2): no suitable image found. Did find: /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so: mach-o, but wrong architecture

2 1