Mailman 3 July 2011 - SciPy-User

scipy.io.netcdf error
by Neil Berg 10 Jul '11

10 Jul '11

Hello scipy community, I recently installed scipy 0.10 through Chris Fonnesbeck's Scipy Superpack script. When trying to read a netCDF file I receive an error message. >>> python >>> from scipy.io import netcdf >>> f = netcdf.netcdf_file('/Users/vert/Downloads/sres.nc','r') Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/Library/Python/2.6/site-packages/scipy-0.10.0.dev_20110629-py2.6-macosx-10.6-universal.egg/scipy/io/netcdf.py", line 205, in __init__ self._read() File "/Library/Python/2.6/site-packages/scipy-0.10.0.dev_20110629-py2.6-macosx-10.6-universal.egg/scipy/io/netcdf.py", line 492, in _read self._read_var_array() File "/Library/Python/2.6/site-packages/scipy-0.10.0.dev_20110629-py2.6-macosx-10.6-universal.egg/scipy/io/netcdf.py", line 576, in _read_var_array buffer=mm, offset=begin_, order=0) TypeError: data type ">d8" not understood Has anyone encountered this error and is there a way to cure it? Many thanks, Neil ----------------------------------------------- Mac OS X 10.6.7 Darwin Kernel Version 10.7.4 i386 -----------------------------------------------

2 1

Bug t-test for identical means with no variance?
by Skipper Seabold 08 Jul '11

08 Jul '11

A ticket was filed [1] for ttest_ind (same issue with ttest_rel and ttest_1samp) in the case of identical means and no variance. Same means, no variance d1 = np.ones(10) d2 = np.array([1,1.]) stats.ttest_ind(d1,d2) (1.0, 0.34089313230206009) Different means, no variance d1 = np.array([ 5., 5., 5., 5., 5., 5., 5., 5., 5., 5.]) d2 = np.array([ 2., 2., 2., 2., 2., 2., 2., 2., 2., 2.]) stats.ttest_ind(d1,d2) (inf, 0.0) The first result doesn't make sense. In the code there are conflicting notes (with each other and what the code does) for catching this https://github.com/scipy/scipy/blob/master/scipy/stats/stats.py#L2873 https://github.com/scipy/scipy/blob/master/scipy/stats/stats.py#L2963 https://github.com/scipy/scipy/blob/master/scipy/stats/stats.py#L3044 I think defining t = 0/0 to be 0 is the least wrong thing to do, but certainly not t = 0/0 as 1, which gives an arbitrary p-value depending on sample sizes. Is there an accepted definition for this case? Does returning (nan, 1.0) make more sense? Skipper [1] http://projects.scipy.org/scipy/ticket/1475

2 3

Today's master branch of Scipy test('10') segfaulting on OSX
by George Nurser 08 Jul '11

08 Jul '11

Hi, Running OSX 10.6.7, python 2.7.1 32/64 bit intel from python.org (compiled with gcc-4.2) , gfortran 4.2, numpy 1.6.1rc2 I just pulled the latest version of scipy master; git log gives top commit as e86854bfab64a16c026106ccdf90761328b83272 scipy.test('10') gives a segfault. First few lines of the Apple DiagnosticReport are: Exception Type: EXC_BAD_ACCESS (SIGSEGV) Exception Codes: KERN_INVALID_ADDRESS at 0x00000002b9e72320 Crashed Thread: 0 Dispatch queue: com.apple.main-thread Thread 0 Crashed: Dispatch queue: com.apple.main-thread 0 libBLAS.dylib 0x00007fff87b9d010 ATL_zdotc_xp0yp0aXbX + 48 1 _iterative.so 0x0000000103b9cf8c zgmresrevcom_ + 1980 2 _iterative.so 0x0000000103b876eb f2py_rout__iterative_zgmresrevcom + 1563 (_iterativemodule.c:5041) 3 org.python.python 0x000000010000c9f2 PyObject_Call + 98 The whole of the segfault trace is attached as scipy.segfault --George Nurser Output of scipy.test('10') below: Running unit tests for scipy NumPy version 1.6.1rc2 NumPy is installed in /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy SciPy version 0.10.0.dev SciPy is installed in /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy Python version 2.7.1 (r271:86882M, Nov 30 2010, 10:35:34) [GCC 4.2.1 (Apple Inc. build 5664)] nose version 0.11.3 ............................................................................................................................................................................................................................K............................................................................................................/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/interpolate/fitpack2.py:678: UserWarning: The coefficients of the spline returned have been computed as the minimal norm least-squares solution of a (numerically) rank deficient system (deficiency=7). If deficiency is large, the results may be inaccurate. Deficiency may strongly depend on the value of eps. warnings.warn(message) ....../Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/interpolate/fitpack2.py:609: UserWarning: The required storage space exceeds the available storage space: nxest or nyest too small, or s too small. The weighted least-squares spline corresponds to the current set of knots. warnings.warn(message) .......................K..K....../Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/core/numeric.py:1920: RuntimeWarning: invalid value encountered in absolute return all(less_equal(absolute(x-y), atol + rtol * absolute(y))) ................................................................................................................................................................................................................................................................................................................................................................................................................./Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/io/wavfile.py:31: WavFileWarning: Unfamiliar format bytes warnings.warn("Unfamiliar format bytes", WavFileWarning) /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/io/wavfile.py:121: WavFileWarning: chunk not understood warnings.warn("chunk not understood", WavFileWarning) ................................................................................./Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/lib/utils.py:139: DeprecationWarning: `get_blas_funcs` is deprecated! warnings.warn(depdoc, DeprecationWarning) ..............................................................................................................................................SSSSSS......SSSSSS......SSSS............................................................................S................................................................................................................................................................................................................................................K.............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................../Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/sparse/linalg/dsolve/linsolve.py:259: DeprecationWarning: scipy.sparse.linalg.dsolve.umfpack will be removed, install scikits.umfpack instead ' install scikits.umfpack instead', DeprecationWarning ) ../Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/sparse/linalg/dsolve/linsolve.py:75: DeprecationWarning: scipy.sparse.linalg.dsolve.umfpack will be removed, install scikits.umfpack instead ' install scikits.umfpack instead', DeprecationWarning ) ................F...F..F.....FF.EF..FEF.E.E...E........E...EF.F.F.FEE..E..E...................................................................F.......E.EFFFE.F..EF.E...EE...FEE...F.F..FEEF...EE..E............................................................EEEFFEEESegmentation fault

2 3

linalg.decomp code contribution
by Collin Stocks 07 Jul '11

07 Jul '11

Over the past several years, there have been various requests for the linalg.decomp.qr() function to (optionally) implement qr with pivots. I have written a solution which, unfortunately, requires cvxopt, since there is no wrapper within scipy for lapack.geqp3(). However, this should be an effective starting place for anyone who wants to complete the job. If this does eventually end up in scipy, I would appreciate a little mention, though! :) The code can also be found at: http://pastebin.com/HFJdQ67H I have also extended the "mode" keyword slightly, and have removed the "econ" keyword, as it is deprecated anyway (assuming that I have the very latest release on my own machine, which is not all that likely...). ######################################################## import cvxopt from cvxopt import lapack def qr(a, overwrite_a=0, lwork=None, mode='qr'): """Compute QR decomposition of a matrix. Calculate the decomposition :lm:`A = Q R` where Q is unitary/ orthogonal and R upper triangular. Parameters ---------- a : array, shape (M, N) Matrix to be decomposed overwrite_a : boolean Whether data in a is overwritten (may improve performance) lwork : integer Work array size, lwork >= a.shape[1]. If None or -1, an optimal size is computed. mode : {'qr', 'r', 'qrp'} Determines what information is to be returned: either both Q and R or only R, or Q and R and P, a permutation matrix. Any of these can be combined with 'economic' using the '+' sign as a separator. Economic mode means the following: Compute the economy-size QR decomposition, making shapes of Q and R (M, K) and (K, N) instead of (M,M) and (M,N). K=min(M,N). Returns ------- (if mode == 'qr') Q : double or complex array, shape (M, M) or (M, K) for econ==True (for any mode) R : double or complex array, shape (M, N) or (K, N) for econ==True Size K = min(M, N) Raises LinAlgError if decomposition fails Notes ----- This is an interface to the LAPACK routines dgeqrf, zgeqrf, dorgqr, and zungqr. Examples -------- >>> from scipy import random, linalg, dot >>> a = random.randn(9, 6) >>> q, r = linalg.qr(a) >>> allclose(a, dot(q, r)) True >>> q.shape, r.shape ((9, 9), (9, 6)) >>> r2 = linalg.qr(a, mode='r') >>> allclose(r, r2) >>> q3, r3 = linalg.qr(a, econ=True) >>> q3.shape, r3.shape ((9, 6), (6, 6)) """ mode = mode.split("+") if "economic" in mode: econ = True else: econ = False a1 = asarray_chkfinite(a) if len(a1.shape) != 2: raise ValueError("expected 2D array") M, N = a1.shape overwrite_a = overwrite_a or (_datanotshared(a1,a)) if 'qrp' in mode: qr = cvxopt.matrix(np.asarray(a1, dtype = float)) tau = cvxopt.matrix(np.zeros(min(M, N), dtype = float)) jpvt = cvxopt.matrix(np.zeros(N, dtype = int)) lapack.geqp3(qr, jpvt, tau) qr = np.asarray(qr) tau = np.asarray(tau) jpvt = (np.asarray(jpvt) - 1).flatten() else: geqrf, = get_lapack_funcs(('geqrf',),(a1,)) if lwork is None or lwork == -1: # get optimal work array qr,tau,work,info = geqrf(a1,lwork=-1,overwrite_a=1) lwork = work[0] qr,tau,work,info = geqrf(a1,lwork=lwork,overwrite_a=overwrite_a) if info<0: raise ValueError("illegal value in %-th argument of internal geqrf" % -info) if not econ or M<N: R = basic.triu(qr) else: R = basic.triu(qr[0:N,0:N]) if 'r' in mode: return R if find_best_lapack_type((a1,))[0]=='s' or find_best_lapack_type((a1,))[0]=='d': gor_un_gqr, = get_lapack_funcs(('orgqr',),(qr,)) else: gor_un_gqr, = get_lapack_funcs(('ungqr',),(qr,)) if M<N: # get optimal work array Q,work,info = gor_un_gqr(qr[:,0:M],tau,lwork=-1,overwrite_a=1) lwork = work[0] Q,work,info = gor_un_gqr(qr[:, 0:M],tau,lwork=lwork,overwrite_a=1) elif econ: # get optimal work array Q,work,info = gor_un_gqr(qr,tau,lwork=-1,overwrite_a=1) lwork = work[0] Q,work,info = gor_un_gqr(qr,tau,lwork=lwork,overwrite_a=1) else: t = qr.dtype.char qqr = numpy.empty((M,M),dtype=t) qqr[:,0:N]=qr # get optimal work array Q,work,info = gor_un_gqr(qqr,tau,lwork=-1,overwrite_a=1) lwork = work[0] Q,work,info = gor_un_gqr(qqr,tau,lwork=lwork,overwrite_a=1) if info < 0: raise ValueError("illegal value in %-th argument of internal gorgqr" % -info) if 'qrp' in mode: return Q, R, jpvt return Q, R ########################################################

3 3

reading in files with fixed with format
by Wolfgang Kerzendorf 07 Jul '11

07 Jul '11

Dear all, I have a couple of data files that were written with fortran at a fixed with. That means its tabular data which might not have spaces (it is just specified how many characters each field has and what type it is). Is there anything to read that with scipy and or numpy? Cheers Wolfgang

3 2

Call for papers: AMS Jan 22-26, 2012
by David Brown 06 Jul '11

06 Jul '11

I would like to call to the attention of the SciPy community the following call for papers: Second Symposium on Advances in Modeling and Analysis Using Python, 22–26 January 2012, New Orleans, Louisiana The Second Symposium on Advances in Modeling and Analysis Using Python, sponsored by the American Meteorological Society, will be held 22–26 January 2012, as part of the 92nd AMS Annual Meeting in New Orleans, Louisiana. Preliminary programs, registration, hotel, and general information will be posted on the AMS Web site (http://www.ametsoc.org/meet/annual/) in late-September 2011. The application of object-oriented programming and other advances in computer science to the atmospheric and oceanic sciences has in turn led to advances in modeling and analysis tools and methods. This symposium focuses on applications of the open-source language Python and seeks to disseminate advances using Python in the atmospheric and oceanic sciences, as well as grow the earth sciences Python community. Papers describing Python work in applications, methodologies, and package development in all areas of meteorology, climatology, oceanography, and space sciences are welcome, including (but not limited to): modeling, time series analysis, air quality, satellite data processing, in-situ data analysis, GIS, Python as a software integration platform, visualization, gridding, model intercomparison, and very large (petabyte) dataset manipulation and access. The $95 abstract fee includes the submission of your abstract, the posting of your extended abstract, and the uploading and recording of your presentation which will be archived on the AMS Web site. Please submit your abstract electronically via the Web by 1 August 2011 (refer to the AMS Web page athttp://www.ametsoc.org/meet/online_submit.html.) An abstract fee of $95 (payable by credit card or purchase order) is charged at the time of submission (refundable only if abstract is not accepted). Authors of accepted presentations will be notified via e-mail by late-September 2011. All extended abstracts are to be submitted electronically and will be available on-line via the Web, Instructions for formatting extended abstracts will be posted on the AMS Web site. Manuscripts (up to 3MB) must be submitted electronically by 22 February 2012. All abstracts, extended abstracts and presentations will be available on the AMS Web site at no cost. For additional information, please contact the program chairperson, Johnny Lin, Physics Department, North Park University (jlin(a)northpark.edu). (5/11)

1 0

Re: [SciPy-User] [Numpy-discussion] ANN: NumPy 1.6.1 release candidate 2
by Ralf Gommers 06 Jul '11

06 Jul '11

On Tue, Jul 5, 2011 at 11:41 PM, Russell E. Owen <rowen(a)uw.edu> wrote: > In article <BANLkTi=LXiTcrv1LgMtP=p9nF8eMr8=+hw(a)mail.gmail.com>, > Ralf Gommers <ralf.gommers(a)googlemail.com> wrote: > > > https://sourceforge.net/projects/numpy/files/NumPy/1.6.1rc2/ > > Will there be a Mac binary for 32-bit pythons (one that is compatible > with older versions of MacOS X)? At present I only see a 64-bit > 10.6-only version. > > > Yes there will be for the final release (10.4-10.6 compatible). I can't create those on my own computer, so sometimes I don't make them for RCs. Cheers, Ralf

1 0

random variable for truncated multivariate normal and t distributions
by josef.pktd＠gmail.com 05 Jul '11

05 Jul '11

(another random question) I would like to generate random variables for multivariate normal and t distributions that are truncated on a rectangular area (element wise upper and lower bounds). Is there anything better available than sampling from the un-truncated distributions and throwing away the out of bounds samples ? Josef

3 5

[ANN] guiqwt v2.1.4
by Pierre.RAYBAUT＠CEA.FR 05 Jul '11

05 Jul '11

Hi all, I am pleased to announce that `guiqwt` v2.1.4 has been released. Based on PyQwt (plotting widgets for PyQt4 graphical user interfaces) and on the scientific modules NumPy and SciPy, guiqwt is a Python library providing efficient 2D data-plotting features (curve/image visualization and related tools) for interactive computing and signal/image processing application development. Complete change log is now available here: http://code.google.com/p/guiqwt/wiki/ChangeLog Documentation with examples, API reference, etc. is available here: http://packages.python.org/guiqwt/ This version of `guiqwt` includes a demo software, Sift (for Signal and Image Filtering Tool), based on `guidata` and `guiqwt`: http://packages.python.org/guiqwt/sift.html Windows users may even download the portable version of Sift 0.2.3 to test it without having to install anything: http://code.google.com/p/guiqwt/downloads/detail?name=sift023_portable.zip When compared to the excellent module `matplotlib`, the main advantages of `guiqwt` are: * Performance: see http://packages.python.org/guiqwt/overview.html#performances * Interactivity: see for example http://packages.python.org/guiqwt/_images/plot.png * Powerful signal processing tools: see for example http://packages.python.org/guiqwt/_images/fit.png * Powerful image processing tools: * Real-time contrast adjustment: http://packages.python.org/guiqwt/_images/contrast.png * Cross sections (line/column, averaged and oblique cross sections!): http://packages.python.org/guiqwt/_images/cross_section.png * Arbitrary affine transforms on images: http://packages.python.org/guiqwt/_images/transform.png * Interactive filters: http://packages.python.org/guiqwt/_images/imagefilter.png * Geometrical shapes/Measurement tools: http://packages.python.org/guiqwt/_images/image_plot_tools.png * Perfect integration of `guidata` features for image data editing: http://packages.python.org/guiqwt/_images/simple_window.png But `guiqwt` is more than a plotting library; it also provides: * Framework for signal/image processing application development: see http://packages.python.org/guiqwt/examples.html * And many other features like making executable Windows programs easily (py2exe helpers): see http://packages.python.org/guiqwt/disthelpers.html guiqwt has been successfully tested on GNU/Linux and Windows platforms. Python package index page: http://pypi.python.org/pypi/guiqwt/ Documentation, screenshots: http://packages.python.org/guiqwt/ Downloads (source + Python(x,y) plugin): http://guiqwt.googlecode.com Cheers, Pierre --- Dr. Pierre Raybaut CEA - Commissariat à l'Energie Atomique et aux Energies Alternatives

1 0

[ANN] guidata v1.3.2
by Pierre.RAYBAUT＠CEA.FR 05 Jul '11

05 Jul '11

Hi all, I am pleased to announce that `guidata` v1.3.2 has been released. Note that the project has recently been moved to GoogleCode: http://guidata.googlecode.com Main change since `guidata` v1.3.1: Since this version, guidata is compatible with PyQt4 API #1 and API #2. Please read carefully the coding guidelines which have been recently added to the documentation. Complete changelog is available here: http://code.google.com/p/guidata/wiki/ChangeLog The `guidata` documentation with examples, API reference, etc. is available here: http://packages.python.org/guidata/ Based on the Qt Python binding module PyQt4, guidata is a Python library generating graphical user interfaces for easy dataset editing and display. It also provides helpers and application development tools for PyQt4. guidata also provides the following features: * guidata.qthelpers: PyQt4 helpers * guidata.disthelpers: py2exe helpers * guidata.userconfig: .ini configuration management helpers (based on Python standard module ConfigParser) * guidata.configtools: library/application data management * guidata.gettext_helpers: translation helpers (based on the GNU tool gettext) * guidata.guitest: automatic GUI-based test launcher * guidata.utils: miscelleneous utilities guidata has been successfully tested on GNU/Linux and Windows platforms. Python package index page: http://pypi.python.org/pypi/guidata/ Documentation, screenshots: http://packages.python.org/guidata/ Downloads (source + Python(x,y) plugin): http://guidata.googlecode.com Cheers, Pierre --- Dr. Pierre Raybaut CEA - Commissariat à l'Energie Atomique et aux Energies Alternatives

1 0