Mailman 3 July 2011 - SciPy-User

estimation problem
by Neal Becker 15 Jul '11

15 Jul '11

I have a known signal (vector) 'x'. I recieve a vector 'y' y = F(k x) + n where n is Gaussian noise, and k is an unknown gain parameter. I want to estimate k. F is a known function (nonlinear, memoryless). What might be a good approach to try? I'd like this to be an 'online' approach - that is, I provide batches of training vectors (x, n), and the estimator will improve the estimate (hopefully) as more data is supplied.

2 6

[ANN] Bento 0.0.6, a packaging solution for python software
by David Cournapeau 15 Jul '11

15 Jul '11

Hi, I am pleased to announce a new release of bento, a packaging solution for python which aims at reproducibility, extensibility and simplicity. It supports every python version from 2.4 to 3.2. You can take a look at its main features on Bento's main page (http://cournape.github.com/Bento). The main features of this 0.0.6 release are: - Completely revamped distutils compatibility layer: it is now a thin layer around bento infrastructure, so that most bento packages should be pip-installable, while still keeping bento customization capabilities. - Build directory is now customizable through bentomaker with --build-directory option - Out of tree builds support (i.e. running bento in a directory which does not contain bento.info), with global --bento-info option - Hook File can now be specified in recursed bento.info - Preliminary support for .mpkg (Mac OS X native packaging) - More consistent API for extension/compiled library build registration - Both numpy and scipy can now be built with bento + waf as a build backend Bento is discussed on the bento mailing list (http://librelist.com/browser/bento). cheers, David

1 0

Deciphering Matlab Structures in Python
by Paul Blelloch 14 Jul '11

14 Jul '11

I used the scipy.io.loadmat function to read a Matlab save file that includes some structures. In particular my Matlab code looks something like: a=randn(10,10); e=eig(a); c.mat=a; c.eig=e; save When I read this in I have not trouble getting the a and e matrices, but when I look at the structure c I get something that looks like: array([[ ([[-1.918381876114684, 0.35909312220124379, 0.071481108399536447, 0.19413316926540342, -0.16044557980861471, -1.2431101120071242, 1.1728553895658809, 0.48682974787947486, -2.6862862203427951, 1.6479268546008081], [-0.13106782428104052, -0.79434474702678692, 0.15562063581539223, 0.29769161930929805, 0.37883517610546497, 0.13185918869913515, -2.1155833182251094, -0.4459044413697299, 1.5483561910050594, -0.30667757004699159], [-0.76863166681707873, -0.22731164066134216, -0.18219139268446782, -0.71012508212938263, -0.10716072249907738, -0.99947508351036363, 0.46216113959481137, 0.32567307426214315, -0.92437868406001578, -0.77404140889205997], [2.3899518363067971, 1.5938213548880649, 0.73101584111958162, -0.68767410135748297, 0.21283509315953025, -0.35421345818508981, -2.4599031616561144, -0.02009780648140742, -0.72040490902555765, -1.3542395842984021], [0.077245679591339572, 0.15520563382674926, -0.34756806580692184, 0.57283812483950614, 0.46806391670861758, 0.10558871537972472, -0.30485718475158774, -1.8884755753587876, 0.83350531035866249, -0.32592678918795287], [0.37560745023340852, 0.17859562900297904, 0.23442400798144603, 0.44521760367790375, -1.6160743410467069, 0.95542626643090245, 0.39823325391367814, 0.33836146832009201, -0.56649185628868159, 1.5532163153526557], [0.39556416936787731, -0.33773084599208325, -0.86772936409388435, -1.3259592591460707, 0.014738006197359526, 0.46479897799135333, -1.7057865505375613, 0.33780540287940392, -0.99915099624077919, 0.62813120976932557], [-0.11249908896048552, -1.5249918897593304, -0.98253858503060265, -1.4998754094582201, -0.037764347672058281, 0.26670638822706283, 0.75940981180842837, 0.42433121858231648, 0.64126705212804702, -1.0275461814124498], [-1.8290367022161331, -0.70940664667282161, -0.29357242375327913, 1.0159682271872925, 0.40380900009311393, -1.4232233423222125, 0.20323761868334428, 0.11287078734861246, -0.25395009085808468, -0.20194229402517888], [2.0914521934834784, -0.86661663365224473, 0.82446963631241454, -0.027515788346229055, 0.30525439285092776, 0.11105730902622701, 0.95868278992227296, -0.11827036575813449, -1.1216999904267995, 0.86106602555594924]], [[(-4.5696334670036887+0j)], [(3.7785600753832629+0j)], [(-2.5789412296806438+0j)], [(0.30103481909678725+1.8315300529614498j)], [(0.30103481909678725-1.8315300529614498j)], [(1.4965987808471322+0j)], [(-0.6068802651298042+1.19212639770675j)], [(-0.6068802651298042-1.19212639770675j)], [(-0.54808825947184969+0j)], [(0.19975366069054107+0j)]])]], dtype=[('mat', '|O4'), ('eig', '|O4')]) Looking through the numpy documentation I can't figure out how to interpret that. How do I pull the fields out of the structure? THANKS, Paul

2 2

statsmodels - rlm
by Jeff Reback 14 Jul '11

14 Jul '11

Hi, Is their a way to recover the final Results class in a robust estimation? see below - I can clearly replicate the WLS regression and get a Results class with the correct results; is their a reference in the RLMResults class (that points to the wls_results) that I am missing? thanks, Jeff # using v2 of statsmodels import numpy as np import scikits.statsmodels as sm #delivery time(minutes) endog = np.array([16.68, 11.50, 12.03, 14.88, 13.75, 18.11, 8.00, 17.83, 79.24, 21.50, 40.33, 21.00, 13.50, 19.75, 24.00, 29.00, 15.35, 19.00, 9.50, 35.10, 17.90, 52.32, 18.75, 19.83, 10.75]) #number of cases, distance (Feet) exog = np.array([[7, 3, 3, 4, 6, 7, 2, 7, 30, 5, 16, 10, 4, 6, 9, 10, 6, 7, 3, 17, 10, 26, 9, 8, 4], [560, 220, 340, 80, 150, 330, 110, 210, 1460, 605, 688, 215, 255, 462, 448, 776, 200, 132, 36, 770, 140, 810, 450, 635,150]]) exog = exog.T exog = sm.add_constant(exog) rlm = sm.RLM(endog, exog).fit() print "RLM params -> %s, r2 -> %s" % (rlm.params, getattr(rlm,'rsquared',None)) wls = sm.WLS(endog, exog, weights = rlm.weights).fit() print "WLS params -> %s, r2 -> %s" % (wls.params, wls.rsquared)

2 1

Scipy 2011 Convore thread now open
by Peter Wang 13 Jul '11

13 Jul '11

Hi folks, I have gone ahead and created a Convore group for the SciPy 2011 conference: https://convore.com/scipy-2011/ I have already created threads for each of the tutorial topics, and once the conference is underway, we'll create threads for each talk, so that audience can interact and post questions. Everyone is welcome to create topics of their own, in addition to the "official" conference topics. For those who are unfamiliar with Convore, it is a cross between a mailing list and a very souped-up IRC. It's usable for aynchronous discussion, but great for realtime, topical chats. Those of you who were at PyCon this year probably saw what a wonderful tool Convore proved to be for a tech conference. People used it for everything from BoF planning to dinner coordination to good-natured heckling of lightning talk speakers. I'm hoping that it will be used to similarly good effect for the SciPy Cheers, Peter

3 2

Sparse matrix question
by Luke Bloy 12 Jul '11

12 Jul '11

Hi, I have what I hope is a fairly simple scipy question about sparse matrices. I have a sparse matrix (A) that i use to build a constraint matrix for an optimisation problem. The constraints, that concern A, are that A_{i,j} d_{j} - A_{j,i} d_{i} == 0. I would then find the optimal d vector. So I'm having problems efficiently building my constraints. The first part is simple as it is just the nonzero elements of A but accessing A transpose in the same order is difficult. The basic logic is this.... rows=numpy.zeros(2*A.nnz, dtype=numpy.int32) cols=numpy.zeros(2*A.nnz, dtype=numpy.int32) data=numpy.zeros(2*A.nnz, dtype=numpy.float64) #A_{i,j} d_{j} numNonZeros = A.nnz rows[:numNonZeros] = numpy.arange(numNonZeros) cols[:numNonZeros] = A.col data[:numNonZeros] = A.data #-A_{j,i} d_{i} rows[numNonZeros:] = numpy.arange(numNonZeros) cols[numNonZeros:] = A.row data[numNonZeros:] = - A[A.col,A.row] ### Atranspose[A.row, A.col] The problem is that this takes too long (> 2minutes for a 3000x3000 matrix with 3 million nonzeros). similary code in matlab is an order of magnitude faster? I've tried both csr and csc for doing the memory access of the transpose, (attached is the csrTest , i can send the matrix file if you are interested) Does anyone have any suggestions on speeding this up? Thanks, Luke

1 0

Install Scipy on Python 3.X
by Peter Norlindh 12 Jul '11

12 Jul '11

Hi, I have just installed Python 3.2 on Ubuntu 11.04 and am now struggling to install Scipy 0.9.0. Earlier, I effortlessly installed Scipy (from the repository) on Python 2.7, but installing it on Python 3.2 seems to be a whole different animal. Are there any easy-to-follow installation instructions available? The INSTALL files that come with the tar files (Numpy and Scipy) are probably very informative, but I find them too extensive and hard to follow. Any help to get it all set up would be greatly appreciated. Best Regards, Peter Norlindh

2 1

Cholesky for semi-definite matrices?
by Nathaniel Smith 12 Jul '11

12 Jul '11

Hi all, I've run into a case where it'd be convenient to be able to compute the Cholesky decomposition of a semi-definite matrix. (It's a covariance matrix computed from not-enough samples, so it's positive semi-definite but rank-deficient.) As any schoolchild knows, the Cholesky is well defined for such cases, but I guess that shows why you shouldn't trust schoolchildren, because I guess the standard implementations blow up if you try: In [155]: np.linalg.cholesky([[1, 0], [0, 0]]) LinAlgError: Matrix is not positive definite - Cholesky decomposition cannot be computed Is there an easy way to do this? -- Nathaniel

4 3

ANN: NumPy 1.6.1 release candidate 3
by Ralf Gommers 11 Jul '11

11 Jul '11

Hi, I am pleased to announce the availability of the third release candidate of NumPy 1.6.1. This is a bugfix release, list of fixed bugs: #1834 einsum fails for specific shapes #1837 einsum throws nan or freezes python for specific array shapes #1838 object <-> structured type arrays regression #1851 regression for SWIG based code in 1.6.0 #1863 Buggy results when operating on array copied with astype() #1870 Fix corner case of object array assignment #1843 Py3k: fix error with recarray #1885 nditer: Error in detecting double reduction loop #1874 f2py: fix --include_paths bug #1749 Fix ctypes.load_library() #1895/1896 iter: writeonly operands weren't always being buffered correctly This third RC has only a single change compared to RC2 (for #1895/1896), which fixes a serious regression in the iterator. If no new problems are reported, the final release will be in one week. Sources and binaries can be found at https://sourceforge.net/projects/numpy/files/NumPy/1.6.1rc2/ Enjoy, Ralf

1 0

SciPy Central: file and link sharing site
by Kevin Dunn 10 Jul '11

10 Jul '11

I'm announcing an alpha release of SciPy Central, a website for sharing code snippets, recipes and links related to scientific computing, specifically using NumPy, SciPy, matplotlib, IPython and similar tools. http://scipy-central.org The idea for the website grew out of previous discussions on this list. The site is currently in alpha mode, which means we'd like you to stress test it by filling in garbage information and trying to break it. We'll keep it in this mode for a couple of days and iron out any bugs. Please report those at https://github.com/kgdunn/SciPyCentral/issues - for those familiar with Django, we've left the site in DEBUG mode, so you can copy/paste the stack trace in your bug reports. Then we will delete all submissions and start in beta mode with DEBUG turned off. If you have any suggestions for improvements, please also post them on the issues list. Thanks, Kevin

2 1