Mailman 3 April 2015 - SciPy-User

linalg question:cholesky for semidefinite, or LDL, QR equivalent for squared matrix
by josef.pktd＠gmail.com April 29, 2015

April 29, 2015

https://github.com/numpy/numpy/pull/4079 I'm looking for the equivalent of Cholesky for possibly singular, symmetric matrices using numpy or scipy linalg. details: I'm writing a function that can either take the data x (nobs, k_vars) or the moment matrix x.T.dot(x) (k_vars, k_vars), and I want to get the same result in both cases. Given data x, all I need is the R of the QR decomposition. If the moment matrix is not singular, then I can get the same with the Cholesky decomposition. However, what's the equivalent of R in QR if x is not of full rank and the moment matrix is singular? QR on the moment matrix gives different numbers and I don't know how to recover the correct R. requirement: I need the same sequential decomposition as qr and cholesky to be used for sequential least squares. Thanks, Josef

1 0

Decimate and filtfilt
by Paul Blelloch April 29, 2015

April 29, 2015

I was using the scipy.signal.decimate function and noticed that it seemed to have some unusual edge effects using the default IIR Chebychev filter. In doing a little research I found a message from 2011 saying that decimate only filtered in one direction, rather than using the filtfilt function to filter both directions in order to get rid of phase shifts. There was also a message that filtfilt had a bug associated with the way that it handled edge effects. I was wondering if anything had changed since 2011. Does decimate still only filter in one direction and are there issues with filtfilt? Using the default FIR filter (ftype='fir') does seem to correct the issue, but that's a more complex and expensive filter so I was wondering if the default IIR filter was working correctly. I'm comparing the Matlab decimate function that does use filtfilt and seems to handle the edge effects correctly. THANKS, Paul Blelloch

2 1

Normalization for optimization in python
by Yuxiang Wang April 27, 2015

April 27, 2015

Dear all, During optimization, it is often helpful to normalize the input parameters to make them on the same order of magnitude, so the convergence can be much better. For example, if we want to minimize f(x), while a reasonable approximation is x0=[1e3, 1e-4], it might be helpful to normalize x0[0] and x0[1] to about the same order of magnitude (often O(1)). My question is, I have been using scipy.optimize and specifically, the L-BFGS-B algorithm. I was wondering that, do I need to normalize that manually by writing a function, or the algorithm already did it for me? Thank you! -Shawn -- Yuxiang "Shawn" Wang Gerling Research Lab University of Virginia yw5aj(a)virginia.edu +1 (434) 284-0836 https://sites.google.com/a/virginia.edu/yw5aj/

4 8

update scipy to v.0.15 from spyder on windows seven 64 bits
by Servant Mathieu April 27, 2015

April 27, 2015

Dear community, I've the latest python (x,y) distribution (2.7.9.0) which incorporates spyder 2.3.2-14 and scipy 0.14.0-7. Unfortunately, I need scipy 0.15 to use the new scipy.optimize.differential_evolution() function. How should I proceed to (easily) update scipy from my spyder? Best, Mathieu

2 1

Creating a Row in SparkSQL from an ArrayList
by Wenlei Xie April 24, 2015

April 24, 2015

Hi, I am wondering if there is any way to create a Row in SparkSQL in Java by using an List? It looks like Row.create(ArrayList) will create a row with single column (and the single column contains the array) Best, Wenlei

1 1

Re: [SciPy-User] Is it good practice to use IPython notebooks as your Python IDE?
by Ghislain Vaillant April 24, 2015

April 24, 2015

On Saturday, March 14, 2015, Brian Merchant <bhmerchant(a)gmail.com> wrote: > I need to write about 2000 lines worth of code (based on my last > implementation of the project I am working on). I anticipate that I'll be > using profilers (including line profilers), numpy, Numba, possibly Cython > and almost definitely matplotlib animations. I will need to write my own > classes, so my program isn't just going to be one long script. > > > Previously, I used Spyder for development, but I am wondering if it makes > sense for one to use an IPython notebook alone. I am considering that > option so that I force myself to program "literately" -- lots of headings, > and nice comments. Graphs embedded to show the effects of optimizations, > and so forth. > > So far, I know that IPython might have some trouble with integrating in > matplotlib animations, but that too is possible with a little research. > Otherwise, I should be able to do everything else quite easily, including > debugging, according to the IPython documentation. In fact, writing > parallel code in IPython might be a very nice idea given how it seems to > offer a lot of functionality in that direction. > > > Do people use IPython (notebooks) as a complete IDE often? If not, why not? > > Kind regards, > Brian > Hi Brian, Do you find anything wrong with Spyder ? I use both IEP [1] and Spyder extensively and find them pretty good for the use case you describe. However if you're looking for more project-style features like refactoring tools, project views, VCS integration, etc..., a full-featured IDE may be more appropriate, like PyCharm as kindly suggested by Alistair. Cheers, Ghis [1] http://www.iep-project.org/

3 2

Crashing
by Andres Gomez-Lievano April 24, 2015

April 24, 2015

Hi, I am reaching out for help. First, the mathematical problem I am trying to solve: Suppose there is a rectangular matrix M = [[m11, m12, m13, m14], [m21, m22, m23, m24], [m31, m32, m33, m34]], from which I only know the totals (i.e., the sum of elements) by row and column. Hence, suppose my data is: tot_of_rows = [80.0, 230.0, 132.0] tot_of_cols = [74.0, 200.0, 91.0, 77.0] Problem: Infer the elements of the matrix using the method of Maximum Entropy. (See Cho, W. and Judge, G. (2008), "Recovering vote choice from incomplete data". Journal of Data Science 6, 155-171) Now, my current code that, in my mind, should solve this problem is (see also http://nbviewer.ipython.org/gist/anonymous/eee0f44d3de1a09570b0): # ----------------- beginning of python code ---------------------- # Importing libraries import numpy as np from scipy.optimize import minimize # Defining functions def Entropy(p_vec): # Function receives a vector of conditional probabilities, that come # from flattening a matrix of conditional probabilities. return -1.0 * np.dot(p_vec, np.log(p_vec)) def vec2mat(vec, nrows, ncols): # Re-generating the matrix from the vector mat = np.array([ [vec[j + i*ncols] for j in range(ncols)] for i in range(nrows) ]) return mat def mat2vec(mat): # Flattening the matrix into a vector nrows = int(np.array(mat).shape[0]) ncols = int(np.array(mat).shape[1]) vec = [mat[i, j] for i in range(nrows) for j in range(ncols)] return np.array(vec) # Functions for the Constraints: # Probabilities sum up to one # Probability: P[p,c] = "conditional probability of country c exporting p" # Xcp[c,p] = P[p,c]*Xc[c] # The problem is to maximize Entropy(P) # Subject to: # 1) sum_p P[p,c] - 1 = 0, for all c # 2) sum_c P[p,c]*Xc[c] - Xp[p] = 0, for all p # 3) 0 <= P[p,c] <= 1, for all c, p # for each c def Constraint_Norm(p_vec, Xc_vec, Xp_vec, c): lenvec_p = len(Xp_vec) lenvec_c = len(Xc_vec) # For clarity, reconstruct the matrix from p_vec p_mat = vec2mat(p_vec, lenvec_p, lenvec_c) return( np.sum([ p_mat[p,c] for p in range(lenvec_p) ]) - 1 ) # About the totals # for each p def Constraint_Mean(p_vec, Xc_vec, Xp_vec, p): lenvec_p = len(Xp_vec) lenvec_c = len(Xc_vec) # For clarity, reconstruct the matrix from p_vec p_mat = vec2mat(p_vec, lenvec_p, lenvec_c) return( np.sum([ p_mat[p,c]*Xc_vec[c] for c in range(lenvec_c) ]) - Xp_vec[p] ) # Initializing the parameters and data of the problem Xc = np.array([80.0, 230.0, 132.0]) Xp = np.array([74.0, 200.0, 91.0, 77.0]) size_c = len(Xc) size_p = len(Xp) # Initializing the tuple of constraints cons = () # Including constraint 1 cons += tuple( ({"type": 'eq', 'fun': lambda x: Constraint_Norm(x, Xc, Xp, c)} for c in range(size_c)) ) # Including constraint 2 cons += tuple( ({"type": 'eq', 'fun': lambda x: Constraint_Mean(x, Xc, Xp, p)} for p in range(size_p)) ) # Generating the bounds for each probability zeroonebounds = tuple((0,1) for c in range(size_c) for p in range(size_p)) # Generating the initial vector to start the 'minimize' algorithm init_vec = np.array([ 1.0/size_p + (np.random.random()-1)/10 for p in range(size_p) for c in range(size_c) ]) ##################################### # Finally, running the algorithm result = minimize(lambda x: -1.0 * Entropy(x), init_vec, bounds=zeroonebounds, constraints=cons, options={'disp': True}) # Display results res_mat = vec2mat(result.x, size_p, size_c) print res_mat # ----------------- end of python code ---------------------- This throws the following error: "Singular matrix C in LSQ subproblem (Exit mode 6)" Why? I don't see anything that would make a matrix in this context non invertible. Any help is highly appreciated.

2 1

wiki.scipy.org down?
by Johann Rohwer April 22, 2015

April 22, 2015

Is the wiki down? I'm getting connection reset errors. Specifically I can't access the Cookbook. Johann The integrity and confidentiality of this email is governed by these terms / Hierdie terme bepaal die integriteit en vertroulikheid van hierdie epos. http://www.sun.ac.za/emaildisclaimer

3 4

lossless 1-D signal compression
by Jason Sachs April 21, 2015

April 21, 2015

Are there any fast lossless 1-D signal compression algorithms out there which do a better job than, say, zlib for dealing with signals that are either a sequence of integers or floating-point numbers with common spectral properties? (usually band-limited) I need to compress a bunch of data, and uncompress it in Python, and it seems like the general-purpose data compression algorithms might be missing opportunities.

2 1

ANN: Veusz 1.23
by Jeremy Sanders April 18, 2015

April 18, 2015

I'm pleased to announce version 1.23 of the Veusz GUI scientific plotting package, which incorporates a object-oriented python plotting interface. Veusz is written in Python, based on PyQt and Numpy. For release notes, please see below. Jeremy

1 0