Mailman 3 March 2009 - SciPy-Dev

SLSQP Constrained Optimizer Status
by Rob Falck 25 Nov '13

25 Nov '13

I'm currently implementing the Sequential Least Squares Quadratic Programming (SLSQP) optimizer, by Dieter Kraft, for use in Scipy. The Fortran code being wrapped with F2PY is here: http://www.netlib.org/toms/733 (its use within Scipy has been cleared) SLSQP provides for bounds on the independent variables, as well as equality and inequality constraint functions, which is a capability that doesn't exist in scipy.optimize. Currently the code works, although the constraint normals are being generated by approximation. I'm working on a way to pass these in. I think the most elegant way will be a single function that returns the matrix of constraint normals. For a demonstration of what the code can do, here is an optimization of f(x,y) = 2xy + 2x - x**2 - 2y**2 Example 14.2 in Chapra & Canale gives the maximum as x=2.0, y=1.0. The unbounded optimization tests find this solution. As expected, its faster when derivatives are provided rather than approximated. Unbounded optimization. Derivatives approximated. Elapsed time: 1.45792961121 ms Results [[1.9999999515712266, 0.99999996181577444], -1.9999999999999984, 4, 0, 'Optimization terminated successfully.'] Unbounded optimization. Derivatives provided. Elapsed time: 1.03211402893 ms Results [[1.9999999515712266, 0.99999996181577444], -1.9999999999999984, 4, 0, 'Optimization terminated successfully.'] The following example uses an equality constraint to find the optimal when x=y. Bound optimization. Derivatives approximated. Elapsed time: 1.384973526 ms Results [[0.99999996845920858, 0.99999996845920858], -0.99999999999999889, 4, 0, 'Optimization terminated successfully.'] I've tried to conform to the syntax used by the other optimizers in scipy.optimize. The function definition and doc string are below. If anyone is interested in testing it out, let me know. def fmin_slsqp( func, x0 , eqcons=[], f_eqcons=None, ieqcons=[], f_ieqcons=None, bounds = [], fprime = None, fprime_cons=None,args = (), iter = 100, acc = 1.0E-6, iprint = 1, full_output = 0, epsilon = _epsilon ): """ Minimize a function using Sequential Least SQuares Programming Description: Python interface function for the SLSQP Optimization subroutine originally implemented by Dieter Kraft. Inputs: func - Objective function (in the form func(x, *args)) x0 - Initial guess for the independent variable(s). eqcons - A list of functions of length n such that eqcons[j](x0,*args) == 0.0 in a successfully optimized problem f_eqcons - A function of the form f_eqcons(x, *args) that returns an array in which each element must equal 0.0 in a successfully optimized problem. If f_eqcons is specified, eqcons is ignored. ieqcons - A list of functions of length n such that ieqcons[j](x0,*args) >= 0.0 in a successfully optimized problem f_ieqcons - A function of the form f_ieqcons(x0, *args) that returns an array in which each element must be greater or equal to 0.0 in a successfully optimized problem. If f_ieqcons is specified, ieqcons is ignored. bounds - A list of tuples specifying the lower and upper bound for each independent variable [ (xl0, xu0), (xl1, xu1), ...] fprime - A function that evaluates the partial derivatives of func fprime_cons - A function of the form f(x, *args) that returns the m by n array of constraint normals. If not provided, the normals will be approximated. Equality constraint normals precede inequality constraint normals. args - A sequence of additional arguments passed to func and fprime iter - The maximum number of iterations (int) acc - Requested accuracy (float) iprint - The verbosity of fmin_slsqp. iprint <= 0 : Silent operation iprint == 1 : Print summary upon completion (default) iprint >= 2 : Print status of each iterate and summary full_output - If 0, return only the minimizer of func (default). Otherwise, output final objective function and summary information. epsilon - The step size for finite-difference derivative estimates. Outputs: ( x, { fx, gnorm, its, imode, smode }) x - The final minimizer of func. fx - The final value of the objective function. its - The number of iterations. imode - The exit mode from the optimizer, as an integer. smode - A string describing the exit mode from the optimizer. Exit modes are defined as follows: -1 : Gradient evaluation required (g & a) 0 : Optimization terminated successfully. 1 : Function evaluation required (f & c) 2 : Number of equality constraints larger than number of independent variables 3 : More than 3*n iterations in LSQ subproblem 4 : Inequality constraints incompatible 5 : Singular matrix E in LSQ subproblem 6 : Singular matrix C in LSQ subproblem 7 : Rank-deficient equality constraint subproblem HFTI 8 : Positive directional derivative for linesearch 9 : Iteration limit exceeded -- - Rob Falck

4 16

Continuous Wavelet Transform for SciPy
by Sean Arms 13 Nov '09

13 Nov '09

Greetings! My name is Sean Arms and I'm a graduate student at the University of Oklahoma in the School of Meteorology. As part of my PhD research, I'm studying coherent structures in atmospheric boundary layer turbulence, primarily using in-situ observations and, secondarily, Direct Numerical Simulation (DNS) output. One common approach for detecting coherent structures in observational datasets relies on the use of the global wavelet power spectrum as estimated from a continuous wavelet transform (CWT). I know SciPy has a DWT impementation, and I've already been in contact with Filip. He recommeded that I post my code in hopes that it would add some momentum to the python-cwt development and create some feedback (I'm currently looking for a good place to post my code). I've implemented the CWT using pure python (that is, I did not write any C extension code myself - nothing to build), along with one mother wavelet (second derivitive of a Gaussian, or the Mexican Hat) - I'll be adding more Mother wavelets as I go along. I've made it a point to (try to) design my MotherWavelet class to be easily extendable. I'm working on documentation and a few tests at the moment, but so far my code compares well with other wavelet routines. The point of this email is to introduce myself and let the SciPy dev community know that I am willing to help develop CWT support for SciPy - I'll already be doing the work for my research, so I might as well put in the extra effort to make is usable by the larger community! Cheers! Sean Arms Graduate Research Assistant School of Meteorology University of Oklahoma

4 7

Porting SciPy to Py3k GSOC project
by ross smith 20 Jul '09

20 Jul '09

Hello everyone, I am interested in porting SciPy/NumPy to Py3k. I've been working this past school year to port an existing code base to py3k for a research group on campus. A lot of the code relies on SciPy and NumPy but the scope of my project didn't let me work on porting either project, to my dismay. I'd love the opportunity to port a project I use heavily in my own code and gain a better understanding of how it works. We are supposed to contact the group we would be working with, to flesh out the details of our application. I've looked at the application and the only thing I know I'll need significant help with is the Milestones portion. Of course, Any and all suggestions are welcome! thank you, Ross Smith (Gaurdro on Freenode)

2 3

rgb_to_hsv in scipy.misc ? (was: [Numpy-discussion] Optimizing speed for large-array inter-element algorithms (specifically, color space conversion))
by Nicolas Pinto 08 Jul '09

08 Jul '09

Hello, Would it be possible to include the following rgb to hsv conversion code in scipy (probably in misc along with misc.imread, etc.) ? What do you think? Thanks in advance. Best regards, -- Nicolas Pinto Ph.D. Candidate, Brain & Computer Sciences Massachusetts Institute of Technology, USA http://web.mit.edu/pinto # ------------------------------------------------------------------------------ import numpy as np def rgb_to_hsv_arr(arr): """ fast rgb_to_hsv using numpy array """ # adapted from Arnar Flatberg # http://www.mail-archive.com/numpy-discussion@scipy.org/msg06147.html # it now handles NaN properly and mimics colorsys.rgb_to_hsv output arr = arr/255. out = np.empty_like(arr) arr_max = arr.max(-1) delta = arr.ptp(-1) s = delta / arr_max s[delta==0] = 0 # red is max idx = (arr[:,:,0] == arr_max) out[idx, 0] = (arr[idx, 1] - arr[idx, 2]) / delta[idx] # green is max idx = (arr[:,:,1] == arr_max) out[idx, 0] = 2. + (arr[idx, 2] - arr[idx, 0] ) / delta[idx] # blue is max idx = (arr[:,:,2] == arr_max) out[idx, 0] = 4. + (arr[idx, 0] - arr[idx, 1] ) / delta[idx] out[:,:,0] = (out[:,:,0]/6.0) % 1.0 out[:,:,1] = s out[:,:,2] = arr_max # rescale back to [0, 255] out *= 255. # remove NaN out[np.isnan(out)] = 0 return out

4 5

Patching fmin_l_bfgs_b in scipy.optimize
by Gilles Rochefort 16 Apr '09

16 Apr '09

Hi, I had some troubles getting a stable version of fmin_l_bfgs_b. I got a segmentation fault as soon as I turned a different value than -1 to iprint argument. As far as I understood, this iprint parameter is directly related to some fortran printing/writing logs. It prints iteration, norm of gradient on screen and also in a file called iterate.dat .. Even recompiling the whole stuff from lapack, atlas etc.. (using gfortran only) up to scipy doesn't solve the problem. Because I definitely gave up the idea to fix the bug by recompiling, I decided to patch the way python interacts with lbfgsb fortran procedure. So, I disabled the fortran iprint and add some iprint functionnality in pure python code directly in lbfgsb.py (scipy/optimize). Doing so, I get rid of the useless iterate.dat file too. While patching, I followed by adding some few things : - an optional stopping rule based on maximum number of iterations. I think this is more usefull than the number of function evaluations because the fortran code performs a linesearch procedure inside the lbfgsb ones. - a callback procedure which allows to inspect solution, criterion, and gradient at current iterate. Maybe my patch should be usefull for someone, so I decided to share it with the scipy community. Gilles. PS: I provide two files : lbfgsb.py -- the full python script to put in scipy/optimize scipy-optimize-lbfgsb-12162008.patch -- a patch file

3 4

JOB: write numpy docs
by Joe Harrington 15 Apr '09

15 Apr '09

Last year's Doc Marathon got us off to a great start on documenting NumPy! But, there's still much work to be done, and SciPy after that. It's time to gear up for doing it again. Critical to last year's success was Stefan van der Walt's committed time, but he will be unable to play that role this year. So, I am looking to hire someone to write NumPy docs and help coordinate the doc project and its volunteers. The job includes working with me, the doc team, doc volunteers, and developers to: write and review a lot of docs, mainly those that others don't want to write help define milestones organize campaigns and volunteer teams to meet them research the NumPy and SciPy source codes to help plan: the eventual SciPy documentation the writing of a good User Manual work with the packaging team to meet their release deadlines perform other duties as assigned I am seeking someone to work full time if possible, and at least half time, from mid-April (or soon thereafter) through at least the (northern) summer. Candidates must be experienced NumPy and SciPy programmers; familiarity under the hood is a strong plus. They must also demonstrate their ability to produce excellent docs on the docs.SciPy.org wiki. Having contributed at a high level to an open-source community, especially to SciPy, is a big plus. Ability to take direction, work with and lead a team, and to work for extended periods without direct supervision on a list of assigned tasks are all critical. The applicant must be able to function well in a Linux environment; familiarity with multiple platforms is a plus. Please reply directly to me by email only. Include the following (PDF or ASCII formats strongly preferred): CV Statement of interest, qualifications per requirements above, availability, and wage expectations. Contact info for at least 3 professional references. Links to doc wiki pages for which you wrote the initial draft Links to doc wiki pages started by others to which you contributed significantly (edited, reviewed, proofed) The position is open until filled; candidates with complete applications by April 15 will receive full consideration. This is an open posting. Candidates who have not written any pages on the doc wiki yet have several weeks in which to do so. Pay will be commensurate with experience (up to a point). Relocation is not necessary. Candidates will need to provide their own computer and internet access. The University of Central Florida is an equal opportunity, equal access, affirmative action employer. --jh-- Prof. Joseph Harrington Department of Physics MAP 414 4000 Central Florida Blvd. University of Central Florida Orlando, FL 32816-2385 (407) 823-3416 voice (407) 823-5112 fax (407) 823-2325 physics office jh(a)physics.ucf.edu

1 1

Re: [SciPy-dev] Numpy/Scipy GSOC project(s)
by ross smith 14 Apr '09

14 Apr '09

Hi David, ross smith wrote: > > I've tried pearu(a)cens.ioc.ee <mailto:pearu@cens.ioc.ee <pearu(a)cens.ioc.ee>> , the issue > has and is that my schools mail server is rejecting my outbound emails. Ok. I have contacted him personally - I will keep the mailing list posted. > > If you are interested in working on scipy-related projects, something > which could help the transition is to replace C code by cython. This > would be particularly helpful in scipy - there are many modules which > are crufty and would benefit from code cleaning, but if you don't know > cython, maybe that's not something that interesting for you. Cython is > not difficult to use, though. scipy.signal comes to mind, to take > something where I could definitely help you - but there are other > modules as well, where other people would be more qualified > (scipy.ndimage, etc...). at least in the > case of scipy.signal, there is definitely some new features which are > needed as well (new feature is generally more fun than code > cleanup/bug > hunting). Of course, you should choose modules which interest you. > > > scipy.signal sounds very interesting to me. I've been toying with the > idea of a soft-synth written in python for a while now. I would be > up for tidying up that module. what other features would be of use to > implement? For scipy.signal: - a lot of C code is 'old', using 'old' idioms. The module works well, but modifying it when there are bugs is challenging. In particular, a lot of array indexing is done manually, but numpy now have array iterator which makes this much easier to handle. - The new numpy generalized ufunc may be useful to improve the functionalities of things like filtering, etc... (I have yet to study generalized ufunc). - some new code to fill the gap compared to matlab signal toolbox (LPC analysis, etc... I already have some code to that purpose in the talkbox scikits, but the code would need to be generalized to arbitrary rank) What's your background in signal processing ? I have a very little from playing with jMax to create beeps and blips. The codebase I worked on porting this school year, also does some signal processing. Would you be more interested in new features or bug cleaning ? At first I'd be more interested in bug cleaning. I think it's the best way to get comfortable with a new codebase. Toward the second half of the summer, I'd be more interested in new features. > > As a start, what would be extremely useful is to be able to build > >Cython ufuncs on the fly. For cython, you should definitely ask the cython ML. cheers, David _______________________________________________ Scipy-dev mailing list Scipy-dev(a)scipy.org http://mail.scipy.org/mailman/listinfo/scipy-dev

2 3

Numpy/Scipy GSOC project(s)
by ross smith 30 Mar '09

30 Mar '09

I apologize for my lack of response on this, I have a lot due in my classes so I was focusing on my schoolwork. On Fri, Mar 27, 2009 at 04:13, David Cournapeau < david(a)ar.media.kyoto-u.ac.jp> wrote: > ross smith wrote: > > I have tried to contact Pearu but my schools mail server has bounced > > the emails I send back for some reason. I'm going to try again tomorrow. > > Which email address did you use ? My impression is that Pearu still > partcipates in the numpy/scipy mailing lists. I've tried pearu(a)cens.ioc.ee , the issue has and is that my schools mail server is rejecting my outbound emails. > > If you are interested in working on scipy-related projects, something > which could help the transition is to replace C code by cython. This > would be particularly helpful in scipy - there are many modules which > are crufty and would benefit from code cleaning, but if you don't know > cython, maybe that's not something that interesting for you. Cython is > not difficult to use, though. scipy.signal comes to mind, to take > something where I could definitely help you - but there are other > modules as well, where other people would be more qualified > (scipy.ndimage, etc...). at least in the > case of scipy.signal, there is definitely some new features which are > needed as well (new feature is generally more fun than code cleanup/bug > hunting). Of course, you should choose modules which interest you. > scipy.signal sounds very interesting to me. I've been toying with the idea of a soft-synth written in python for a while now. I would be up for tidying up that module. what other features would be of use to implement? >One module means that it would be easier for >you to work on, easier for use to help/coordinate After looking at how much code is in a single module, I wholeheartedly agree. I'm planning on hedging my bets a bit on the GSOC applications. I can submit up to 20 applications. I'm going to write up one for a 'preliminary toward Numpy for py3k' and one for 'clean up and add XXXX to scipy.signal'. > As a start, what would be extremely useful is to be able to build >Cython ufuncs on the fly. This also sounds like something very useful that I'd be interested in. right at the moment, I'm a bit unsure on the details of how to implement this or where. as always, feedback and ideas are welcome. thanks for your input thus far! -Ross > > And if you become a regular numpy/scipy contributor, you will be able to > help for the transition later anyway. > > cheers, > > David > _______________________________________________ > Scipy-dev mailing list > Scipy-dev(a)scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev >

2 1

Another GSoC idea
by David Warde-Farley 29 Mar '09

29 Mar '09

I've been fiddling with ideas for GSoC related to SciPy and I wanted to run this by people on the list. David C. and others are often complaining that C and Fortran code is an order of magnitude harder to maintain than Python/Cython code. Thus, would there be interest in a proposal that included rewriting Damian Eads' excellent scipy.spatial.distance and scipy.cluster.vq in Cython? I've already been scoping this out as I had wanted to add output matrix functionality to scipy.spatial.pdist and scipy.spatial.cdist, which would make scenarios where distances are recomputed frequently (as in some sort of tracking application) much less memory-intensive. kmeans Also at the back of my mind have been implementing some of the tricks found in the literature for speeding up k-means (optimized versions that take advantage of the triangle inequality, for instance; "online" k-means, by which I mean updating the means with the contribution of each data point sequentially as opposed to considering them all at once). I'd also like to see the addition of exemplar based methods such as k-centers and the relatively new affinity propagation (there is a reference implementation of the latter which would be unsuitable for direct translation from MATLAB due to licensing, so I'd be proposing a clean-room implementation). Any feedback, additional suggestions would be welcome. Thanks, David

5 5

GSoC Project Proposal: Datasource and Jonathan Taylor's statistical models
by Skipper Seabold 28 Mar '09

28 Mar '09

Hello all, I am a first year PhD student in Economics at American University, and I would very much like to participate in the GSoC with the NumPy/SciPy community. I am looking for some feedback and discussion before I submit a proposal. Judging by the ideas page and the discussion in this thread ( http://mail.scipy.org/pipermail/scipy-dev/2009-February/011373.html ) I think the following project proposal would be useful to the community. My proposal would have two parts, the first would be to improve datasource and integrate it into the numpy/scipy io. I see this as a way to get my feet wet working on a project. I do not imagine that it would take more than 2-3 weeks work on my end. The second part would be to get Jonathan Taylor's statistical models from the NiPy project into scipy.stats. I think that I would be a good candidate for this work, as I am currently studying statistics and learning the ins and outs of NumPy/SciPy, so I don't mind doing some of the less appealing work as this is also a great learning opportunity. Also I see this as a great way to get involved in the SciPy community in an area that currently needs some attention. I am a student, so I would be able to help maintain the code, bug fix, and address other areas of the statistical capabilities that need attention. Below is a general outline of my proposal with some areas that I have identified as needing work. I am eager to discuss some aspects of the projects with those that are interested and to work on the appropriate milestones. 1) Improve datasource and integrate it into all the numpy/scipy io Bug Fixes Catch and handle malformed URLs Refactoring Enhancements Improve findfile method Improve cache method Add zip archive, tar file handling capabilities Improve networking interface to handle timeouts and proxies if there is sufficient interest Documentation Document changes Tests Implement test coverage for new changes Copy/Move to scipy.io 2) Integrate Jonathan Taylor's statistical models into scipy.stats These models are currently in the NiPy project Merge relevant branches (branch trunk-josef models has the most recent changes, I believe) I will focus mostly on bringing over the linear models, which I believe would include at the least: bspline.py, contrast.py, gam.py, glm.py, model.py, regression.py, utils.py Bug Fixes Bug hunting Improve existing test coverage Refactoring Eliminate existing and created duplicate functionality Make sure parameters are consistent, etc. Enhancements Documentation Document changes Make any necessary changes to stats/info.py Testing Make sure test coverage is adequate

5 8