Mailman 3 April 2011 - SciPy-Dev

SLSQP Constrained Optimizer Status
by Rob Falck Nov. 24, 2013

Nov. 24, 2013

I'm currently implementing the Sequential Least Squares Quadratic Programming (SLSQP) optimizer, by Dieter Kraft, for use in Scipy. The Fortran code being wrapped with F2PY is here: http://www.netlib.org/toms/733 (its use within Scipy has been cleared) SLSQP provides for bounds on the independent variables, as well as equality and inequality constraint functions, which is a capability that doesn't exist in scipy.optimize. Currently the code works, although the constraint normals are being generated by approximation. I'm working on a way to pass these in. I think the most elegant way will be a single function that returns the matrix of constraint normals. For a demonstration of what the code can do, here is an optimization of f(x,y) = 2xy + 2x - x**2 - 2y**2 Example 14.2 in Chapra & Canale gives the maximum as x=2.0, y=1.0. The unbounded optimization tests find this solution. As expected, its faster when derivatives are provided rather than approximated. Unbounded optimization. Derivatives approximated. Elapsed time: 1.45792961121 ms Results [[1.9999999515712266, 0.99999996181577444], -1.9999999999999984, 4, 0, 'Optimization terminated successfully.'] Unbounded optimization. Derivatives provided. Elapsed time: 1.03211402893 ms Results [[1.9999999515712266, 0.99999996181577444], -1.9999999999999984, 4, 0, 'Optimization terminated successfully.'] The following example uses an equality constraint to find the optimal when x=y. Bound optimization. Derivatives approximated. Elapsed time: 1.384973526 ms Results [[0.99999996845920858, 0.99999996845920858], -0.99999999999999889, 4, 0, 'Optimization terminated successfully.'] I've tried to conform to the syntax used by the other optimizers in scipy.optimize. The function definition and doc string are below. If anyone is interested in testing it out, let me know. def fmin_slsqp( func, x0 , eqcons=[], f_eqcons=None, ieqcons=[], f_ieqcons=None, bounds = [], fprime = None, fprime_cons=None,args = (), iter = 100, acc = 1.0E-6, iprint = 1, full_output = 0, epsilon = _epsilon ): """ Minimize a function using Sequential Least SQuares Programming Description: Python interface function for the SLSQP Optimization subroutine originally implemented by Dieter Kraft. Inputs: func - Objective function (in the form func(x, *args)) x0 - Initial guess for the independent variable(s). eqcons - A list of functions of length n such that eqcons[j](x0,*args) == 0.0 in a successfully optimized problem f_eqcons - A function of the form f_eqcons(x, *args) that returns an array in which each element must equal 0.0 in a successfully optimized problem. If f_eqcons is specified, eqcons is ignored. ieqcons - A list of functions of length n such that ieqcons[j](x0,*args) >= 0.0 in a successfully optimized problem f_ieqcons - A function of the form f_ieqcons(x0, *args) that returns an array in which each element must be greater or equal to 0.0 in a successfully optimized problem. If f_ieqcons is specified, ieqcons is ignored. bounds - A list of tuples specifying the lower and upper bound for each independent variable [ (xl0, xu0), (xl1, xu1), ...] fprime - A function that evaluates the partial derivatives of func fprime_cons - A function of the form f(x, *args) that returns the m by n array of constraint normals. If not provided, the normals will be approximated. Equality constraint normals precede inequality constraint normals. args - A sequence of additional arguments passed to func and fprime iter - The maximum number of iterations (int) acc - Requested accuracy (float) iprint - The verbosity of fmin_slsqp. iprint <= 0 : Silent operation iprint == 1 : Print summary upon completion (default) iprint >= 2 : Print status of each iterate and summary full_output - If 0, return only the minimizer of func (default). Otherwise, output final objective function and summary information. epsilon - The step size for finite-difference derivative estimates. Outputs: ( x, { fx, gnorm, its, imode, smode }) x - The final minimizer of func. fx - The final value of the objective function. its - The number of iterations. imode - The exit mode from the optimizer, as an integer. smode - A string describing the exit mode from the optimizer. Exit modes are defined as follows: -1 : Gradient evaluation required (g & a) 0 : Optimization terminated successfully. 1 : Function evaluation required (f & c) 2 : Number of equality constraints larger than number of independent variables 3 : More than 3*n iterations in LSQ subproblem 4 : Inequality constraints incompatible 5 : Singular matrix E in LSQ subproblem 6 : Singular matrix C in LSQ subproblem 7 : Rank-deficient equality constraint subproblem HFTI 8 : Positive directional derivative for linesearch 9 : Iteration limit exceeded -- - Rob Falck

4 16

Question about subpackage/submodule API
by Warren Weckesser May 23, 2011

May 23, 2011

The recent discussion about what to call the module in Pim Schellart's nice Lomb-Scargle contribution brought up a question for me that I have not seen discussed before. In many cases, a subpackage contains many modules, and the assorted functions and classes in these modules are made available in the (sub)package-level namespace by importing them from within __init__.py. For example, scipy/stats contains distributions.py, kde.py, morestats.py, mstats.py, rv.py, stats.py and vonmises.py. Except for mstats.py, the contents of all these are imported into the __init__.py namespace and included in __all__, so users can say, for example, >>> from scipy.stats import bayes_mvs instead of >>> from scipy.stats.more_stats import bayes_mvs Is this an *intentional* definition of an API? For example, is it "wrong" for a user to refer to the submodule 'more_stats' explicitly when importing? I think the answer is yes, because I don't think any promise is being made that the submodule won't be renamed or refactored as scipy development progresses. But I'd like to be sure that that is everyone else's understanding. If that is not the case, then changes that I have made in the past have violated the deprecation policy. For example, a while back I moved the functions in signal/filter_design.py that were related to FIR filters to their own module, fir_filter_design.py. By making appropriate changes to signal/__init__.py, all the function were still importable from scipy.signal. So, from my point of view, I didn't change the public API and no deprecation was necessary. But if scipy.signal.filter_design is part of the public API, then I should have made sure that something like >>> from scipy.signal.filter_design import firwin still worked after the refactor. Note: I'm not saying that *all* submodules should be private and have their objects exposed only through __init__.py. I'm just looking for some clarification of existing policy. Warren

6 12

distributions.ncf.fit -- never converges?
by Yaroslav Halchenko May 6, 2011

May 6, 2011

Since 0.9.0 (not present in 0.7.2), distributions.ncf.fit seems to never converge. could anyone check in development tree if the below call ever returns (I was waiting for minutes... with 0.7.2 it was taking just a moment): python -c 'import scipy.stats.distributions as ssd, numpy as np; d=ssd.ncf; d.fit(np.array([-0.03756533, 0.18115522, 0.06605958, -0.09195276, 0.06162608, 0.07139736, -0.1570828 , -0.11967677, -0.19186652, 0.29411319]));' Thanks in advance! -- =------------------------------------------------------------------= Keep in touch www.onerussian.com Yaroslav Halchenko www.ohloh.net/accounts/yarikoptic

3 8

The state of weave.accelerate?
by Helio Perroni Filho May 5, 2011

May 5, 2011

Dera all, I have been looking for ways to speed up some portions of Python code in my current scipy project. The ideal solution would allow me to keep writing all my code in Python 2.7 – even if the "accelerated" parts had to written in a subset of the language, as in PyPy's Restricted Python – and be easily turned off for debugging. When I learned of scipy.weave.inline, my first thought was "now if only there was a Python-to-C++ translator I could hook up to this!" I was intrigued when I found that cryptic reference about sicpy.weave.accelerate providing "automatic acceleration of Python code"; all the more after searching for more information on it, and finding none. So I decided to take a look at the code. The API seemed simple enough (just a decorator for top-level functions), so I tried it on some toy scripts I keep around. After hacking around a couple simple bugs (two incorrect assertions in bytecodecompiler, a typo defining the one-argument form of the range function and the need to add weave's directory to the compiler include-path in accelerate_tools) I got stuck on the many not-yet-implemented instructions in bytecodecompiler: it seems even basic arithmetic is still to be included. I find this frustrating. From the tests I've run it seems accelerate() already does an adequate work of generating the necessary C++ boilerplate code, compiling it to an extension and then wiring the result back to the executing script. The only thing missing is getting the C++ translation to work – not trivial, I know, but not very hard either, considering what's already in place. Is a lack of manpower holding accelerate() back? If that's the case, I volunteer to work on bytecodecompiler, and get it to at least an RPython-level of functionality. I have read the documentation on the Developer Zone, but didn't find how do I create a Trac account. And besides that, is there anything else not on that page that I should know? -- Ja ne, Helio Perroni Filho http://machineawakening.blogspot.com/

3 3

Discrete-time additions to scipy.signal
by Jeffrey Armstrong May 3, 2011

May 3, 2011

Hi folks, I've been working on some additions to the scipy.signal module. A significant portion of my work involves dealing with linear systems in the discrete-time domain, but I found that most of the functionality I would need in scipy is implemented only in the continuous domain. I've therefore added some code to handle the discrete time cases. My fork is visible at: https://github.com/ArmstrongJ/scipy Additionally, I presented at PyCon 2011, and I briefly discussed a discrete algebraic Riccati equation solver I had implemented in Python. I've included that code into scipy.signal along with a continuous implementation. I'd appreciate some comments and review of the code. Everything seems to be working for me at this point, but any constructive criticism is greatly appreciated. -Jeff Jeff Armstrong - jba(a)sdf.lonestar.org SDF Public Access UNIX System - http://sdf.lonestar.org

3 9

replace print by custom Warnings
by josef.pktd＠gmail.com April 30, 2011

April 30, 2011

should scipy get additional custom warnings instead of print statements ? eg. InterpolationWarning, OptimizationWarning, .... Josef ---------- Forwarded message ---------- From: Yaroslav Halchenko <debian(a)onerussian.com> Date: Wed, Apr 27, 2011 at 10:45 AM Subject: Re: [pymvpa] minor bug ? To: pkg-exppsy-pymvpa(a)lists.alioth.debian.org On Wed, 27 Apr 2011, josef.pktd(a)gmail.com wrote: > You can just temporarily turn off the warnings in the test, if you add > test for corner cases. > That's how Ralf silenced the scipy test suite. and we silence our tests ;-) but apparently not externals checks > If there are print statements, then it would be a bug in scipy, but > these look like "Warnings" where you have control over what happens to > them. Ha -- poking in my eyes works at times! Thanks Josef! Apparently it is indeed numpy warnings so np.seterr(all="ignore") was effective! BUT scipy does use straight printouts of warnings, and that is what made me believe that it was one of them: $> dpkg -L python-scipy | grep '\.py' | xargs grep -i "print.*warn" | head /usr/share/pyshared/scipy/signal/signaltools.py: print "Warning: imaginary part of x ignored." /usr/share/pyshared/scipy/signal/signaltools.py: print "Warning: imaginary part of x ignored." /usr/share/pyshared/scipy/signal/filter_design.py: print "Warning, order is zero...check input parametegstop." /usr/share/pyshared/scipy/interpolate/fitpack.py: if quiet<2:print 'Warning: Setting x[%d][%d]=x[%d][0]'%(i,m,i) /usr/share/pyshared/scipy/interpolate/fitpack.py: print "Warning: "+_iermess[ier][0] /usr/share/pyshared/scipy/interpolate/fitpack.py: print "Warning: "+_iermess[ier][0] /usr/share/pyshared/scipy/interpolate/fitpack.py: print "Warning: the number of zeros exceeds mest" /usr/share/pyshared/scipy/interpolate/fitpack.py: print "Warning: "+_iermess2[ierm][0] /usr/share/pyshared/scipy/integrate/quadpack.py: print "Warning: " + msg /usr/share/pyshared/scipy/maxentropy/maxentutils.py:# print "Warning: could not load the fast FORTRAN library for logsumexp()." (git)novo:~/proj/scipy[tags/v0.9.0]git $> git grep -i 'print .*Warning:' scipy/integrate/quadpack.py: print "Warning: " + msg scipy/interpolate/fitpack.py: if quiet<2:print 'Warning: Setting x[%d][%d]=x[%d][0]'%(i,m,i) scipy/interpolate/fitpack.py: print "Warning: "+_iermess[ier][0] scipy/interpolate/fitpack.py: print "Warning: "+_iermess[ier][0] scipy/interpolate/fitpack.py: print "Warning: the number of zeros exceeds mest" scipy/interpolate/fitpack.py: print "Warning: "+_iermess2[ierm][0] scipy/maxentropy/maxentutils.py:# print "Warning: could not load the fast FORTRAN library for logsumexp()." scipy/maxentropy/maxentutils.py: print "Warning: OverflowError using numpy.exp(). Using slower Python"\ scipy/optimize/anneal.py: print "Warning: Cooled to %f at %s but this is not" \ scipy/optimize/anneal.py: print "Warning: Maximum number of iterations exceeded." scipy/optimize/optimize.py: print "Warning: Maximum number of function evaluations has "\ scipy/optimize/optimize.py: print "Warning: Maximum number of iterations has been exceeded" scipy/optimize/optimize.py: print "Warning: Desired error not necessarily achieved" \ scipy/optimize/optimize.py: print "Warning: Maximum number of iterations has been exceeded" scipy/optimize/optimize.py: print "Warning: Desired error not necessarily achieved due to precision loss" scipy/optimize/optimize.py: print "Warning: Maximum number of iterations has been exceeded" scipy/optimize/optimize.py: print "Warning: Maximum number of iterations has been exceeded" scipy/optimize/optimize.py: print "Warning: Maximum number of function evaluations has "\ scipy/optimize/optimize.py: print "Warning: Maximum number of iterations has been exceeded" scipy/optimize/optimize.py: print "Warning: Final optimization did not succeed" scipy/signal/signaltools.py: print "Warning: imaginary part of x ignored." scipy/signal/signaltools.py: print "Warning: imaginary part of x ignored." scipy/sparse/linalg/dsolve/umfpack/umfpack.py: print 'warning: singular matrix' scipy/sparse/linalg/dsolve/umfpack/umfpack.py: print 'warning: recomputing symbolic' scipy/sparse/linalg/dsolve/umfpack/umfpack.py: print 'warning: (almost) singular matrix! '\ scipy/weave/blitz_tools.py: print 'warning: compilation failed. Executing as python code' scipy/weave/build_tools.py: #print "warning: build directory was not part of python path."\ scipy/weave/build_tools.py: print "warning: specified temp_dir '%s' does not exist " \ scipy/weave/build_tools.py: print "warning:, neither the module's directory nor the "\ scipy/weave/build_tools.py: print 'WARNING: failed to build import library for gcc. Linking will fail.' scipy/weave/catalog.py: print 'warning: default directory is not write accessible.' scipy/weave/catalog.py: print 'warning: default directory is not write accessible.' scipy/weave/catalog.py: print 'warning: unable to repair catalog entry\n %s\n in\n %s' % \ scipy/weave/swig2_spec.py: print "WARNING: Multiple SWIG versions detected. No version was" -- =------------------------------------------------------------------= Keep in touch www.onerussian.com Yaroslav Halchenko www.ohloh.net/accounts/yarikoptic _______________________________________________ Pkg-ExpPsy-PyMVPA mailing list Pkg-ExpPsy-PyMVPA(a)lists.alioth.debian.org http://lists.alioth.debian.org/mailman/listinfo/pkg-exppsy-pymvpa

2 3

Possible funding opportunity for numpy and scipy development
by Thouis (Ray) Jones April 29, 2011

April 29, 2011

(Posted for a friend) This NSF opportunity just renewed last week and might interest the NumPy/SciPy community: NSF Software Infrastructure for Sustained Innovation (SI2) http://www.nsf.gov/funding/pgm_summ.jsp?pims_id=503489 It sounds like it's not suited for end-user software in a particular domain but is intended to be more software infrastructure oriented.

1 0

Editing NumPy User Guide
by Gaston Fiore April 27, 2011

April 27, 2011

Hello, I've just started learning NumPy and I'm reading the Numpy User Guide. While reading, I've noticed a few minor typos that I'd be happy to fix. As such, I was wondering whether I could get edit permissions so I can start slowly contributing to the project. Thanks, -Gaston

2 1

I want to do some translation about the SciPy Reference Guide
by 江大伟 April 23, 2011

April 23, 2011

Hi guys I am a Chinese postgraduate student in the Australian National University. And I am using SciPy to develop some machine learning algorithms. But I found I could not use classes and functions smoothly, because I often encountered many technical and mathematical words which I could understand in Chinese but couldn’t in English. So I was often interrupted when I was thinking, which was quite annoying. So I guess it is absolutely a major stumbling block for other Chinese students to use SciPy. You can’t expect them to learn English well as well as mathematics. So I want to translate the SciPy Reference Guide into Chinese. Is it ok? Can you give me some advice or some help?

5 6

Re: [SciPy-Dev] calc_lwork.f different from Lapack?
by Fabian Pedregosa April 23, 2011

April 23, 2011

On Wed, Apr 13, 2011 at 9:11 PM, Josh Lawrence <josh.k.lawrence(a)gmail.com> wrote: > Hello, > > After receiving an error in using linalg.lstsq, I started some > investigating. I found that scipy/linalg/src/calc_lwork.f is essentially > a stripped down version of the appropriate functions in Lapack (for > linalg.lstsq using double precision complex numbers, it is ZGELSS). It > seems that the portions for gelss in calc_lwork.f do not match the > current version of lapack. I believe this is the source of my error (it > doesn't occur if I use linalg.pinv2 for least squares instead). Is there > a particular rationale for these being different or perhaps an incorrect > copy and paste? The relevant portions are shown below (a line of % > characters are prepended before the codes from the two locations). Hi Josh! I think we could avoid some of these problems by using the LAPACK workspace query mechanism instead of calc_lwork. I submitted a patch for this: https://github.com/scipy/scipy/pull/13 It would be great if you could comment on it and confirm that it solves the problem. Best, Fabian

1 0