Mailman 3 February 2011 - SciPy-Dev

SLSQP Constrained Optimizer Status
by Rob Falck Nov. 24, 2013

Nov. 24, 2013

I'm currently implementing the Sequential Least Squares Quadratic Programming (SLSQP) optimizer, by Dieter Kraft, for use in Scipy. The Fortran code being wrapped with F2PY is here: http://www.netlib.org/toms/733 (its use within Scipy has been cleared) SLSQP provides for bounds on the independent variables, as well as equality and inequality constraint functions, which is a capability that doesn't exist in scipy.optimize. Currently the code works, although the constraint normals are being generated by approximation. I'm working on a way to pass these in. I think the most elegant way will be a single function that returns the matrix of constraint normals. For a demonstration of what the code can do, here is an optimization of f(x,y) = 2xy + 2x - x**2 - 2y**2 Example 14.2 in Chapra & Canale gives the maximum as x=2.0, y=1.0. The unbounded optimization tests find this solution. As expected, its faster when derivatives are provided rather than approximated. Unbounded optimization. Derivatives approximated. Elapsed time: 1.45792961121 ms Results [[1.9999999515712266, 0.99999996181577444], -1.9999999999999984, 4, 0, 'Optimization terminated successfully.'] Unbounded optimization. Derivatives provided. Elapsed time: 1.03211402893 ms Results [[1.9999999515712266, 0.99999996181577444], -1.9999999999999984, 4, 0, 'Optimization terminated successfully.'] The following example uses an equality constraint to find the optimal when x=y. Bound optimization. Derivatives approximated. Elapsed time: 1.384973526 ms Results [[0.99999996845920858, 0.99999996845920858], -0.99999999999999889, 4, 0, 'Optimization terminated successfully.'] I've tried to conform to the syntax used by the other optimizers in scipy.optimize. The function definition and doc string are below. If anyone is interested in testing it out, let me know. def fmin_slsqp( func, x0 , eqcons=[], f_eqcons=None, ieqcons=[], f_ieqcons=None, bounds = [], fprime = None, fprime_cons=None,args = (), iter = 100, acc = 1.0E-6, iprint = 1, full_output = 0, epsilon = _epsilon ): """ Minimize a function using Sequential Least SQuares Programming Description: Python interface function for the SLSQP Optimization subroutine originally implemented by Dieter Kraft. Inputs: func - Objective function (in the form func(x, *args)) x0 - Initial guess for the independent variable(s). eqcons - A list of functions of length n such that eqcons[j](x0,*args) == 0.0 in a successfully optimized problem f_eqcons - A function of the form f_eqcons(x, *args) that returns an array in which each element must equal 0.0 in a successfully optimized problem. If f_eqcons is specified, eqcons is ignored. ieqcons - A list of functions of length n such that ieqcons[j](x0,*args) >= 0.0 in a successfully optimized problem f_ieqcons - A function of the form f_ieqcons(x0, *args) that returns an array in which each element must be greater or equal to 0.0 in a successfully optimized problem. If f_ieqcons is specified, ieqcons is ignored. bounds - A list of tuples specifying the lower and upper bound for each independent variable [ (xl0, xu0), (xl1, xu1), ...] fprime - A function that evaluates the partial derivatives of func fprime_cons - A function of the form f(x, *args) that returns the m by n array of constraint normals. If not provided, the normals will be approximated. Equality constraint normals precede inequality constraint normals. args - A sequence of additional arguments passed to func and fprime iter - The maximum number of iterations (int) acc - Requested accuracy (float) iprint - The verbosity of fmin_slsqp. iprint <= 0 : Silent operation iprint == 1 : Print summary upon completion (default) iprint >= 2 : Print status of each iterate and summary full_output - If 0, return only the minimizer of func (default). Otherwise, output final objective function and summary information. epsilon - The step size for finite-difference derivative estimates. Outputs: ( x, { fx, gnorm, its, imode, smode }) x - The final minimizer of func. fx - The final value of the objective function. its - The number of iterations. imode - The exit mode from the optimizer, as an integer. smode - A string describing the exit mode from the optimizer. Exit modes are defined as follows: -1 : Gradient evaluation required (g & a) 0 : Optimization terminated successfully. 1 : Function evaluation required (f & c) 2 : Number of equality constraints larger than number of independent variables 3 : More than 3*n iterations in LSQ subproblem 4 : Inequality constraints incompatible 5 : Singular matrix E in LSQ subproblem 6 : Singular matrix C in LSQ subproblem 7 : Rank-deficient equality constraint subproblem HFTI 8 : Positive directional derivative for linesearch 9 : Iteration limit exceeded -- - Rob Falck

4 16

Question about subpackage/submodule API
by Warren Weckesser May 23, 2011

May 23, 2011

The recent discussion about what to call the module in Pim Schellart's nice Lomb-Scargle contribution brought up a question for me that I have not seen discussed before. In many cases, a subpackage contains many modules, and the assorted functions and classes in these modules are made available in the (sub)package-level namespace by importing them from within __init__.py. For example, scipy/stats contains distributions.py, kde.py, morestats.py, mstats.py, rv.py, stats.py and vonmises.py. Except for mstats.py, the contents of all these are imported into the __init__.py namespace and included in __all__, so users can say, for example, >>> from scipy.stats import bayes_mvs instead of >>> from scipy.stats.more_stats import bayes_mvs Is this an *intentional* definition of an API? For example, is it "wrong" for a user to refer to the submodule 'more_stats' explicitly when importing? I think the answer is yes, because I don't think any promise is being made that the submodule won't be renamed or refactored as scipy development progresses. But I'd like to be sure that that is everyone else's understanding. If that is not the case, then changes that I have made in the past have violated the deprecation policy. For example, a while back I moved the functions in signal/filter_design.py that were related to FIR filters to their own module, fir_filter_design.py. By making appropriate changes to signal/__init__.py, all the function were still importable from scipy.signal. So, from my point of view, I didn't change the public API and no deprecation was necessary. But if scipy.signal.filter_design is part of the public API, then I should have made sure that something like >>> from scipy.signal.filter_design import firwin still worked after the refactor. Note: I'm not saying that *all* submodules should be private and have their objects exposed only through __init__.py. I'm just looking for some clarification of existing policy. Warren

6 12

Initial support for Harwell Boeing sparse matrix format
by David Cournapeau March 1, 2011

March 1, 2011

Hi there, I have added support for simple read/write for HB matrix format. The updated branch is on github: https://github.com/cournape/scipy3/compare/master..._new_hb_io It adds a high-level API (read_hb/write_hb) as well as a lower-level API for fine-grained control. The functions live in scipy.sparse.io. I already put this patch last year, without any reaction, so I will assume that no reaction means yes :) cheers, David

6 7

ANN: SciPy 0.9.0
by Ralf Gommers Feb. 28, 2011

Feb. 28, 2011

I'm pleased to announce the release of SciPy 0.9.0. SciPy is a package of tools for science and engineering for Python. It includes modules for statistics, optimization, integration, linear algebra, Fourier transforms, signal and image processing, ODE solvers, and more. This release comes seven months after the 0.8.0 release and contains several new features, numerous bug-fixes, improved test coverage, and better documentation. This is the first release that supports Python 3 (with the exception of the scipy.weave module). Sources, binaries, documentation and release notes can be found at http://sourceforge.net/projects/scipy/files/scipy/0.9.0/ Thank you to everybody who contributed to this release. Enjoy, The SciPy developers ========================= SciPy 0.9.0 Release Notes ========================= .. contents:: SciPy 0.9.0 is the culmination of 6 months of hard work. It contains many new features, numerous bug-fixes, improved test coverage and better documentation. There have been a number of deprecations and API changes in this release, which are documented below. All users are encouraged to upgrade to this release, as there are a large number of bug-fixes and optimizations. Moreover, our development attention will now shift to bug-fix releases on the 0.9.x branch, and on adding new features on the development trunk. This release requires Python 2.4 - 2.7 or 3.1 - and NumPy 1.5 or greater. Please note that SciPy is still considered to have "Beta" status, as we work toward a SciPy 1.0.0 release. The 1.0.0 release will mark a major milestone in the development of SciPy, after which changing the package structure or API will be much more difficult. Whilst these pre-1.0 releases are considered to have "Beta" status, we are committed to making them as bug-free as possible. However, until the 1.0 release, we are aggressively reviewing and refining the functionality, organization, and interface. This is being done in an effort to make the package as coherent, intuitive, and useful as possible. To achieve this, we need help from the community of users. Specifically, we need feedback regarding all aspects of the project - everything - from which algorithms we implement, to details about our function's call signatures. Python 3 ======== Scipy 0.9.0 is the first SciPy release to support Python 3. The only module that is not yet ported is ``scipy.weave``. Scipy source code location to be changed ======================================== Soon after this release, Scipy will stop using SVN as the version control system, and move to Git. The development source code for Scipy can from then on be found at http://github.com/scipy/scipy New features ============ Delaunay tesselations (``scipy.spatial``) ----------------------------------------- Scipy now includes routines for computing Delaunay tesselations in N dimensions, powered by the Qhull_ computational geometry library. Such calculations can now make use of the new ``scipy.spatial.Delaunay`` interface. .. _Qhull: http://www.qhull.org/ N-dimensional interpolation (``scipy.interpolate``) --------------------------------------------------- Support for scattered data interpolation is now significantly improved. This version includes a ``scipy.interpolate.griddata`` function that can perform linear and nearest-neighbour interpolation for N-dimensional scattered data, in addition to cubic spline (C1-smooth) interpolation in 2D and 1D. An object-oriented interface to each interpolator type is also available. Nonlinear equation solvers (``scipy.optimize``) ----------------------------------------------- Scipy includes new routines for large-scale nonlinear equation solving in ``scipy.optimize``. The following methods are implemented: * Newton-Krylov (``scipy.optimize.newton_krylov``) * (Generalized) secant methods: - Limited-memory Broyden methods (``scipy.optimize.broyden1``, ``scipy.optimize.broyden2``) - Anderson method (``scipy.optimize.anderson``) * Simple iterations (``scipy.optimize.diagbroyden``, ``scipy.optimize.excitingmixing``, ``scipy.optimize.linearmixing``) The ``scipy.optimize.nonlin`` module was completely rewritten, and some of the functions were deprecated (see above). New linear algebra routines (``scipy.linalg``) ---------------------------------------------- Scipy now contains routines for effectively solving triangular equation systems (``scipy.linalg.solve_triangular``). Improved FIR filter design functions (``scipy.signal``) ------------------------------------------------------- The function ``scipy.signal.firwin`` was enhanced to allow the design of highpass, bandpass, bandstop and multi-band FIR filters. The function ``scipy.signal.firwin2`` was added. This function uses the window method to create a linear phase FIR filter with an arbitrary frequency response. The functions ``scipy.signal.kaiser_atten`` and ``scipy.signal.kaiser_beta`` were added. Improved statistical tests (``scipy.stats``) -------------------------------------------- A new function ``scipy.stats.fisher_exact`` was added, that provides Fisher's exact test for 2x2 contingency tables. The function ``scipy.stats.kendalltau`` was rewritten to make it much faster (O(n log(n)) vs O(n^2)). Deprecated features =================== Obsolete nonlinear solvers (in ``scipy.optimize``) -------------------------------------------------- The following nonlinear solvers from ``scipy.optimize`` are deprecated: - ``broyden_modified`` (bad performance) - ``broyden1_modified`` (bad performance) - ``broyden_generalized`` (equivalent to ``anderson``) - ``anderson2`` (equivalent to ``anderson``) - ``broyden3`` (obsoleted by new limited-memory broyden methods) - ``vackar`` (renamed to ``diagbroyden``) Removed features ================ The deprecated modules ``helpmod``, ``pexec`` and ``ppimport`` were removed from ``scipy.misc``. The ``output_type`` keyword in many ``scipy.ndimage`` interpolation functions has been removed. The ``econ`` keyword in ``scipy.linalg.qr`` has been removed. The same functionality is still available by specifying ``mode='economic'``. Old correlate/convolve behavior (in ``scipy.signal``) ----------------------------------------------------- The old behavior for ``scipy.signal.convolve``, ``scipy.signal.convolve2d``, ``scipy.signal.correlate`` and ``scipy.signal.correlate2d`` was deprecated in 0.8.0 and has now been removed. Convolve and correlate used to swap their arguments if the second argument has dimensions larger than the first one, and the mode was relative to the input with the largest dimension. The current behavior is to never swap the inputs, which is what most people expect, and is how correlation is usually defined. ``scipy.stats`` --------------- Many functions in ``scipy.stats`` that are either available from numpy or have been superseded, and have been deprecated since version 0.7, have been removed: `std`, `var`, `mean`, `median`, `cov`, `corrcoef`, `z`, `zs`, `stderr`, `samplestd`, `samplevar`, `pdfapprox`, `pdf_moments` and `erfc`. These changes are mirrored in ``scipy.stats.mstats``. ``scipy.sparse`` ---------------- Several methods of the sparse matrix classes in ``scipy.sparse`` which had been deprecated since version 0.7 were removed: `save`, `rowcol`, `getdata`, `listprint`, `ensure_sorted_indices`, `matvec`, `matmat` and `rmatvec`. The functions ``spkron``, ``speye``, ``spidentity``, ``lil_eye`` and ``lil_diags`` were removed from ``scipy.sparse``. The first three functions are still available as ``scipy.sparse.kron``, ``scipy.sparse.eye`` and ``scipy.sparse.identity``. The `dims` and `nzmax` keywords were removed from the sparse matrix constructor. The `colind` and `rowind` attributes were removed from CSR and CSC matrices respectively. ``scipy.sparse.linalg.arpack.speigs`` ------------------------------------- A duplicated interface to the ARPACK library was removed. Other changes ============= ARPACK interface changes ------------------------ The interface to the ARPACK eigenvalue routines in ``scipy.sparse.linalg`` was changed for more robustness. The eigenvalue and SVD routines now raise ``ArpackNoConvergence`` if the eigenvalue iteration fails to converge. If partially converged results are desired, they can be accessed as follows:: import numpy as np from scipy.sparse.linalg import eigs, ArpackNoConvergence m = np.random.randn(30, 30) try: w, v = eigs(m, 6) except ArpackNoConvergence, err: partially_converged_w = err.eigenvalues partially_converged_v = err.eigenvectors Several bugs were also fixed. The routines were moreover renamed as follows: - eigen --> eigs - eigen_symmetric --> eigsh - svd --> svds

1 0

2to3 backup files in scipy installers
by Christoph Gohlke Feb. 27, 2011

Feb. 27, 2011

Hello, the scipy 0.9 rc5 installers for Python 3.1 contain many *.bak files created during the 2to3 conversion. A patch is attached. Other than that, for the first time I got scipy.test() to run without any crashes, errors, or failures on win-amd64-py2.7. Much better than the 16 errors and 9 failures reported for scipy 0.8b1 <http://mail.scipy.org/pipermail/scipy-user/2010-July/025961.html>. Thank you! Christoph

3 2

Re: [SciPy-Dev] optimize.fsolve too accurate
by J Vankerschaver Feb. 25, 2011

Feb. 25, 2011

Hi guys, Thanks for the great replies to my optimize.fsolve query! I'm still not too happy with minpack.hybrd only dealing with relative tolerances, so I'll stick to using the Newton solver for my problem (it doesn't have a double zero -- that only came up when I was looking for a toy problem). Numscons looks like an easy to use build system, so I'm glad that I can play with the scipy internals too! All the best, Joris > Message: 3 > Date: Thu, 24 Feb 2011 22:01:00 +0800 > From: Ralf Gommers <ralf.gommers(a)googlemail.com> > Subject: Re: [SciPy-Dev] optimize.fsolve too accurate > To: SciPy Developers List <scipy-dev(a)scipy.org> > Message-ID: > <AANLkTimfpvTXX34NOnBXiRKBASgKjH6Fc-3mv_b-T+Wp(a)mail.gmail.com> > Content-Type: text/plain; charset=ISO-8859-1 > > On Thu, Feb 24, 2011 at 2:38 PM, Joris Vankerschaver > <joris.vankerschaver(a)gmail.com> wrote: > > > > Secondly, is there a quick and easy way to build parts of the scipy > > library in place for testing? ?I looked at the implementation and there > > are a few things I would like to experiment with, but I don't know how > > to go about this other than by rebuilding scipy in its entirety. > > > If you want to only experiment with Python code, you can build in-place > with > $ python setup.py build_ext -i > > For compiled code, you're much better off with numscons: > $ # in-place build > $ python setupscons.py scons -i > $ # partial rebuild of optimize module > $ python setupscons.py scons -i --package-list=scipy.optimize > You'll need to install numscons, see > http://projects.scipy.org/numpy/wiki/NumScons. > > Ralf > > > ------------------------------ > > Message: 4 > Date: Thu, 24 Feb 2011 16:44:02 +0200 > From: Yosef Meller <mellerf(a)netvision.net.il> > Subject: Re: [SciPy-Dev] optimize.fsolve too accurate > To: scipy-dev(a)scipy.org > Message-ID: <4D666EB2.5070605(a)netvision.net.il> > Content-Type: text/plain; charset=UTF-8; format=flowed > > ?????? 24/02/11 08:38, ????? Joris Vankerschaver: > > I'm relatively new to SciPy, so I hope you will excuse the occasional > > inaccuracies, but I noticed something slightly strange with > > scipy.optimize.fsolve: the following snippet > > > > import scipy.optimize > > fun = lambda x: x**2 > > scipy.optimize.fsolve(fun, 0.5) > > > > returns "Warning: The number of calls to function has reached maxfev = > > 400", while the answer is presented as 2.8405269166788003e-84 (you > > might have to change the initial conditions somewhat to obtain this > > error). As the default tolerance for fsolve is roughly 1.4e-8, the > > computation should have terminated long before reaching this level of > > precision. > > > > My question: is this a bug or am I invoking fsolve in the wrong way? > > I can confirm that this happens on the first time the code is run. > Second call to fsolve with the same arguments runs as expected. There > might be something wrong with module initialization or something of the > sort. > > > ------------------------------ > > Message: 5 > Date: Thu, 24 Feb 2011 14:48:17 +0000 (UTC) > From: Pauli Virtanen <pav(a)iki.fi> > Subject: Re: [SciPy-Dev] optimize.fsolve too accurate > To: scipy-dev(a)scipy.org > Message-ID: <ik5r3g$n9f$1(a)dough.gmane.org> > Content-Type: text/plain; charset=UTF-8 > > Thu, 24 Feb 2011 16:44:02 +0200, Yosef Meller wrote: > [clip] > > I can confirm that this happens on the first time the code is run. > > Second call to fsolve with the same arguments runs as expected. There > > might be something wrong with module initialization or something of the > > sort. > > Python by default shows each warning only once per occurrence. The > interactive session is treated as a single location, and so each warning > shows up only once, even though it is issued every time. > > The main issue here is that MINPACK only terminates based on the relative > tolerance of the result --- which does not make much sense if the exact > solution happens to be 0. > > -- > Pauli Virtanen > > > > ------------------------------ > > Message: 6 > Date: Thu, 24 Feb 2011 07:54:38 -0700 > From: Charles R Harris <charlesr.harris(a)gmail.com> > Subject: Re: [SciPy-Dev] optimize.fsolve too accurate > To: SciPy Developers List <scipy-dev(a)scipy.org> > Message-ID: > <AANLkTi=s5nT06aMpdN9yiYnJ70CTddJ9hB=_WD8unDih(a)mail.gmail.com> > Content-Type: text/plain; charset="iso-8859-1" > > On Wed, Feb 23, 2011 at 11:38 PM, Joris Vankerschaver < > joris.vankerschaver(a)gmail.com> wrote: > > > Hi all, > > > > I'm relatively new to SciPy, so I hope you will excuse the occasional > > inaccuracies, but I noticed something slightly strange with > > scipy.optimize.fsolve: the following snippet > > > > import scipy.optimize > > fun = lambda x: x**2 > > scipy.optimize.fsolve(fun, 0.5) > > > > > Just a side note, but the zero solvers are generally much better for the > one > dimensional case. Although I suspect the problem here is the double zero. > > In [1]: import scipy.optimize > > In [2]: fun = lambda x: x**2 - 1e-10 > > In [3]: scipy.optimize.fsolve(fun, 0.5) > Out[3]: array([ 1.00000000e-05]) > > > > > returns "Warning: The number of calls to function has reached maxfev = > > 400", while the answer is presented as 2.8405269166788003e-84 (you > > might have to change the initial conditions somewhat to obtain this > > error). As the default tolerance for fsolve is roughly 1.4e-8, the > > computation should have terminated long before reaching this level of > > precision. > > > > My question: is this a bug or am I invoking fsolve in the wrong way? > > > > > It looks like fsolve only uses relative errors, so that is probably a > problem near zero. Newton does better > > In [5]: fun = lambda x: x**2 > > In [6]: scipy.optimize.newton(fun, 0.5) > Out[6]: 2.0701071395499238e-08 > > >

1 0

optimize.fsolve too accurate
by Joris Vankerschaver Feb. 24, 2011

Feb. 24, 2011

Hi all, I'm relatively new to SciPy, so I hope you will excuse the occasional inaccuracies, but I noticed something slightly strange with scipy.optimize.fsolve: the following snippet import scipy.optimize fun = lambda x: x**2 scipy.optimize.fsolve(fun, 0.5) returns "Warning: The number of calls to function has reached maxfev = 400", while the answer is presented as 2.8405269166788003e-84 (you might have to change the initial conditions somewhat to obtain this error). As the default tolerance for fsolve is roughly 1.4e-8, the computation should have terminated long before reaching this level of precision. My question: is this a bug or am I invoking fsolve in the wrong way? Secondly, is there a quick and easy way to build parts of the scipy library in place for testing? I looked at the implementation and there are a few things I would like to experiment with, but I don't know how to go about this other than by rebuilding scipy in its entirety. Thanks! Joris

5 4

ANN: inSCIght, The Scientific Computing Podcast
by Anthony Scopatz Feb. 23, 2011

Feb. 23, 2011

Hello All, I am very pleased to announce inSCIght, a new scientific computing podcast (press release below). I apologize for those of you in the intersection of these lists that may receive this message multiple times. As I mention in the press release, we are very open to your contributions! Be Well Anthony inSCIght: The Scientific Computing Podcast ========================================== 'inSCIght' is a podcast that focuses on scientific computing in all of its various forms. Every week we have a few panelists engage head-to-head on poignant and interesting topics. The panelists are drawn from all across the scientific computing community. From embedded systems experts to very high level language gurus, biologists and nuclear engineers, the hosts of inSCIght use computers to solve science and engineering problems everyday. This podcast throws people, ideas, and opinions into an audio-blender hoping to educate and entice each other and the world. You can find us at: * inSCIght.org (http://inscight.org/), * Twitter (http://twitter.com/inscight/), * Convore (https://convore.com/inscight/), * and GitHub (https://github.com/inscight/). Furthermore, we are are always looking to supplement our current repertoire of hosts and special guests. So if you would like to contribute to inSCIght or have something interesting to present on a show, feel free to email us at info_AT_inscight.org. We'd love to have you join the conversation! The inSCIght podcast is a co-production of Enthought, Software Carpentry, and The Hacker Within. Thanks for listening! The inSCIght podcast is licensed under the Creative Commons Attribution 3.0 Unported (CC BY 3.0) license.

1 0

scipy.integrate's dopri5 methods doesn't pass additional arguments
by James Hensman Feb. 23, 2011

Feb. 23, 2011

in scipy/integrate/ode.py, line 745 in the dopri5 class. the following is needed in order to pass additional arguments to the function to be integrated: < x,y,iwork,idid = self.runner(*((f,t0,y0,t1) + tuple(self.call_args))) --- > x,y,iwork,idid = self.runner(*((f,t0,y0,t1) + tuple(self.call_args)+(f_params,))) Please excuse any faux pas I may have made: this is my first post to such a list. James.

2 1

SciPy and the Google Summer of Code
by Joosep Pata Feb. 22, 2011

Feb. 22, 2011

Hi, This is my first post to the mailing list. My name is Joosep and I'm a physics major from Estonia. Are SciPy and NumPy taking part in GSOC this year? If so, how would one go about applying for it? I'd really like to get into open-source development and SciPy/NumPy seems like the way to go. I have some previous experience with C++ and Python and I have done numerical methods before. What would need doing? I'd of course be happy to do development outside of GSOC as well. Cheers, Joosep

1 0