Mailman 3 February 2012 - SciPy-Dev

SLSQP Constrained Optimizer Status
by Rob Falck Nov. 24, 2013

Nov. 24, 2013

I'm currently implementing the Sequential Least Squares Quadratic Programming (SLSQP) optimizer, by Dieter Kraft, for use in Scipy. The Fortran code being wrapped with F2PY is here: http://www.netlib.org/toms/733 (its use within Scipy has been cleared) SLSQP provides for bounds on the independent variables, as well as equality and inequality constraint functions, which is a capability that doesn't exist in scipy.optimize. Currently the code works, although the constraint normals are being generated by approximation. I'm working on a way to pass these in. I think the most elegant way will be a single function that returns the matrix of constraint normals. For a demonstration of what the code can do, here is an optimization of f(x,y) = 2xy + 2x - x**2 - 2y**2 Example 14.2 in Chapra & Canale gives the maximum as x=2.0, y=1.0. The unbounded optimization tests find this solution. As expected, its faster when derivatives are provided rather than approximated. Unbounded optimization. Derivatives approximated. Elapsed time: 1.45792961121 ms Results [[1.9999999515712266, 0.99999996181577444], -1.9999999999999984, 4, 0, 'Optimization terminated successfully.'] Unbounded optimization. Derivatives provided. Elapsed time: 1.03211402893 ms Results [[1.9999999515712266, 0.99999996181577444], -1.9999999999999984, 4, 0, 'Optimization terminated successfully.'] The following example uses an equality constraint to find the optimal when x=y. Bound optimization. Derivatives approximated. Elapsed time: 1.384973526 ms Results [[0.99999996845920858, 0.99999996845920858], -0.99999999999999889, 4, 0, 'Optimization terminated successfully.'] I've tried to conform to the syntax used by the other optimizers in scipy.optimize. The function definition and doc string are below. If anyone is interested in testing it out, let me know. def fmin_slsqp( func, x0 , eqcons=[], f_eqcons=None, ieqcons=[], f_ieqcons=None, bounds = [], fprime = None, fprime_cons=None,args = (), iter = 100, acc = 1.0E-6, iprint = 1, full_output = 0, epsilon = _epsilon ): """ Minimize a function using Sequential Least SQuares Programming Description: Python interface function for the SLSQP Optimization subroutine originally implemented by Dieter Kraft. Inputs: func - Objective function (in the form func(x, *args)) x0 - Initial guess for the independent variable(s). eqcons - A list of functions of length n such that eqcons[j](x0,*args) == 0.0 in a successfully optimized problem f_eqcons - A function of the form f_eqcons(x, *args) that returns an array in which each element must equal 0.0 in a successfully optimized problem. If f_eqcons is specified, eqcons is ignored. ieqcons - A list of functions of length n such that ieqcons[j](x0,*args) >= 0.0 in a successfully optimized problem f_ieqcons - A function of the form f_ieqcons(x0, *args) that returns an array in which each element must be greater or equal to 0.0 in a successfully optimized problem. If f_ieqcons is specified, ieqcons is ignored. bounds - A list of tuples specifying the lower and upper bound for each independent variable [ (xl0, xu0), (xl1, xu1), ...] fprime - A function that evaluates the partial derivatives of func fprime_cons - A function of the form f(x, *args) that returns the m by n array of constraint normals. If not provided, the normals will be approximated. Equality constraint normals precede inequality constraint normals. args - A sequence of additional arguments passed to func and fprime iter - The maximum number of iterations (int) acc - Requested accuracy (float) iprint - The verbosity of fmin_slsqp. iprint <= 0 : Silent operation iprint == 1 : Print summary upon completion (default) iprint >= 2 : Print status of each iterate and summary full_output - If 0, return only the minimizer of func (default). Otherwise, output final objective function and summary information. epsilon - The step size for finite-difference derivative estimates. Outputs: ( x, { fx, gnorm, its, imode, smode }) x - The final minimizer of func. fx - The final value of the objective function. its - The number of iterations. imode - The exit mode from the optimizer, as an integer. smode - A string describing the exit mode from the optimizer. Exit modes are defined as follows: -1 : Gradient evaluation required (g & a) 0 : Optimization terminated successfully. 1 : Function evaluation required (f & c) 2 : Number of equality constraints larger than number of independent variables 3 : More than 3*n iterations in LSQ subproblem 4 : Inequality constraints incompatible 5 : Singular matrix E in LSQ subproblem 6 : Singular matrix C in LSQ subproblem 7 : Rank-deficient equality constraint subproblem HFTI 8 : Positive directional derivative for linesearch 9 : Iteration limit exceeded -- - Rob Falck

4 16

prepending underscores to private module names
by Ralf Gommers May 16, 2012

May 16, 2012

Hi, A while ago we discussed making a clear distinction between public and private modules by using underscores consistently, see http://thread.gmane.org/gmane.comp.python.scientific.devel/14936. I've done this for a few modules now, see https://github.com/rgommers/scipy/tree/underscores. Most files could simply be underscored, some other would conflict with extension modules so I also had to append _py to the name. Also I added a file doc/source/api.rst that states which modules are public (up for discussion of course). With google code search I checked how common it is to import from modules that are supposed to be private, for example "from scipy.optimize.optimize import fmin". It's not very common but does happen, so deprecation warnings should be put in. Before going further I'd like to check that this looks okay to people. What do you think? Cheers, Ralf <https://github.com/rgommers/scipy/tree/underscores>

4 4

studentized range approximations
by Roger Lew March 28, 2012

March 28, 2012

Hi, I have implemented some approximations for studentized range quantiles and probabilities based on John R. Gleason's (1999) "An accurate, non-iterative approximation for studentized range quantiles." Computational Statistics & Data Analysis, (31), 147-158. Probability approximations rely on scipy.optimize.fminbound. The functions accept both scalars or array-like data thanks to numpy.vectorize. A fair amount of validation and testing has been conducted on the code. More details can be found here: http://code.google.com/p/qsturng-py/ I welcome any thoughts as to whether you all think this might be useful to add to SciPy or make into a scikit. Any general comments would be helpful as well. I should mention I'm a cognitive neuroscientist by trade, my use of statistical jargon probably isn't that good. Regards, Roger Roger Lew

3 5

new.scipy.org
by Thomas Kluyver March 24, 2012

March 24, 2012

I've found that searches can turn up pages from new.scipy.org, a subdomain which was seemingly last updated in 2009, and now has out of date info - e.g. references to SVN (http://new.scipy.org/download.html) and FAQ about Python 3 support (http://new.scipy.org/faq.html#python-version-support). I can't see any details of who maintains the website, but I think leaving out of date information around is liable to cause confusion - can it be taken down or brought up to date? Thanks, Thomas

5 6

Update on pythonequations unit tests
by James Phillips March 14, 2012

March 14, 2012

From: Alan G Isaac <alan.isaac <at> gmail.com> Subject: Re: Subversion scipy.stats irregular problem with source code example<http://news.gmane.org/find-root.php?message_id=%3c4CA22FA3.5080402%40gmail.…> Newsgroups: gmane.comp.python.scientific.devel<http://news.gmane.org/gmane.comp.python.scientific.devel> Date: 2010-09-28 18:10:43 GMT (1 year, 2 weeks, 1 day, 15 hours and 46 minutes ago) As long as you can provide unit tests, I don't see a problem. But you and Skipper shd work out the details. Cheers, Alan Isaac ------------------------------------------------------------------------------------------------------------------ I have 76 unit tests so far, see below. I'm working on more, and expect to be done sometime around the end of this year. James Phillips http://zunzun.com testCalculateCoefficientAndFitStatisticsUsingDE_2D (Test_CalculateCoefficientAndFitStatistics.TestCalculateCoefficientAndFitStatistics) ... ok testCalculateCoefficientAndFitStatisticsUsingSpline_2D (Test_CalculateCoefficientAndFitStatistics.TestCalculateCoefficientAndFitStatistics) ... ok testDataCache_2D (Test_DataCache.TestDataCache) ... ok testDataCache_3D (Test_DataCache.TestDataCache) ... ok testReducedDataSize_2D (Test_DataCache.TestDataCache) ... ok testConversionOfColumns_ASCII_2D_NoWeights (Test_DataConverterService.TestConversions) ... ok testConversionOfColumns_ASCII_2D_NoWeights_ExampleData (Test_DataConverterService.TestConversions) ... ok testConversionOfColumns_ASCII_2D_Weights (Test_DataConverterService.TestConversions) ... ok testConversionOfColumns_ASCII_3D_NoWeights (Test_DataConverterService.TestConversions) ... ok testConversionOfColumns_ASCII_3D_Weights (Test_DataConverterService.TestConversions) ... ok testPolynomialLinear_2D (Test_EquationPolynomials.TestEquationPolynomials) ... ok testPolynomialLinear_3D (Test_EquationPolynomials.TestEquationPolynomials) ... ok testExtendedVersion_Exponential_WithExponentialDecayAndOffset_2D (Test_ExtendedVersionHandlers.TestExtendedVersionHandlers) ... ok testExtendedVersion_Exponential_WithExponentialDecay_2D (Test_ExtendedVersionHandlers.TestExtendedVersionHandlers) ... ok testExtendedVersion_Exponential_WithExponentialGrowthAndOffset_2D (Test_ExtendedVersionHandlers.TestExtendedVersionHandlers) ... ok testExtendedVersion_Exponential_WithExponentialGrowth_2D (Test_ExtendedVersionHandlers.TestExtendedVersionHandlers) ... ok testExtendedVersion_Exponential_WithLinearDecayAndOffset_2D (Test_ExtendedVersionHandlers.TestExtendedVersionHandlers) ... ok testExtendedVersion_Exponential_WithLinearDecay_2D (Test_ExtendedVersionHandlers.TestExtendedVersionHandlers) ... ok testExtendedVersion_Exponential_WithLinearGrowthAndOffset_2D (Test_ExtendedVersionHandlers.TestExtendedVersionHandlers) ... ok testExtendedVersion_Exponential_WithLinearGrowth_2D (Test_ExtendedVersionHandlers.TestExtendedVersionHandlers) ... ok testExtendedVersion_Exponential_WithOffset_2D (Test_ExtendedVersionHandlers.TestExtendedVersionHandlers) ... ok testExtendedVersion_Inverse_Exponential_2D (Test_ExtendedVersionHandlers.TestExtendedVersionHandlers) ... ok testExtendedVersion_Inverse_Exponential_WithOffset_2D (Test_ExtendedVersionHandlers.TestExtendedVersionHandlers) ... ok testExtendedVersion_Reciprocal_Exponential_2D (Test_ExtendedVersionHandlers.TestExtendedVersionHandlers) ... ok testExtendedVersion_Reciprocal_Exponential_WithOffset_2D (Test_ExtendedVersionHandlers.TestExtendedVersionHandlers) ... ok testArcTangent_Term (Test_IndividualPolyFunctions.TestPolyFunctions) ... ok testCosine_Term (Test_IndividualPolyFunctions.TestPolyFunctions) ... ok testExponential_VariableTimesNegativeOne_Term (Test_IndividualPolyFunctions.TestPolyFunctions) ... ok testExponential_VariableUnchanged_Term (Test_IndividualPolyFunctions.TestPolyFunctions) ... ok testHyperbolicCosine_Term (Test_IndividualPolyFunctions.TestPolyFunctions) ... ok testHyperbolicSine_Term (Test_IndividualPolyFunctions.TestPolyFunctions) ... ok testHyperbolicTangent_Term (Test_IndividualPolyFunctions.TestPolyFunctions) ... ok testLog_Term (Test_IndividualPolyFunctions.TestPolyFunctions) ... ok testOffset_Term (Test_IndividualPolyFunctions.TestPolyFunctions) ... ok testPower_NegativeOne_OfLog_Term (Test_IndividualPolyFunctions.TestPolyFunctions) ... ok testPower_NegativeOne_Term (Test_IndividualPolyFunctions.TestPolyFunctions) ... ok testPower_NegativeTwo_OfLog_Term (Test_IndividualPolyFunctions.TestPolyFunctions) ... ok testPower_NegativeTwo_Term (Test_IndividualPolyFunctions.TestPolyFunctions) ... ok testPower_NegativeZeroPointFive_Term (Test_IndividualPolyFunctions.TestPolyFunctions) ... ok testPower_OnePointFive_Term (Test_IndividualPolyFunctions.TestPolyFunctions) ... ok testPower_Two_OfLog_Term (Test_IndividualPolyFunctions.TestPolyFunctions) ... ok testPower_Two_Term (Test_IndividualPolyFunctions.TestPolyFunctions) ... ok testPower_ZeroPointFive_Term (Test_IndividualPolyFunctions.TestPolyFunctions) ... ok testSine_Term (Test_IndividualPolyFunctions.TestPolyFunctions) ... ok testTangent_Term (Test_IndividualPolyFunctions.TestPolyFunctions) ... ok testVariableUnchanged_Term (Test_IndividualPolyFunctions.TestPolyFunctions) ... ok testSplineSolve_2D (Test_ModelSolveMethods.TestModelSolveMethods) ... ok testSplineSolve_3D (Test_ModelSolveMethods.TestModelSolveMethods) ... ok testUserDefinedFunctionSolve_2D (Test_ModelSolveMethods.TestModelSolveMethods) ... ok testUserDefinedFunctionSolve_3D (Test_ModelSolveMethods.TestModelSolveMethods) ... ok testConversionFromCppToCSHARP (Test_OutputSourceCodeService.TestConversionsFromCPP) ... ok testConversionFromCppToJAVA (Test_OutputSourceCodeService.TestConversionsFromCPP) ... ok testConversionFromCppToMATLAB (Test_OutputSourceCodeService.TestConversionsFromCPP) ... ok testConversionFromCppToPYTHON (Test_OutputSourceCodeService.TestConversionsFromCPP) ... ok testConversionFromCppToSCILAB (Test_OutputSourceCodeService.TestConversionsFromCPP) ... ok testConversionFromCppToVBA (Test_OutputSourceCodeService.TestConversionsFromCPP) ... ok testGenerationOfCPP (Test_OutputSourceCodeService.TestGenerationOfOutputSourceCode) ... ok testGenerationOfCSHARP (Test_OutputSourceCodeService.TestGenerationOfOutputSourceCode) ... ok testGenerationOfJAVA (Test_OutputSourceCodeService.TestGenerationOfOutputSourceCode) ... ok testGenerationOfMATLAB (Test_OutputSourceCodeService.TestGenerationOfOutputSourceCode) ... ok testGenerationOfPYTHON (Test_OutputSourceCodeService.TestGenerationOfOutputSourceCode) ... ok testGenerationOfSCILAB (Test_OutputSourceCodeService.TestGenerationOfOutputSourceCode) ... ok testGenerationOfVBA (Test_OutputSourceCodeService.TestGenerationOfOutputSourceCode) ... ok testSolveUsingDE_2D (Test_SolverService.TestSolverService) ... ok testSolveUsingDE_3D (Test_SolverService.TestSolverService) ... ok testSolveUsingLevenbergMarquardt_2D (Test_SolverService.TestSolverService) ... ok testSolveUsingLevenbergMarquardt_3D (Test_SolverService.TestSolverService) ... ok testSolveUsingLinear_2D (Test_SolverService.TestSolverService) ... ok testSolveUsingLinear_3D (Test_SolverService.TestSolverService) ... ok testSolveUsingODR_2D (Test_SolverService.TestSolverService) ... ok testSolveUsingODR_3D (Test_SolverService.TestSolverService) ... ok testSolveUsingSimplex_3D (Test_SolverService.TestSolverService) ... ok testSolveUsingSimplex_SSQABS_2D (Test_SolverService.TestSolverService) ... ok testSolveUsingSimplex_SSQREL_2D (Test_SolverService.TestSolverService) ... ok testSolveUsingSpline_2D (Test_SolverService.TestSolverService) ... ok testSolveUsingSpline_3D (Test_SolverService.TestSolverService) ... ok ---------------------------------------------------------------------- Ran 76 tests in 94.615s OK

1 2

fftconvolve speedup / non powers of two
by Nicolas Rougier March 12, 2012

March 12, 2012

Hi, FFTW documentation (http://www.fftw.org/fftw2_doc/fftw_3.html) states that: "FFTW is best at handling sizes of the form 2^a*3^b*5^c*7^d*11^e*13^f, where e+f is either 0 or 1." But the code from fftconvolve is only using (internally) sizes that are power of two, is there a specific reason ? I tried a naive implementation handling size of the above form and it seems to speedup things a bit: In [1]: from fftconvolve import * In [2]: %timeit fftconvolve(Z,K,'full') 10 loops, best of 3: 57.3 ms per loop In [3]: %timeit fftconvolve2(Z,K,'full') 100 loops, best of 3: 13.2 ms per loop This is for the worst case where the internal size is 257. fftconvolve uses a size of 512 while fftconvolve2 uses 260. For powers of two, it should not change performances (only the time to compute best fft shape that may be probably improved). Nicolas Here is the code:

2 2

Documentation issue
by Robert Milgate Feb. 29, 2012

Feb. 29, 2012

Forgive me if this isn't the right path for this comment, I 'm quite new at this. I believe that the documentation for the hamming window implementation ( http://docs.scipy.org/doc/numpy/reference/generated/numpy.hamming.html ) in numpy is incorrect. Someone with more experience in signal conditioning than I should have a look, but I think this is wrong. The function works correctly, but I believe the equation in the documentation is for an interval symmetric about 0, not for the unipolar interval stated in the documentation. The + sign in front of the 0.46cos should be a - for teh interval as described.

2 1

scipy.optimize.fmin_l_bfgs_b versus openopt scipy_lbfgsb
by Nils Wagner Feb. 29, 2012

Feb. 29, 2012

Hi all, who can shed some light on the different behavior of scipy.optimize and openopt wrt. l_bfgs_b ? Nils >>> scipy.__version__ '0.11.0.dev-491f9db' >>> openopt.__version__ '0.37' python -i test_lbfgsb.py Optimal solution by L_BFGS_B (array([ 2000. , 1100. , 1614.65585984, 1050. , 900. , 1300. , 1100. , 1200. , 1400. , 1000. , 28. , 28. , 73. , 54. , 30. ]), 22.685511313514301, {'warnflag': 0, 'task': 'CONVERGENCE: NORM OF PROJECTED GRADIENT <= PGTOL', 'grad': array([ 8.38440428e-04, 9.89075488e-04, 5.32907052e-06, 3.17612603e-04, 4.14956958e-04, 2.43041143e-03, 1.26121336e-03, 1.87938554e-04, 2.43041143e-03, 1.10297549e-02, -2.24627428e-02, -2.86387802e-02, -5.51217738e-02, -1.89885441e-02, -5.40191181e-01]), 'funcalls': 36}) ------------------------- OpenOpt 0.37 ------------------------- solver: scipy_lbfgsb problem: unnamed type: NLP goal: minimum iter objFunVal 0 2.355e+01 RUNNING THE L-BFGS-B CODE * * * Machine precision = 2.220D-16 N = 15 M = 10 At X0 0 variables are exactly at the bounds Traceback (most recent call last): File "test_lbfgsb.py", line 68, in <module> r = p.solve('scipy_lbfgsb') File "/home/nwagner/local/lib/python2.7/site-packages/openopt-0.37-py2.7.egg/openopt/kernel/baseProblem.py", line 235, in solve return runProbSolver(self, *args, **kwargs) File "/home/nwagner/local/lib/python2.7/site-packages/openopt-0.37-py2.7.egg/openopt/kernel/runProbSolver.py", line 246, in runProbSolver solver(p) File "/home/nwagner/local/lib/python2.7/site-packages/openopt-0.37-py2.7.egg/openopt/solvers/scipy_optim/scipy_lbfgsb_oo.py", line 36, in __solver__ iprint=p.iprint, maxfun=p.maxFunEvals) File "/home/nwagner/local/lib64/python2.7/site-packages/scipy/optimize/lbfgsb.py", line 157, in fmin_l_bfgs_b options=opts, full_output=True) File "/home/nwagner/local/lib64/python2.7/site-packages/scipy/optimize/lbfgsb.py", line 270, in _minimize_lbfgsb isave, dsave) ValueError: failed to initialize intent(inout) array -- input not fortran contiguous

1 0

Possible bug in SLSQP
by Nils Wagner Feb. 28, 2012

Feb. 28, 2012

Hi all, I tried slsqp to solve an optimization problem. Traceback (most recent call last): File "test_slsqp.py", line 63, in <module> x_opt=fmin_slsqp(func2,x0,bounds=bounds) File "/home/nwagner/local/lib64/python2.7/site-packages/scipy/optimize/slsqp.py", line 184, in fmin_slsqp full_output=full_output) File "/home/nwagner/local/lib64/python2.7/site-packages/scipy/optimize/slsqp.py", line 309, in _minimize_slsqp raise IndexError('SLSQP Error: If bounds is specified, ' IndexError: SLSQP Error: If bounds is specified, bounds.shape[1] == len(x0) How should I define the box constraints for slsqp ? It seems to be not unique for different optimizers in scipy. Nils

2 3

ANN: SciPy 0.10.1 released
by Ralf Gommers Feb. 28, 2012

Feb. 28, 2012

Hi all, I am pleased to announce the availability of SciPy 0.10.1. This is a maintenance release, with no new features compared to 0.10.0. Sources and binaries can be found at http://sourceforge.net/projects/scipy/files/scipy/0.10.1/, release notes are copied below. Enjoy, The SciPy developers ========================== SciPy 0.10.1 Release Notes ========================== .. contents:: SciPy 0.10.1 is a bug-fix release with no new features compared to 0.10.0. Main changes ------------ The most important changes are:: 1. The single precision routines of ``eigs`` and ``eigsh`` in ``scipy.sparse.linalg`` have been disabled (they internally use double precision now). 2. A compatibility issue related to changes in NumPy macros has been fixed, in order to make scipy 0.10.1 compile with the upcoming numpy 1.7.0 release. Other issues fixed ------------------ - #835: stats: nan propagation in stats.distributions - #1202: io: netcdf segfault - #1531: optimize: make curve_fit work with method as callable. - #1560: linalg: fixed mistake in eig_banded documentation. - #1565: ndimage: bug in ndimage.variance - #1457: ndimage: standard_deviation does not work with sequence of indexes - #1562: cluster: segfault in linkage function - #1568: stats: One-sided fisher_exact() returns `p` < 1 for 0 successful attempts - #1575: stats: zscore and zmap handle the axis keyword incorrectly

1 0