[SciPy-Dev] feedback on adding global optimiser methods to `minimize`

[Matt Newville]

On Sat, Sep 7, 2019 at 10:57 PM Andrew Nelson <andyfaff at gmail.com> wrote:

> I'd like to gauge the support for adding the global minimizers
> (dual_annealing, shgo, differential_evolution, basinhopping) as new
> `minimize` methods.
>
> The problems the the 'local' and 'global' optimizers are trying to solve
> are very similar and both are specified in the same way, so my thought is
> that it would be nice to access them via the same unified interface that
> `minimize` offers (but not deprecating the `shgo` function, etc).
>
> It's important that the users are able to understand the distinction
> between local and global optimisation and how they go about finding a
> minimum. I'm hoping that this could be made plain in the documentation.
>
> The change would allow the following:
>
> ```
> # global minimizer
> minimize(func, x0, bounds=bounds, method='differential-evolution',
> constraints=constraints)
> # local minimizers
> minimize(func, x0,  bounds=bounds, method='SLSQP', constraints=constraints)
> minimize(func, x0, bounds=bounds, method='trust-constr',
> constraints=constraints)
> minimize(func, x0, bounds=bounds, method='L-BFGS-B')
> ```
>
> Please chip in with what your thoughts are, is it a bad idea, good idea,
> etc.
>
> --
>


Personally, I find `minimize()` to be a bit unwieldy  I do understand the
desire to make things simple and uniform, but I'm not sure `minimize()` is
really doing that. Correct me if any of this is wrong, but the interface
for the current `minimize()` has the following features:

  1. `minimize` takes 2 required arguments:  func, x0.  So far, so good.
  2. There are 14 named "methods" and a "custom object" method (which,
arguably means that the "global" methods are already supported).
  3. There are 3 optional arguments used for all methods with sensible
defaults: "args" , "tol" , and "callback".
  4. The "method" argument is actually optional with the default used being
a slightly complicated combination of other optional arguments.
  5. There are 6 optional arguments that each depends on which "method" is
used.
  6. Most but not all of the 14 methods support "jac". 4 of 6 options
("hess", "hessp", "bounds", "constraints") are supported by fewer than half
the methods.
  7. One of the 6 optional arguments is called "options" and holds keyword
options specific to each method and not explicitly listed.
  8. Some of the options in "options" (say, "eps", "ftol") are supported by
more methods than some of the explicitly named optional arguments (say,
"constraints").
  9. "constraints" is supported for 3 methods. It has 2 incompatible forms.
  10. "bounds" can either be a tuple of (called "min", "max" in the docs)
values or a "Bounds" object that has attributes "lb" and "ub".  Are "lower"
and "upper" are too verbose?
  11. As if to stick a finger in your eye, "callback" requires a different
signature for exactly one of the 14 methods.

It is all very well documented. But it is also extremely complicated.
 And, honestly, not very good.

It seems to me that there are really 14 (or 15) different minimization
methods.  The `minimize()` function smashes these all into one function
with a jumble of options.  I think the idea was to emphasize the
commonality of the interfaces, but the result sort of emphasizes what is
not common among the methods.  Instead of creating a common way to say
"Bounds", "Constraints", "Hessian", "Tolerance", "Max Number of Function
Calls", or "Step Size" for all methods, it allows multiple variations and
then works extra hard to document the differences.

How does this help the user (or person writing a wrapping library)?  Can
the user just change the name of the method argument?  In a simple case
that might work. But, in general, No, definitely not. Too many of the
arguments change validity when the `method` is changed.  To the extent that
the user could change "minimize(...., method='foo', ...)" to
"minimize(...., method='bar', ...)", they could just as easily change
"_minimize_foo(....)" to "_minimize_bar(...)".   That actually is less
typing.

I guess in a fatalistic sense, I would say "Sure, why not add more
methods?  You aren't going to make it any worse".  Except that you will
want to add more options that are not common to the other methods.
Differential evolution itself has a dozen or so "strategy" options
("strategy" being obviously different from "method", I guess).  There are
"seeds" and "nworkers"  and "max number of iterations" (whatever
"iteration" means for each method).  I do not know if any of these have
uniform meaning; I guess not.  It looks to me like "callback" would have
yet another variation.  And, eventually, you will want to deprecate the
current working functions, breaking downstream code for the sake of a
uniform API that really is not uniform.

The minimization methods really are different.  They have different
options.  IMO, it would be better to have one function per method and avoid
the a sprawling mess of a dispatch function. It would be nice if the
interfaces were as similar as possible, with similar concepts using
consistent names and values.

I would definitely encourage a class-based Minimizer, but mostly in the
sense of being able to use inheritance to make it easier to have a common
interface across the different solvers:  so not one Minimizer, but a
BaseMinizer and then PowellMinizer, BFGSMinimizer, KrylovMinimizer, etc.  I
would suggest looking at the stats module and scikit-learn for
inspiration.   And *then* make powell_minimize(), etc functions that create
and use these.  Oh, and a Parameters class would be nice ;).

I have no expectation of being listened too. We'll all be able to work
around the added complexity, just like we can sort of handle the similarly
infuriating complexity of `least_squares` (by basically treating it as 3
separate functions).

Finally, I will say that I am very glad to have `numpy.sin(x)`,
`numpy.cos(x)`, etc instead of `numpy.ufunc(x, method='sin')`.

Sorry if that seems like a rant.
Cheers,

--Matt
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20190909/0a157f0c/attachment-0001.html>