I can use the MATLAB optimizers and solve_ivp as points of reference. When compared to MATLAB, the current interface stacks up pretty well. However, both MATLAB and `scipy.optimize` have a less-than-ideal interface for gathering optimization history.

Perhaps it would be more useful to align `scipy.optimize` with `solve_ivp`. In both cases you’re calculating a new point using the history of the previous points (though of course the goal—and the formulas—are different). The difference in user interface means that `scipy.optimize` only returns the last point, whereas `solve_ivp` returns the entire solution history. It’s obvious why: a solution to a set of differential equations mathematically must span the domain of interest whereas an optimization result mathematically only consists of a single point. But there no reason, at least that I can tell, not to return all the points that we tried while running the optimizer. It would at least provide a much simpler way to understand what the optimizer is doing.

From: Andrew Nelson
Sent: Tuesday, February 13, 2018 9:21 PM
To: scipy-dev
Subject: [SciPy-Dev] WIP: Class based Optimizers

The #8414 PR on github (https://github.com/scipy/scipy/pull/8414) introduces an `Optimize` class for `scipy.optimize`. The `Optimize` class can be advanced like an iterator, or be advanced to convergence with the `solve` method.

The PR is intended to provide a cleaner interface to the minimisation process, with the user being able to interact with the solver during iteration. This allows the variation of solver hyper-parameters, or other user definable interactions. As such `callback`s become redundant, as you can access all the optimizer information required. 

Other historical parameters may become redundant in these objects, such as `disp`. This is also a chance for a ground-zero approach to how people interact with optimizers.

The PR is intended as a Request For Comment to gain feedback on architecture of the solvers. Given that it's a key part of infrastructure this may create a large amount of feedback, if I don't reply to it all bear in mind that I don't have unlimited time resource.

In it's current state I've implemented `Optimizer`, `Function`, `LBFGS` and `NelderMead` classes. `LBFGS` and `NelderMead` are being called by the corresponding functions already in existence. The current test suite passes, but no unit tests have been written for the functionality. I'd prefer to add such a specific test suite once the overall architecture is settled.

These optimisers were straightforward to translate. However, I don't know how this approach would translate to other minimisers, or things like `least_squares`. If a design could be agreed, then perhaps the best approach would be to tackle the remaining optimizers on a case by case basis once this PR is merged.