[SciPy-Dev] RFC: non-linear least squares

[Evgeni Burovski]

Hi,

I'd like to draw attention to the great work Nikolay Mayorov is doing
for his GSoC on improving non-linear least squares functionality in
scipy.

So far he single-handedly implemented two trust-region type algorithms
for nonlinear LSQ: rectangular trust region dogleg ('dogbox') and
trust region reflective algorithm ('trf', this one is similar to what
MATLAB is using as its default NLSQ method).

Both of these algorithms natively handle box constraints on fit
parameters. Both are implemented to work with dense or sparse
Jacobians, so that they are suitable for large-scale problems (up to
and in excess of 10k variables).

One other feature of the implementation is modified loss functions for
robust LSQ. Several useful loss functions are available out of the
box, and there is flexibility for using custom user-supplied loss
functions.

Several examples are available in Nikolay's GSoC blog at
<https://nmayorov.wordpress.com/>, along with narrative discussion of
the mathematical content, and results of benchmarking his algorithms
against general-purpose optimizers (L-BFGS-B in particular) and
MINUIT-type constrained LSQ optimizer (specifically, leastsqbound ---
which is equivalent to LMFIT).

The relevant PR is under review at <https://github.com/scipy/scipy/pull/5044>.

We plan on merging this PR in some reasonably near future, and we'd
like to ask for help:
* If you are an expert on mathematical optimization, we'd be grateful
if you help reviewing the code;
* If you are a user of nonlinear least squares, we'd be grateful if
you test drive the new functionality and let us know how it works on
your problems.
* We'd welcome comments on the user interface.

Cheers,

Evgeni