Re: [SciPy-Dev] Return residual and derivative in gradient descent methods like Newton

June 25, 2018

      Hi Josef,

Sorry, I didn't phrase my question very well.

I meant from the callback function, not the output from the solver.

For example, say we want to solve:

y = exp(Ax+y)

dy/dx = A*y

My proposal is when supplying arguments to the solver instead of writing
two callbacks we just supply one:
...
...
...
def f(x, y, A):
...     # callback returns both fval and jac
...     F = y - exp(Ax+y)  # residual
...     J = -A*y  # jacobian
...     return F, J
...
...
...
newton(f, x0, fprime='anystring', args=(y0, A))
'anything' indicated to the solver that `fprime` is returned by the
callback `f`. So inside the solver instead of calling them separately it
just makes one call:

...     try:
...         J = fprime(x, *args)
...     except Exception:
...         F, J = f(x, *args)
...     else:
...         F = f(x, *args)

This is a common API in MATLAB, and my tests show that for some functions
with this can reduce the number of calls and improve the speed of the
solver.

Maybe I could get the same improvements by precalculating some things and
passing them in `args`?

On Mon, Jun 25, 2018, 7:58 AM <josef.pktd@gmail.com> wrote:
...
...
Hi,
Would anyone disagree or would anyone be interested in a proposal to
allow
the derivative to be returned from the user supplied function as an
...
second argument in gradient search method like Newton? EG
...
...
...
lambda x,a: (x**3-a, 3*x*"2)
newton(f, x0, fprime='f2', args=(a,))
Some simple tests show that this may have a 2X speed in cases where the
derivative expression requires the value of the original function call.
See this issue in which I proposed this idea and wore a sample test
On Mon, Jun 25, 2018 at 10:30 AM, Mark Alexander Mikofski
<mikofski@berkeley.edu> wrote:
optional
script
...
https://github.com/scipy/scipy/issues/8354
I proposed a way to keep the existing API and add this new feature.
* If `fprime` is a callable, then same as before
* If it's a string like "f2" the get fprime from second output
I think using something like a `full_output` option as in the fmin_xxx
function would be more explicit and more flexible.
e.g. the second optional return could be a dict or similar.
Josef
...
Thanks,
Mark
_______________________________________________
SciPy-Dev mailing list
SciPy-Dev@python.org
https://mail.python.org/mailman/listinfo/scipy-dev
_______________________________________________
SciPy-Dev mailing list
SciPy-Dev@python.org
https://mail.python.org/mailman/listinfo/scipy-dev