Return residual and derivative in gradient descent methods like Newton
Hi, Would anyone disagree or would anyone be interested in a proposal to allow the derivative to be returned from the user supplied function as an optional second argument in gradient search method like Newton? EG
lambda x,a: (x**3-a, 3*x*"2) newton(f, x0, fprime='f2', args=(a,))
Some simple tests show that this may have a 2X speed in cases where the derivative expression requires the value of the original function call. See this issue in which I proposed this idea and wore a sample test script https://github.com/scipy/scipy/issues/8354 I proposed a way to keep the existing API and add this new feature. * If `fprime` is a callable, then same as before * If it's a string like "f2" the get fprime from second output Thanks, Mark
On Mon, Jun 25, 2018 at 10:30 AM, Mark Alexander Mikofski <mikofski@berkeley.edu> wrote:
Hi,
Would anyone disagree or would anyone be interested in a proposal to allow the derivative to be returned from the user supplied function as an optional second argument in gradient search method like Newton? EG
lambda x,a: (x**3-a, 3*x*"2) newton(f, x0, fprime='f2', args=(a,))
Some simple tests show that this may have a 2X speed in cases where the derivative expression requires the value of the original function call.
See this issue in which I proposed this idea and wore a sample test script https://github.com/scipy/scipy/issues/8354
I proposed a way to keep the existing API and add this new feature. * If `fprime` is a callable, then same as before * If it's a string like "f2" the get fprime from second output
I think using something like a `full_output` option as in the fmin_xxx function would be more explicit and more flexible. e.g. the second optional return could be a dict or similar. Josef
Thanks, Mark
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev
Hi Josef, Sorry, I didn't phrase my question very well. I meant from the callback function, not the output from the solver. For example, say we want to solve: y = exp(Ax+y) dy/dx = A*y My proposal is when supplying arguments to the solver instead of writing two callbacks we just supply one:
def f(x, y, A): ... # callback returns both fval and jac ... F = y - exp(Ax+y) # residual ... J = -A*y # jacobian ... return F, J
newton(f, x0, fprime='anystring', args=(y0, A))
'anything' indicated to the solver that `fprime` is returned by the callback `f`. So inside the solver instead of calling them separately it just makes one call: ... try: ... J = fprime(x, *args) ... except Exception: ... F, J = f(x, *args) ... else: ... F = f(x, *args) This is a common API in MATLAB, and my tests show that for some functions with this can reduce the number of calls and improve the speed of the solver. Maybe I could get the same improvements by precalculating some things and passing them in `args`? On Mon, Jun 25, 2018, 7:58 AM <josef.pktd@gmail.com> wrote:
Hi,
Would anyone disagree or would anyone be interested in a proposal to allow the derivative to be returned from the user supplied function as an
second argument in gradient search method like Newton? EG
lambda x,a: (x**3-a, 3*x*"2) newton(f, x0, fprime='f2', args=(a,))
Some simple tests show that this may have a 2X speed in cases where the derivative expression requires the value of the original function call.
See this issue in which I proposed this idea and wore a sample test
On Mon, Jun 25, 2018 at 10:30 AM, Mark Alexander Mikofski <mikofski@berkeley.edu> wrote: optional script
https://github.com/scipy/scipy/issues/8354
I proposed a way to keep the existing API and add this new feature. * If `fprime` is a callable, then same as before * If it's a string like "f2" the get fprime from second output
I think using something like a `full_output` option as in the fmin_xxx function would be more explicit and more flexible. e.g. the second optional return could be a dict or similar.
Josef
Thanks, Mark
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev
On Mon, Jun 25, 2018 at 10:30 AM, Mark Alexander Mikofski
<mikofski@berkeley.edu> wrote:
Would anyone disagree or would anyone be interested in a proposal to
the derivative to be returned from the user supplied function as an
allow optional
second argument in gradient search method like Newton? EG
lambda x,a: (x**3-a, 3*x*"2) newton(f, x0, fprime='f2', args=(a,))
Some simple tests show that this may have a 2X speed in cases where the derivative expression requires the value of the original function call.
The `minimize` function does something like this for the `jac` keyword: "If *jac* is a Boolean and is True, *fun* is assumed to return the gradient along with the objective function.". If something like this is desirable, then I suggest that the same minimize pattern is used.
It can indeed be desirable to return f and f’ (and f’’) in a single call, especially if the additional computation for "f’-while-computing-f" is very small. Not all of the derivative-based methods would necessarily benefit (e.g Homeier’s method which evaluates two f' and one f every iteration), but the base Newton’s method wouldn’t see any additional computation. There is an existing PR https://github.com/scipy/scipy/pull/8890 <https://github.com/scipy/scipy/pull/8890> to add `optimize.root_scalar(f, args=(), method=None, bracket=None, fprime=None, fprime2=None,x0=None, x1=None,xtol=None, rtol=None, maxiter=None,options=None)` This would be a 1-dim analog to match the multi-dimensional `optimize.root()`. The `fprime` argument is either a callable or a Boolean. If it is `True`, then calling `f(x, *args)` is expected to return both f and f’. Similarly if `fprime2` is True, then calling `f(x, *args)` is expected to return f, f’ and f’'. -Paul
On Jun 25, 2018, at 3:12 PM, Andrew Nelson <andyfaff@gmail.com> wrote:
On Mon, Jun 25, 2018 at 10:30 AM, Mark Alexander Mikofski <mikofski@berkeley.edu <mailto:mikofski@berkeley.edu>> wrote:
Would anyone disagree or would anyone be interested in a proposal to allow the derivative to be returned from the user supplied function as an optional second argument in gradient search method like Newton? EG
lambda x,a: (x**3-a, 3*x*"2) newton(f, x0, fprime='f2', args=(a,))
Some simple tests show that this may have a 2X speed in cases where the derivative expression requires the value of the original function call.
The `minimize` function does something like this for the `jac` keyword: "If jac is a Boolean and is True, fun is assumed to return the gradient along with the objective function.". If something like this is desirable, then I suggest that the same minimize pattern is used. _______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev
participants (4)
-
Andrew Nelson -
josef.pktd＠gmail.com -
Mark Alexander Mikofski -
Paul van Mulbregt