Hello, I have been using the approx_fprime and I noticed that the source code as it is only uses the one sided numerical differentiation. I had to build my own two sided one by copying and modifying the one sided code. One sided: F'(x) = ( F(x+h) - F(x) ) / h aka Newton method Two sided: F'(x) = ( F(x+h) - F(x-h) ) / (2*h) aka Symmetric method The two sided method tends to be more accurate, but can be more computationally expensive. Is this something that would be good to add? I am newer to python and I haven't contributed to open source yet, but it seems like a good fit. I think it would be good to pass in a kwarg called method and set the default to one sided, so no one's code breaks, but they can choose which method to use. Should I keep the decision logic at approx_fprime and have two separate functions for the two methods? Here's a wikipedia article on it <https://en.wikipedia.org/wiki/Numerical_differentiation> Here's a link to the source code <https://github.com/scipy/scipy/blob/v0.19.1/scipy/optimize/optimize.py#L633-...> Cheers, Jeff
Hm, tell me if I'm wrong, but in that case you can choose an arbitrary small epsilon. So accuracy should not be a problem (except in case of rounding error). By curiosity, what is your application and why do you think that one sided isn't enough? And if accuracy loss is really a problem, have you checked automatic differenciation? Nicolas C. Le 24 sept. 2017, à 05:09, Jeffrey Manville <jeffmanvillejr@gmail.com> a écrit: Hello, I have been using the approx_fprime and I noticed that the source code as it is only uses the one sided numerical differentiation. I had to build my own two sided one by copying and modifying the one sided code. One sided: F'(x) = ( F(x+h) - F(x) ) / h aka Newton methodTwo sided: F'(x) = ( F(x+h) - F(x-h) ) / (2*h) aka Symmetric method The two sided method tends to be more accurate, but can be more computationally expensive. Is this something that would be good to add? I am newer to python and I haven't contributed to open source yet, but it seems like a good fit. I think it would be good to pass in a kwarg called method and set the default to one sided, so no one's code breaks, but they can choose which method to use. Should I keep the decision logic at approx_fprime and have two separate functions for the two methods? Here's a wikipedia article on it Here's a link to the source code Cheers, Jeff SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev Le 24 sept. 2017 05:09, à 05:09, Jeffrey Manville <jeffmanvillejr@gmail.com> a écrit:
Hello,
I have been using the approx_fprime and I noticed that the source code as it is only uses the one sided numerical differentiation. I had to build my own two sided one by copying and modifying the one sided code.
One sided: F'(x) = ( F(x+h) - F(x) ) / h aka Newton method Two sided: F'(x) = ( F(x+h) - F(x-h) ) / (2*h) aka Symmetric method
The two sided method tends to be more accurate, but can be more computationally expensive.
Is this something that would be good to add? I am newer to python and I haven't contributed to open source yet, but it seems like a good fit.
I think it would be good to pass in a kwarg called method and set the default to one sided, so no one's code breaks, but they can choose which method to use.
Should I keep the decision logic at approx_fprime and have two separate functions for the two methods?
Here's a wikipedia article on it <https://en.wikipedia.org/wiki/Numerical_differentiation> Here's a link to the source code <https://github.com/scipy/scipy/blob/v0.19.1/scipy/optimize/optimize.py#L633-...>
Cheers,
Jeff
------------------------------------------------------------------------
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev
In my opinion, this would be a useful addition. The computational cost looks essentially the same because the number of evaluations of F is the same. The Wikipedia article that you cite also provides a formula for the so-called five-point method. This method is more computationally expensive, but converges very rapidly, so it might be good to include this option as well. Phillip On Sat, Sep 23, 2017 at 8:05 PM, Jeffrey Manville <jeffmanvillejr@gmail.com> wrote:
Hello,
I have been using the approx_fprime and I noticed that the source code as it is only uses the one sided numerical differentiation. I had to build my own two sided one by copying and modifying the one sided code.
One sided: F'(x) = ( F(x+h) - F(x) ) / h aka Newton method Two sided: F'(x) = ( F(x+h) - F(x-h) ) / (2*h) aka Symmetric method
The two sided method tends to be more accurate, but can be more computationally expensive.
Is this something that would be good to add? I am newer to python and I haven't contributed to open source yet, but it seems like a good fit.
I think it would be good to pass in a kwarg called method and set the default to one sided, so no one's code breaks, but they can choose which method to use.
Should I keep the decision logic at approx_fprime and have two separate functions for the two methods?
Here's a wikipedia article on it <https://en.wikipedia.org/wiki/Numerical_differentiation> Here's a link to the source code <https://github.com/scipy/scipy/blob/v0.19.1/scipy/optimize/optimize.py#L633-...>
Cheers,
Jeff
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev
la, 2017-09-23 kello 23:05 -0400, Jeffrey Manville kirjoitti:
I have been using the approx_fprime and I noticed that the source code as it is only uses the one sided numerical differentiation. I had to build my own two sided one by copying and modifying the one sided code.
The purpose of approx_fprime was mainly to serve as an internal differentiation tool for scipy.optimize when the user doesn't provide gradients explicitly. That it is a public method in scipy.optimize is sort of a historical artifact. I do not think it is an useful starting point for building a more general or more powerful numerical differentiation library, which then also raises the question that maybe it would be better to just deprecate/soft-deprecate it rather than trying to extend it. scipy.optimize currently has a second, more advanced numerical differentiator that it uses internally (see _numdiff.py), which also does multipoint schemes. There has been previous interest to add a module for numerical differentiation to scipy. There however are other existing modules for this on PyPi (e.g. numdifftools --- in addition to non-numerics modules for AD). So before starting, it would be a good idea to think about the scope of such a module, what functinality it should have etc. Some work towards this was done this summer but afaik it stalled, I don't recall what the status of intermediate results are. -- Pauli Virtanen
On Sun, Sep 24, 2017 at 3:52 PM, Pauli Virtanen <pav@iki.fi> wrote:
scipy.optimize currently has a second, more advanced numerical differentiator that it uses internally (see _numdiff.py), which also does multipoint schemes.
For people looking for something simple to use in code right now: ``` from scipy.optimize._numdiff import approx_derivative ``` -Robert
su, 2017-09-24 kello 16:15 -0400, Robert T. McGibbon kirjoitti:
On Sun, Sep 24, 2017 at 3:52 PM, Pauli Virtanen <pav@iki.fi> wrote:
scipy.optimize currently has a second, more advanced numerical differentiator that it uses internally (see _numdiff.py), which also does multipoint schemes.
For people looking for something simple to use in code right now:
``` from scipy.optimize._numdiff import approx_derivative ```
Please don't: it's a non-public method (see the leading underscore in the module name), and it may be removed in any release, and then your code is broken. If you want to use it in your own project, you can copy the `_numdiff.py` file (and what it needs, probably just _group_columns.pyx), maybe slapping the BSD license blurb from LICENSE.txt on top. Pauli
On Sun, Sep 24, 2017 at 3:52 PM, Pauli Virtanen <pav@iki.fi> wrote:
la, 2017-09-23 kello 23:05 -0400, Jeffrey Manville kirjoitti:
I have been using the approx_fprime and I noticed that the source code as it is only uses the one sided numerical differentiation. I had to build my own two sided one by copying and modifying the one sided code.
The purpose of approx_fprime was mainly to serve as an internal differentiation tool for scipy.optimize when the user doesn't provide gradients explicitly. That it is a public method in scipy.optimize is sort of a historical artifact.
I do not think it is an useful starting point for building a more general or more powerful numerical differentiation library, which then also raises the question that maybe it would be better to just deprecate/soft-deprecate it rather than trying to extend it.
scipy.optimize currently has a second, more advanced numerical differentiator that it uses internally (see _numdiff.py), which also does multipoint schemes.
There has been previous interest to add a module for numerical differentiation to scipy. There however are other existing modules for this on PyPi (e.g. numdifftools --- in addition to non-numerics modules for AD). So before starting, it would be a good idea to think about the scope of such a module, what functinality it should have etc. Some work towards this was done this summer but afaik it stalled, I don't recall what the status of intermediate results are.
Don't forget that there is also `scipy.misc.derivative` ( https://docs.scipy.org/doc/scipy/reference/generated/scipy.misc.derivative.h...), which uses central differences. See https://github.com/scipy/scipy/issues/7457 and the links therein for more background on the ongoing interest in improving scipy's tools for numerical differentiation. Warren --
Pauli Virtanen _______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev
On Mon, Sep 25, 2017 at 8:52 AM, Pauli Virtanen <pav@iki.fi> wrote:
la, 2017-09-23 kello 23:05 -0400, Jeffrey Manville kirjoitti:
I have been using the approx_fprime and I noticed that the source code as it is only uses the one sided numerical differentiation. I had to build my own two sided one by copying and modifying the one sided code.
The purpose of approx_fprime was mainly to serve as an internal differentiation tool for scipy.optimize when the user doesn't provide gradients explicitly. That it is a public method in scipy.optimize is sort of a historical artifact.
I do not think it is an useful starting point for building a more general or more powerful numerical differentiation library, which then also raises the question that maybe it would be better to just deprecate/soft-deprecate it rather than trying to extend it.
scipy.optimize currently has a second, more advanced numerical differentiator that it uses internally (see _numdiff.py), which also does multipoint schemes.
There has been previous interest to add a module for numerical differentiation to scipy. There however are other existing modules for this on PyPi (e.g. numdifftools --- in addition to non-numerics modules for AD). So before starting, it would be a good idea to think about the scope of such a module, what functinality it should have etc. Some work towards this was done this summer but afaik it stalled, I don't recall what the status of intermediate results are.
Unfortunately the results of that one month of GSoC work are not useable. So there isn't much code to build on. Also, numdifftools development pace has increased since the ~2014 days (IIRC) when we first considered adding it to scipy. And we have things like autograd coming along that understand pretty much all of numpy and even some things in scipy. So after a couple of failed attempts at creating a scipy.diff, I'm no longer convinced that it's worth spending a lot of effort on. Ralf
On Mon, Sep 25, 2017 at 10:55 AM Ralf Gommers <ralf.gommers@gmail.com> wrote:
Unfortunately the results of that one month of GSoC work are not useable.
So there isn't much code to build on. Also, numdifftools development pace has increased since the ~2014 days (IIRC) when we first considered adding it to scipy. And we have things like autograd coming along that understand pretty much all of numpy and even some things in scipy. So after a couple of failed attempts at creating a scipy.diff, I'm no longer convinced that it's worth spending a lot of effort on.
Just to further muddy the waters, I felt the need for a better numerical differentiator so I put together an adaptive one that is able to use parallelism: https://github.com/aarchiba/futuretools This allows otherwise non-parallel optimization algorithms to experience a speedup equal to roughly twice the dimensionality of the problem. It hasn't seen heavy use, though, so don't use it without careful checking. numdifftools is aimed at high accuracy and fails to exploit parallelism; I don't think it's suitable for use within an optimization algorithm, but if you need derivatives for, say, error analysis it may be worth using. Numerical differentiation is hard, and there are a number of approaches to obtaining it with quite complicated cost-benefit analyses. I don't like that the scipy default is terrible, but a minor improvement that doubles the computational cost (f(x) is almost always needed anyway) is probably not that helpful. More useful would be if scipy had a good derivative-free optimization algorithm, but the one I know of, BOBYQA, has licensing complexities and anyways is available in the python package nlopt. (scipy's powell is really terrible, at least at making progress on my problem.) Anne
On Wed, Sep 27, 2017 at 1:41 AM, Anne Archibald <peridot.faceted@gmail.com> wrote:
On Mon, Sep 25, 2017 at 10:55 AM Ralf Gommers <ralf.gommers@gmail.com> wrote:
Unfortunately the results of that one month of GSoC work are not useable.
So there isn't much code to build on. Also, numdifftools development pace has increased since the ~2014 days (IIRC) when we first considered adding it to scipy. And we have things like autograd coming along that understand pretty much all of numpy and even some things in scipy. So after a couple of failed attempts at creating a scipy.diff, I'm no longer convinced that it's worth spending a lot of effort on.
Just to further muddy the waters, I felt the need for a better numerical differentiator so I put together an adaptive one that is able to use parallelism: https://github.com/aarchiba/futuretools This allows otherwise non-parallel optimization algorithms to experience a speedup equal to roughly twice the dimensionality of the problem. It hasn't seen heavy use, though, so don't use it without careful checking.
Interesting, thanks for sharing. If you plan to keep developing it, maybe good to add to https://scipy.org/topical-software.html ? Ralf
numdifftools is aimed at high accuracy and fails to exploit parallelism; I don't think it's suitable for use within an optimization algorithm, but if you need derivatives for, say, error analysis it may be worth using.
Numerical differentiation is hard, and there are a number of approaches to obtaining it with quite complicated cost-benefit analyses. I don't like that the scipy default is terrible, but a minor improvement that doubles the computational cost (f(x) is almost always needed anyway) is probably not that helpful. More useful would be if scipy had a good derivative-free optimization algorithm, but the one I know of, BOBYQA, has licensing complexities and anyways is available in the python package nlopt. (scipy's powell is really terrible, at least at making progress on my problem.)
Anne
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev
participants (8)
-
Anne Archibald -
Jeffrey Manville -
Nicolas CELLIER -
Pauli Virtanen -
Phillip Feldman -
Ralf Gommers -
Robert T. McGibbon -
Warren Weckesser