[Automatic differentiation] is more efficient and widely used in machine learning.

Good to know. Another thing to add to the "Learn to Do" list. However, does automatic differentiation this negate the value of this? Scipy already has one of these methods and its the less accurate method. Is it inappropriate to have it in the library? I really don't know how this works. Are features decided by general consensus? Do I get drawn and quartered for bad ideas? On Sun, Sep 24, 2017 at 3:10 PM, <scipy-dev-request@python.org> wrote:

Send SciPy-Dev mailing list submissions to scipy-dev@python.org

To subscribe or unsubscribe via the World Wide Web, visit https://mail.python.org/mailman/listinfo/scipy-dev or, via email, send a message with subject or body 'help' to scipy-dev-request@python.org

You can reach the person managing the list at scipy-dev-owner@python.org

When replying, please edit your Subject line so it is more specific than "Re: Contents of SciPy-Dev digest..."

Today's Topics:

1. Re: Add feature to approx_fprime (Nicolas CELLIER) (Scott Sievert)

----------------------------------------------------------------------

Message: 1 Date: Sun, 24 Sep 2017 14:10:27 -0500 From: "Scott Sievert" <sievert.scott@gmail.com> To: "SciPy Developers List" <scipy-dev@python.org> Subject: Re: [SciPy-Dev] Add feature to approx_fprime (Nicolas CELLIER) Message-ID: <747CD49D-AF33-45E4-8A58-609FD464A5D0@gmail.com> Content-Type: text/plain; charset="us-ascii"; Format="flowed"

...

I suspect that it can be prohibitively expensive for many machine learning algorithms

This method of differentiation is expensive for machine learning (ML) applications. This methods grows like O(n) when the variable is n-dimensional.

[Automatic differentiation] is more efficient and widely used in machine learning. For example, [HIPS/autograd] is a thinly wrapped NumPy that can compute the gradient in approximately O(5k) when the function takes k seconds to compute. It does this by recording operations done on a variable then using the chain rule.

Various ML libraries and their automatic differentiation strategies can be found at https://docs.chainer.org/en/stable/comparison.html.

Scott

[HIPS/autograd]:https://github.com/HIPS/autograd [Automatic differentiation]:https://en.wikipedia.org/wiki/Automatic_differentiation

On 24 Sep 2017, at 13:34, Jeffrey Manville wrote:

...

Hello Nick,

So, I am implementing Andrew Ng's Machine learning coursework in Python which was originally in Matlab/Octave. I was using the approx_fprime to evaluate by implementation of a back propagation algorithm.

To be honest, I didn't know about automatic differentiation. I suspect that it can be prohibitively expensive for many machine learning algorithms, but probably not for my project. I could use a smaller epsilon, but then I am just getting closer and closer to the floating point error problem.

*My project:* In my project, the "difference" between backprop and the numerical gradient was expected to be <1e-9 with an epsilon of 1e-4. I was getting about 2.48e-5 with the current approx_fprime vs 1.65e-11 with the two sided method and the same epsilon. The two sided was significantly more accurate.

*The problem:* The problem with the one sided method is that it is inherently biased. The change isn't centered around the point in question for the one sided method.

The two methods are essentially the same method with the difference being "where" they guess that calculated derivative is. One sided guesses that the calculated derivative is at one side of the two points. The two sided guesses the derivative is in the middle.

*Proof:* I attached an excel doc that has a kind of example of the difference.

I have the project on github. Here is the branch <https://github.com/manvillej/Python-AndrewNgML/tree/Ex4NNtraining> that I am currently working on. If you pull it and run ex4checker.py you will see the two sided. If you change line 3: import checker as op to import checker as op you will see the relative difference change from the two sided to the one sided.

Cheers,

Jeff

On Sun, Sep 24, 2017 at 12:00 PM, <scipy-dev-request@python.org> wrote:

...

Send SciPy-Dev mailing list submissions to scipy-dev@python.org

To subscribe or unsubscribe via the World Wide Web, visit https://mail.python.org/mailman/listinfo/scipy-dev or, via email, send a message with subject or body 'help' to scipy-dev-request@python.org

You can reach the person managing the list at scipy-dev-owner@python.org

When replying, please edit your Subject line so it is more specific than "Re: Contents of SciPy-Dev digest..."

Today's Topics:

1. Add feature to approx_fprime (Jeffrey Manville) 2. Re: Add feature to approx_fprime (Nicolas CELLIER)

----------------------------------------------------------------------

Message: 1 Date: Sat, 23 Sep 2017 23:05:57 -0400 From: Jeffrey Manville <jeffmanvillejr@gmail.com> To: scipy-dev@python.org Subject: [SciPy-Dev] Add feature to approx_fprime Message-ID: <CAFUATRLXKY6y=5VEANjrd2ATtZ8GHKxWEeLmZ=Tk6RkfxABEvw@mail. gmail.com> Content-Type: text/plain; charset="utf-8"

Hello,

I have been using the approx_fprime and I noticed that the source code as it is only uses the one sided numerical differentiation. I had to build my own two sided one by copying and modifying the one sided code.

One sided: F'(x) = ( F(x+h) - F(x) ) / h aka Newton method Two sided: F'(x) = ( F(x+h) - F(x-h) ) / (2*h) aka Symmetric method

The two sided method tends to be more accurate, but can be more computationally expensive.

Is this something that would be good to add? I am newer to python and I haven't contributed to open source yet, but it seems like a good fit.

I think it would be good to pass in a kwarg called method and set the default to one sided, so no one's code breaks, but they can choose which method to use.

Should I keep the decision logic at approx_fprime and have two separate functions for the two methods?

Here's a wikipedia article on it <https://en.wikipedia.org/wiki/Numerical_differentiation> Here's a link to the source code <https://github.com/scipy/scipy/blob/v0.19.1/scipy/optimize/ optimize.py#L633-L688>

Cheers,

Jeff