[SciPy-User] beginner's question regarding optimize.fmin_l_bfgs_b

[Tveraa, Torkild]

-----Original Message-----
From: scipy-user-bounces at scipy.org [mailto:scipy-user-bounces at scipy.org] On Behalf Of Skipper Seabold
Sent: 14. oktober 2010 23:33
To: SciPy Users List
Subject: Re: [SciPy-User] beginner's question regarding optimize.fmin_l_bfgs_b

On Thu, Oct 14, 2010 at 5:21 PM, Skipper Seabold <jsseabold at gmail.com> wrote:
> On Tue, Oct 12, 2010 at 9:10 AM, Tveraa, Torkild <Torkild.Tveraa at nina.no> wrote:
>> Dear All,
>>
>> I have been able to use the optimize.leastsq - module to minimize a given function (see below), but since my data is sparse I have convergence problems and would ideally be able to put bounds on the parameters. If I have understood this correctly this can be done with the optimize.fmin_l_bfgs_b - module, but I am unable to figure out how to do this. Some helps & hints would be most appreciated :-)
>>
>>        Cheers,
>>        Torkild
>>
>> -------------------------------------------------------
>> import numpy
>> import pylab
>> from scipy import *
>> from scipy import optimize
>>
>> ## This is y-data:
>> y_data = (([0.2867, 0.1171, -0.0087, 0.1326, 0.2415, 0.2878, 0.3133, 0.3701, 0.3996, 0.3728, 0.3551, 0.3587, 0.1408, 0.0416, 0.0708, 0.1142, 0, 0, 0]))
>>
>> ## This is x-data:
>> t = (([67, 88, 104, 127, 138, 160, 169, 188, 196, 215, 240, 247, 271, 278, 303, 305, 321, 337, 353]))
>>
>> ## This is the equation:
>> fitfunc = lambda p, x:    p[0] + (p[1] -p[0]) * ((1/(1+exp(-p[2]*(t-p[3])))) + (1/(1+exp(p[4]*(t-p[5])))) -1)
>>
>> ##
>> errfunc = lambda p, x, y: fitfunc(p,x) -y
>>
>> guess = [0, max(y_data), 0.1, 140, -0.1, 270]
>>
>> bounds = [(-0.2, 0.1),(0.1,0.97), (0.05,0.8), (120,190), (-0.8, -0.05), (200,300) ]
>>
>> ## This seems to work ok:
>> p2,success = optimize.leastsq(errfunc, guess, args=(t, y_data),full_output=0)
>> print 'Estimates from leastsq \n', p2,success
>>
>>
>> ## But this does not:
>> best, val, d = optimize.fmin_l_bfgs_b(errfunc, guess, bounds=bounds, args=(t, y_data), iprint=2)
>
> The minimization routines, I believe, in fmin expect a function that
> maps from to a scalar.  So you need to tell fmin_l_bfgs that you want
> to minimize the sum of squared errors, optimze.leastsq assumes this.
> So just define one more function that sums the squared errors and
> minimize it
>
> errfuncsumsq = lambda p, x, y: np.sum(errfunc(p,x,y)**2)
>
> Now, run it without bounds to make sure we get the same thing
>
> boundsnone = [(None,None)]*6
>
> Notice that you also have to tell fmin_l_bfgs_b to approximate the
> gradient or else it assumes that your objective function also returns
> its gradient
>
> best, val, d = optimize.fmin_l_bfgs_b(errfuncsum, guess,
> approx_grad=True, bounds=boundsnone, args=(t, y_data), iprint=2)
>
> p2
> array([  6.79548883e-02,   3.68922503e-01,   7.55565728e-02,
>         1.41378227e+02,   2.91307814e+00,   2.70608242e+02])
>
> best
> array([  6.79585333e-02,  -2.33026316e-01,  -7.55409880e-02,
>         1.41388265e+02,  -1.36069434e+00,   2.70160779e+02])
>

I just realized that these don't come up with the same thing.  I don't
have an answer for why yet.

Skipper
_______________________________________________
SciPy-User mailing list
SciPy-User at scipy.org
http://mail.scipy.org/mailman/listinfo/scipy-user

Thanks a lot for your help and insight Skipper. 

Could it be that the difference between the two simply arise because it is a bit ambiguous to estimate 6 parameters based on just 21 observations? At least that is my gut feeling :-) The data are a bit noisy (see attached figure) so the results might vary accordingly?

Thanks,
Torkild

Estimates from leastsq:
[  6.79548906e-02   3.68922902e-01   7.55558888e-02   1.41378372e+02
   2.25898321e+00   2.70494848e+02]

Estimates from fmin_l_bfgs_b (without bounds):
[ -2.24017466e-01   1.51747173e+00   1.62043085e-02   1.20009182e+02
   1.27344545e-02   2.10006033e+02]

Estimates from fmin_l_bfgs_b (with bounds):
(if you try this please note that bounds for p[4] should be positive)
 [  7.56582544e-02   3.72058134e-01   7.57033694e-02   1.42823970e+02
   7.99990970e-01   2.50821346e+02]

-------------- next part --------------
A non-text attachment was scrubbed...
Name: plot_of_data.png
Type: image/png
Size: 15343 bytes
Desc: plot_of_data.png
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20101014/7dad8e48/attachment.png>