Patching fmin_l_bfgs_b in scipy.optimize
Hi, I had some troubles getting a stable version of fmin_l_bfgs_b. I got a segmentation fault as soon as I turned a different value than -1 to iprint argument. As far as I understood, this iprint parameter is directly related to some fortran printing/writing logs. It prints iteration, norm of gradient on screen and also in a file called iterate.dat .. Even recompiling the whole stuff from lapack, atlas etc.. (using gfortran only) up to scipy doesn't solve the problem. Because I definitely gave up the idea to fix the bug by recompiling, I decided to patch the way python interacts with lbfgsb fortran procedure. So, I disabled the fortran iprint and add some iprint functionnality in pure python code directly in lbfgsb.py (scipy/optimize). Doing so, I get rid of the useless iterate.dat file too. While patching, I followed by adding some few things : - an optional stopping rule based on maximum number of iterations. I think this is more usefull than the number of function evaluations because the fortran code performs a linesearch procedure inside the lbfgsb ones. - a callback procedure which allows to inspect solution, criterion, and gradient at current iterate. Maybe my patch should be usefull for someone, so I decided to share it with the scipy community. Gilles. PS: I provide two files : lbfgsb.py -- the full python script to put in scipy/optimize scipy-optimize-lbfgsb-12162008.patch -- a patch file
Gilles Rochefort wrote:
Hi,
I had some troubles getting a stable version of fmin_l_bfgs_b. I got a segmentation fault as soon as I turned a different value than -1 to iprint argument.
As far as I understood, this iprint parameter is directly related to some fortran printing/writing logs. It prints iteration, norm of gradient on screen and also in a file called iterate.dat .. Even recompiling the whole stuff from lapack, atlas etc.. (using gfortran only) up to scipy doesn't solve the problem.
Because I definitely gave up the idea to fix the bug by recompiling, I decided to patch the way python interacts with lbfgsb fortran procedure. So, I disabled the fortran iprint and add some iprint functionnality in pure python code directly in lbfgsb.py (scipy/optimize). Doing so, I get rid of the useless iterate.dat file too.
While patching, I followed by adding some few things : - an optional stopping rule based on maximum number of iterations. I think this is more usefull than the number of function evaluations because the fortran code performs a linesearch procedure inside the lbfgsb ones. - a callback procedure which allows to inspect solution, criterion, and gradient at current iterate.
Maybe my patch should be usefull for someone, so I decided to share it with the scipy community.
Thanks for sharing. As a general rule, I like having callback functions inside of these types of procedures, if only for educational purposes (as long as the callback check is negligible if no callback is provided). Along these lines, I notice that several times you use "!=None" in an if statement. It is faster to use "is not None" versus "!= None". I think this has to do with there being only one None object, so a memory reference comparison (done with "is") is sufficient and faster. If a python wizard knows better, please correct me! In [1]: a=lambda x: x In [2]: %timeit a is not None 10000000 loops, best of 3: 113 ns per loop In [3]: %timeit a != None 10000000 loops, best of 3: 146 ns per loop In [4]: a=None In [5]: %timeit a is not None 10000000 loops, best of 3: 99.2 ns per loop In [6]: %timeit a != None 10000000 loops, best of 3: 147 ns per loop Thanks, Jason
Thanks for the advice, here is my new patch ! Regards, Gilles. jason-sage@creativetrax.com a écrit :
Gilles Rochefort wrote:
Hi,
I had some troubles getting a stable version of fmin_l_bfgs_b. I got a segmentation fault as soon as I turned a different value than -1 to iprint argument.
As far as I understood, this iprint parameter is directly related to some fortran printing/writing logs. It prints iteration, norm of gradient on screen and also in a file called iterate.dat .. Even recompiling the whole stuff from lapack, atlas etc.. (using gfortran only) up to scipy doesn't solve the problem.
Because I definitely gave up the idea to fix the bug by recompiling, I decided to patch the way python interacts with lbfgsb fortran procedure. So, I disabled the fortran iprint and add some iprint functionnality in pure python code directly in lbfgsb.py (scipy/optimize). Doing so, I get rid of the useless iterate.dat file too.
While patching, I followed by adding some few things : - an optional stopping rule based on maximum number of iterations. I think this is more usefull than the number of function evaluations because the fortran code performs a linesearch procedure inside the lbfgsb ones. - a callback procedure which allows to inspect solution, criterion, and gradient at current iterate.
Maybe my patch should be usefull for someone, so I decided to share it with the scipy community.
Thanks for sharing. As a general rule, I like having callback functions inside of these types of procedures, if only for educational purposes (as long as the callback check is negligible if no callback is provided). Along these lines, I notice that several times you use "!=None" in an if statement. It is faster to use "is not None" versus "!= None". I think this has to do with there being only one None object, so a memory reference comparison (done with "is") is sufficient and faster.
If a python wizard knows better, please correct me!
In [1]: a=lambda x: x
In [2]: %timeit a is not None 10000000 loops, best of 3: 113 ns per loop
In [3]: %timeit a != None 10000000 loops, best of 3: 146 ns per loop
In [4]: a=None
In [5]: %timeit a is not None 10000000 loops, best of 3: 99.2 ns per loop
In [6]: %timeit a != None 10000000 loops, best of 3: 147 ns per loop
Thanks,
Jason
_______________________________________________ Scipy-dev mailing list Scipy-dev@scipy.org http://projects.scipy.org/mailman/listinfo/scipy-dev
Hi Gilles If your patch hasn't yet been applied, please file a ticket on the developer page http://projects.scipy.org/scipy/scipy so that we don't forget to attend to this issue. Cheers Stéfan 2008/12/17 Gilles Rochefort <gilles.rochefort@gmail.com>:
Thanks for the advice, here is my new patch !
Regards, Gilles.
jason-sage@creativetrax.com a écrit :
Gilles Rochefort wrote:
Hi,
I had some troubles getting a stable version of fmin_l_bfgs_b. I got a segmentation fault as soon as I turned a different value than -1 to iprint argument.
First, I apology not to did it sooner, but as one said, it is never too late ;-) So, I finally issued a ticket with a patch file generated from current svn revision r5661 ... http://projects.scipy.org/scipy/ticket/923 Gilles.
Hi Gilles
If your patch hasn't yet been applied, please file a ticket on the developer page
http://projects.scipy.org/scipy/scipy
so that we don't forget to attend to this issue.
Cheers Stéfan
2008/12/17 Gilles Rochefort <gilles.rochefort@gmail.com>:
Thanks for the advice, here is my new patch !
Regards, Gilles.
jason-sage@creativetrax.com a écrit :
Gilles Rochefort wrote:
Hi,
I had some troubles getting a stable version of fmin_l_bfgs_b. I got a segmentation fault as soon as I turned a different value than -1 to iprint argument.
Index: scipy/optimize/lbfgsb.py =================================================================== --- scipy/optimize/lbfgsb.py (révision 5661) +++ scipy/optimize/lbfgsb.py (copie de travail) @@ -26,7 +26,19 @@ ## Modifications by Travis Oliphant and Enthought, Inc. for inclusion in SciPy -from numpy import zeros, float64, array, int32 +## Modifications by Gilles Rochefort include : +## +## - Removed internal logging made from fortran code (as it always seems to failed) and get +## bored with iterate.dat file. Logging informations are now be done with pure python code. +## +## - Adding a new stopping rule based on maximum iterations (maxiter) because maximum +## function evaluations don't always sound right ( just think that an internal linesearch +## procedure also contribute to increase the number of function evaluations ) +## +## - Adding a callback mecanism, which allows to monitor solution, criterion, and gradient +## at current iteration. + +from numpy import zeros, float64, array, int32, mod import _lbfgsb import optimize @@ -36,7 +48,7 @@ approx_grad=0, bounds=None, m=10, factr=1e7, pgtol=1e-5, epsilon=1e-8, - iprint=-1, maxfun=15000): + iprint=-1, maxfun=15000, maxiter=None, callback=None): """ Minimize a function func using the L-BFGS-B algorithm. @@ -82,8 +94,12 @@ iprint -- controls the frequency of output. <0 means no output. - maxfun -- maximum number of function evaluations. + maxfun -- Maximum number of function evaluations. + + maxiter -- Maximum number of iterations allowed. + callback -- function to inspect parameters iter,f,x,g at each iteration. + ex: def cb(iter,f,x,g): print "iter %d f = %f" % (iter,f) Returns: x, f, d = fmin_lbfgs_b(func, x0, ...) @@ -97,8 +113,8 @@ 2 if stopped for another reason, given in d['task'] d['grad'] is the gradient at the minimum (should be 0 ish) d['funcalls'] is the number of function calls made. + d['nbiter'] is the number of iterations made. - License of L-BFGS-B (Fortran code) ================================== @@ -190,12 +206,12 @@ dsave = zeros((29,), float64) task[:] = 'START' - + iter = 0 n_function_evals = 0 while 1: # x, f, g, wa, iwa, task, csave, lsave, isave, dsave = \ _lbfgsb.setulb(m, x, low_bnd, upper_bnd, nbd, f, g, factr, - pgtol, wa, iwa, task, iprint, csave, lsave, + pgtol, wa, iwa, task, -1, csave, lsave, isave, dsave) task_str = task.tostring() if task_str.startswith('FG'): @@ -205,6 +221,13 @@ f, g = func_and_grad(x) elif task_str.startswith('NEW_X'): # new iteration + iter += 1 + if maxiter is not None and iter > maxiter: + break + if callback is not None: + callback(iter,f,x,g) + if iprint > 0 and mod(iter,iprint) == 0: + print "[iter = %d] f = %f, ||grad f||^2 = %f" % ( iter,f,sum(g**2) ) if n_function_evals > maxfun: task[:] = 'STOP: TOTAL NO. of f AND g EVALUATIONS EXCEEDS LIMIT' else: @@ -222,7 +245,8 @@ d = {'grad' : g, 'task' : task_str, 'funcalls' : n_function_evals, - 'warnflag' : warnflag + 'warnflag' : warnflag, + 'nbiter' : iter } return x, f, d
participants (3)
-
Gilles Rochefort -
jason-sage＠creativetrax.com -
Stéfan van der Walt