Large Scale Optimization algorithm implementation
Hi, So, the most efficient optimization algorithm implemented in Scipy is LBFGS-B in terms of memory consumption and runtime speed. It would be good to see algorithms suited for large scale optimization in scipy for the scenarios when Gradient computation is quite expensive and/or Number of variables is huge. SVRG and (SVRG+LBFGS) can both attain much better performance than LBFGS-B for this scenario. The links: (SVRG) : https://papers.nips.cc/paper/4937-accelerating-stochastic-gradient-descent-u... <https://ml-trckr.com/link/https%3A%2F%2Fpapers.nips.cc%2Fpaper%2F4937-accelerating-stochastic-gradient-descent-using-predictive-variance-reduction.pdf/dzjpYvcNc3EU6zyPRAsq> (SVRG + LBFGS) : http://opt-ml.org/papers/OPT2015_paper_41.pdf <https://ml-trckr.com/link/http%3A%2F%2Fopt-ml.org%2Fpapers%2FOPT2015_paper_41.pdf/dzjpYvcNc3EU6zyPRAsq> What do you think? Cheers, Touqir -- Computing Science Master's student at University of Alberta, Canada, specializing in Machine Learning. Website : https://ca.linkedin.com/in/touqir-sajed-6a95b1126 <https://ml-trckr.com/link/https%3A%2F%2Fca.linkedin.com%2Fin%2Ftouqir-sajed-6a95b1126/dzjpYvcNc3EU6zyPRAsq>
These algorithms look specialized to problems where the optimization objective can be written as an average over many examples. This is not necessarily the case for the functions optimized by scipy.optimize.minimize -- we don't have any equivalent of stochastic gradient descent. This might be more suitable for libraries that specialize in these sort of optimization problems, like TensorFlow or pytorch. On Fri, Sep 14, 2018 at 3:42 PM Touqir Sajed <touqir@ualberta.ca> wrote:
Hi,
So, the most efficient optimization algorithm implemented in Scipy is LBFGS-B in terms of memory consumption and runtime speed. It would be good to see algorithms suited for large scale optimization in scipy for the scenarios when Gradient computation is quite expensive and/or Number of variables is huge. SVRG and (SVRG+LBFGS) can both attain much better performance than LBFGS-B for this scenario. The links: (SVRG) : https://papers.nips.cc/paper/4937-accelerating-stochastic-gradient-descent-u... <https://ml-trckr.com/link/https%3A%2F%2Fpapers.nips.cc%2Fpaper%2F4937-accelerating-stochastic-gradient-descent-using-predictive-variance-reduction.pdf/dzjpYvcNc3EU6zyPRAsq> (SVRG + LBFGS) : http://opt-ml.org/papers/OPT2015_paper_41.pdf <https://ml-trckr.com/link/http%3A%2F%2Fopt-ml.org%2Fpapers%2FOPT2015_paper_41.pdf/dzjpYvcNc3EU6zyPRAsq>
What do you think?
Cheers, Touqir
-- Computing Science Master's student at University of Alberta, Canada, specializing in Machine Learning. Website : https://ca.linkedin.com/in/touqir-sajed-6a95b1126 <https://ml-trckr.com/link/https%3A%2F%2Fca.linkedin.com%2Fin%2Ftouqir-sajed-6a95b1126/dzjpYvcNc3EU6zyPRAsq> _______________________________________________ SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev
participants (2)
-
Stephan Hoyer
-
Touqir Sajed