On Mon, Feb 15, 2021 at 2:02 PM Ralf Gommers <ralf.gommers@gmail.com> wrote:
On Mon, Feb 15, 2021 at 6:24 PM Pamphile Roy <roy.pamphile@gmail.com> wrote:
*scipy.optimize:* Would it be wanted to have a possibility to have workers to evaluate the function during an optimization? In most industrial context, the function is not trivial and might require minutes if not hours or even days to compute. Having a simple way to first parallelise the runs would help. We have machines with easily ten cores now and it would be great to leverage this here.
Definitely - see the mention of workers under http://scipy.github.io/devdocs/roadmap.html#performance-improvements.
Going that direction, having a more general infrastructure to handle
external workers would be great.
I'm assuming you mean something like standard multiprocessing, or using a custom Pool object, for code that's trivially parallelizable. Both are covered by the `workers` pattern. If you're thinking about something else, can you elaborate?
A standard approach for this is to organize the implementation of the optimization algorithms in what's usually called an "ask-tell" interface. The minimize()-style interface is easy to implement from an ask-tell interface, but not vice-versa. Basically, you have the optimizer object expose two methods, ask(), which returns a next point to evaluate, and tell(), where you feed back the point and its evaluated function value. You're in charge of evaluating that function. This gives you a lot of flexibility in how to dispatch that function evaluation, and importantly, we don't have to commit to any dependencies! That's the user's job! scikit-optimize implements their optimizers in this style, for example. It's pretty common for optimizers that are geared towards expensive evaluations. https://scikit-optimize.github.io/stable/auto_examples/ask-and-tell.html I think it might be a well-scoped GSoC project to start re-implementing a chosen handful of the algorithms in scipy.optimize in such an interface. It could even be a trial run as an external package (even in scikit-optimize, if they're amenable). Then we can evaluate whether we want to adopt that framework inside scipy.optimize and make a roadmap for re-implementing all of the algorithms in that style. It will be a technical challenge to adapt the FORTRAN-implemented algorithms to such an interface. I will not be available to mentor such a project, but that's the general approach that I would recommend. I think it would be a valuable addition. -- Robert Kern