Dear SciPy Developers & Users,
long time no see :-) . I thought to start 2021 with a bit of a bang, to try and forget how bad 2020 has been... So I am happy to present you with a revamped version of the Global Optimization Benchmarks from my previous exercise in 2013.
Hi Andrea, this is awesome! Thanks for sharing!
I am happy you like it :-) .
This could be really useful to link to and use as a guide for providing recommendations for solvers to use in the scipy.optimize tutorials. It's good to see that SciPy overall is much more competitive than it was in 2013. Overall it seems SHGO is our most accurate solver, and making it faster seems worthwhile. That shouldn't be very difficult, given that it's all pure Python still.
I have to say that, compared to back in 2013, the addition of SHGO and DualAnnealing to SciPy has made the global optimization world in SciPy much more powerful, pretty much at the top of what can currently be done with open source solvers.
MCS isn't open source, but both DIRECT and BiteOpt are MIT-licensed and seem the best candidates to be considered for inclusion in SciPy.
I couldn't find a license restriction for MCS, but maybe I haven't looked hard enough... Do you have a link for it? I am just curious.
If you have recommendations or takeaways from all this work for SciPy, I'd love to hear them.
I have a few thoughts, but please bear in mind that it's my opinion and only based on this exercise plus a couple of real-life problems I have been working on recently:
1. Assuming we are dealing with a low dimensional problem, SHGO is close to unbeatable. I have found a glitch when the number of variables gets 10+, or for repeated continuous optimizations, SHGO seems to require enormous amounts of memory: in the SciPy Extended benchmark, trying all the 100 restarts got my RAM consumption to 190 GB for SHGO only - not something you want to ty unless you have a monster machine like mine.
2. DualAnnealing is also extremely powerful - ranking consistently close to the top for most of the benchmarks. It probably requires some tuning of the parameters (which I haven't done), especially when the allowable number of functions evaluations is large. That said, you can clearly see DualAnnealing shining in the SciPy Extended, GKLS, LGMVG and RandomFields benchmarks.
3. BasinHopping and DifferentialEvolution are generally slightly weaker, at least on these benchmarks. That said, I have used both of them with great success on real life problems - albeit with generally generous budgets of functions evaluations.
4. Real-life-wise, I recently had three very tough problems to work on: one 9-dimensional objective function describing multi-phase decline curves for oil/gas wells, which I am now satisfactorily fitting with DualAnnealing. Another one on optimization of 3D well trajectories, which I am happily handing over to SHGO or MCS depending on the problem. And another one related to wind and renewable data fitting which DifferentialEvolution is handling quite well.
So, all in all, benchmarks only give you so much information: real-life problems sometimes defy the accepted wisdom that occurs because of contrived (but synthetic) objective functions. That said, if I had to attack a new problem and I had no idea where to start, I would definitely give SHGO and DualAnnealing the first go, as they are quite powerful across a large spectrum of problems.
In the end, I believe that SciPy will definitely benefit from the addition of a couple (few?) more robust global solvers, especially if they implement techniques that are completely different from the existing ones (such as DIRECT, MCS, BiteOpt, of course). Giving more options to users is always going to make people happy - but of course you have to balance it with the maintenance efforts in the library.
Andrea.