Hi Andrea,
On 8. Jan 2021, at 10:20, Andrea Gavana <andrea.gavana@gmail.com> wrote:
long time no see :-) . I thought to start 2021 with a bit of a bang, to try and forget how bad 2020 has been... So I am happy to present you with a revamped version of the Global Optimization Benchmarks from my previous exercise in 2013.
This new set of benchmarks pretty much superseeds - and greatly expands - the previous analysis that you can find at this location: http://infinity77.net/global_optimization/ .
thank you for sharing this. I was going to point out the "No Free Lunch Theorem" but you mention it yourself on your website, good. I have a few questions/comments: - The exclusion of gradient-based solvers is unfortunate. It is of course up to you what you investigate, but gradient-based solvers are surely useful in practice. Not all real-life problems involve a (non-analytical) simulation. - "This effort stems from the fact that I got fed up with the current attitude of most mathematicians/numerical optimization experts, who tend to demonstrate the advantages of an algorithm based on “elapsed time” or “CPU time” or similar meaningless performance indicators." I don't know where you get that from, what are your sources? I have never seen an academic paper that used elapsed time or CPU time. The scientific papers I have read use the number of function evaluations to compare performance, which is a meaningful machine-independent performance measure if the total time is dominated by the time spend in the function, as it usually is. - I feel uneasy about your performance measure. It is whether the solver finds the minimum value of the function (why not the location of the minimum? Isn't that usually of interest?) within some fixed tolerance in 2000 function evaluations. a) The maximum number of function evaluations that you use does not depend on the dimensionality of the problem, but it clearly should. The search space is larger in higher dimensional problems, so more evaluations are needed by any algorithm. That is also obvious from your results. b) Instead of recording a binary outcome (success/failure to find the minimum in N evaluations), I think it would be more useful to record the number of evaluations until the minimum is reached and then give the mean or median number of function evaluations over many trials as well as the percentage of successful convergences. Algorithms can be ranked by robustness (the number of correctly solved problems) and convergence rate (the average/median number of function evaluations). Robustness and convergence rate may be anti-correlated. You are mixing the two in your performance measure, which makes it more difficult to interpret. - http://infinity77.net/global_optimization/ does not list the same algorithms as the table in http://infinity77.net/go_2021/thebenchmarks.html#info-general-results - While I think we agree that CPU-time is not a useful means to compare algorithms, it is then quite surprising to see Fig 0.6 with the CPU times. I suppose the pure-Python implementations perform so badly because the benchmark functions are all rather small python functions which are quick to evaluate so that the time spend inside the solver does matter. - I see some an inconsistency between your declared goals and your benchmark. You say you only care about optimization of non-analytical functions in which the function computation involves a simulation, but then most of your benchmark functions are analytical functions. In other words, you do not measure the performance on non-analytical functions. It is quite possible that the ranking would look different if you used non-analytical functions in the benchmarks. - I would prefer to read technical documents written in a more professional detached writing style and I think others would do, too. Regards, Hans