What do the benchmark results mean?
Closed this issue · 2 comments
This documentation page discusses a benchmark of Percival vs IPOpt. It says:
it [the result of benchmarks] can also be used to make performance profiles.
...and that's where the page ends.
Now, what do these graphs mean? Are these cumulative distribution functions? What does "within this factor of the best" refer to? "The best" what, exactly?
Concerning the first graph. If these are CDFs, I see that Percival's results were within a factor of a bit more than 1 from "the best" (whatever this means) for 75% of the optimization problems solved during the benchmark. IPOpt achieved that less than 50% of the time, so Percival is (often) faster, right? However, while IPOpt maxed out at a factor of 16, Percival kept getting slower for about 20% of the problems. So, more than 75% of the time Percival is much faster, but sometimes it can be significantly slower (a factor of 2^4 or 2^5 doesn't seem that great). Is this how this graph should be understood?
Concerning the second graph, it's not quite clear which of the two uses less function evaluations...
I think potential users of Percival would appreciate at least some kind of discussion and explanation of the benchmark results. Currently there's a huge table with a lot of columns (and also rows) full of seemingly random numbers (almost impossible to understand, IMO) and these two graphs that are just there and aren't described in any way, so it's not clear at all what they mean.
Users will likely try to use these results to help them choose a good optimizer, but interpreting graphs on one's own is hard and error-prone, so people can easily give up or misunderstand them and thus not choose Percival and potentially miss out on a great optimizer.
In a nutshell, some explanation could help a lot here.
Hi @ForceBru ! Thanks for the comment. These are performance profiles. Here are two references explaining it:
- https://link.springer.com/article/10.1007/s101070100263
- https://openresearchsoftware.metajnl.com/articles/10.5334/jors.81/
Overall, I think you are right, we should add more comments to these figures as an extension to the tutorials.