RatingBenchmark comparator can crash when there are many players
matejdro opened this issue · 3 comments
matejdro commented
Under certain circumstances, RatingBenchmark can crash when there are many competing players with this exception:
Exception in thread "main" java.lang.IllegalArgumentException: Comparison method violates its general contract!
at java.util.TimSort.mergeLo(TimSort.java:777)
at java.util.TimSort.mergeAt(TimSort.java:514)
at java.util.TimSort.mergeCollapse(TimSort.java:441)
at java.util.TimSort.sort(TimSort.java:245)
at java.util.Arrays.sort(Arrays.java:1512)
at java.util.ArrayList.sort(ArrayList.java:1454)
at java.util.Collections.sort(Collections.java:175)
at org.um.feri.ears.benchmark.RatingBenchmark.setWinLoseFromResultList(RatingBenchmark.java:164)
at org.um.feri.ears.benchmark.RatingBenchmark.run(RatingBenchmark.java:222)
at CrashTest.main(CrashTest.java:38)
Code to reproduce:
(Slightly modified code of the MainBenchmarkTest example that just adds single algorithm 100x to simulate many players)
Util.rnd.setSeed(20);
ArrayList<Algorithm> players = new ArrayList<Algorithm>();
for (int i = 0; i < 100; i++) {
Algorithm algorithm = new ES1p1sAlgorithm();
// Give each instance different name, so they don't clash.
algorithm.setAlgorithmInfo(new AlgorithmInfo("", "", Integer.toString(i), ""));
players.add(algorithm);
}
ResultArena ra = new ResultArena(100);
RatingRPUOed2 suopm = new RatingRPUOed2(); //Create banchmark
suopm.setDisplayRatingIntervalChart(false);
for (Algorithm al:players) {
ra.addPlayer(al.getID(), 1500, 350, 0.06,0,0,0); //init rating 1500
suopm.registerAlgorithm(al);
}
BankOfResults ba = new BankOfResults();
suopm.run(ra, ba, 20); //repeat competition 20X
DKorosec commented
Can confirm, same error. It's impossible to optimize algorithm parameters using grid search.
(Which is actually our homework 😄 ... )