Partial replication failure on ClueWeb12-B13 for ax models
lintool opened this issue · 8 comments
According to #590 @matthew-z wasn't able to replicate results with ax models on ClueWeb12-B13. We should look into this...
Somehow the AX results matched after indexing again with oracle JVM (1.8.0_201)....
Interesting.
Hypothesis is that pseudo-random number generation works differently in OpenJDK vs. Oracle JVM. This would explain the differences in ax results, which rely on sampling docs.
@matthew-z Just to be clear, with Oracle JVM you are able to replicate our figures exactly, while with OpenJDK you come close? (If so, how close is "close" in the latter case?)
Did you mean no-AX algorithms? They matched exactly.
E.g., 201-250, Bm25, NDCG@20, it got 0.12862 and expected 0.1286
201-250, QL, NDCG@20, it got 0.11067 and expected 0.1107
Okay, thanks! I'll send a PR to capture this discussion and ask you to review.
@matthew-z please look at #594
Hypothesis is that pseudo-random number generation works differently in OpenJDK vs. Oracle JVM.
I am not sure, but this is unlikely IMO.
I am going to test it again by myself
Whatever the issue, Oracle vs. openJDK isn't the issue. All regressions run fine on damiano:
[jimmylin@damiano ~]$ java --version
openjdk 11.0.4 2019-07-16 LTS
OpenJDK Runtime Environment 18.9 (build 11.0.4+11-LTS)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.4+11-LTS, mixed mode, sharing)