CLUE paper reproducibility code

Question

CLUE paper reproducibility code

Closed this issue a year ago · 6 comments

In the supplementary material to the CLUE paper, you mention a .zip file with all the relevant code; however I can't find this anywhere. Did you use the exact same setup as published on the competition git? I'm trying to run some benchmarks and want to stay as close to your paper as possible.

Greetings
Valentin

Answer 1 · 2023-07-20T02:13:17.000Z

Hi @valentinwust! Yes that's the identical setup as the competition git.

Answer 2 · 2023-07-20T09:05:12.000Z

Thank you, then I'll probably just use that. That being said, this uses scclue 0.1.1, and I've noticed some slight changes in the code of CLUE since then. Do they meaningfully affect the preformance?

Answer 3 · 2023-07-20T11:22:11.000Z

Actually that was not 0.1.1, but a version between 0.1.1 and 0.2.0. We didn't tag it as a dedicated public release. To exactly reproduce the CLUE model in the competition, I'd suggest using the whl file in the competition repo to install scglue. Apologies for the inconvenience.

Answer 4 · 2023-07-20T11:43:15.000Z

No problem, this is indeed what I am currently doing. I was just wondering whether the current release (0.3.2) would perform better or similar, if it has a relevant performance increase it might make sense to use the most current version instead.

Answer 5 · 2023-07-20T11:48:03.000Z

Great! There shouldn't be a noticeable performance difference. You may just use the whl version to avoid needing to change some of the function calls due to API changes along the way.

Answer 6 · 2023-07-20T12:29:47.000Z

Thanks, then I'll do that.