Reproducing results on Deezer dataset
Chenhui1016 opened this issue · 5 comments
Hi there,
Thanks for the fantastic work!
I'm running run.sh and find out that the metric for Deezer is set to be "rocauc", while the paper uses accuracy as the metric on Deezer (shown in Figure 2). When I change the metric from "rocauc" to "acc" in run.sh, the averaged accuracy is 65.12%, which is much lower than the accuracy reported in the paper (~71%). Could you kindly let me know the proper hyperparameter setting for reproducing the results on Deezer? Thanks in advance!
Hi Chenhui,
The metric we used for Deezer is rocauc (for binary classification on this dataset) and the results can be reproduced using our provided hyper-parameters in run.sh. After double check, the plotted scores in Figure 2 for Deezer are indeed rocauc and correct, and we have fixed the name on the y-axis of Deezer in a new version uploaded to ResearchGate. Sorry for the confusion.
Thanks for your prompt reply! I can reproduce NodeFormer's rocauc score on Deezer.
Btw, I just got the rocauc of 72.42 ± 0.38% on ogbn-proteins by using the hyperparameters in run.sh, which seems to be lower than 77.45 ± 1.15% reported in the paper. Could you kindly check those default hyperparameters on ogbn-proteins in run.sh? Thanks again for your help!
I just checked for this, and we indeed use the hyper-parameters in run.sh for ogbn-proteins. We run the model on a RTX 2080Ti, and can reproduce the result. Can you tell me what GPU is used for your experiment?
I'm using RTX A6000. Got it, thx!
Hi Chenhui,
I have added the model checkpoints and code for testing on ogbn-proteins, which may help to make it more convenient to reproduce the score (77+) in our paper if needed for further research.
The reason you achieved the score of 72 could be that your installed packages are inconsistent with ours (see the requirements.txt for details), and maybe you can check for that if you need to reproduce the results with training from scratch