High variance of the result on DrugOOD.

[panmianzhi]

I run your code on DrugOOD for 5 times, but the result has very high variance. For example, the AUC-ROC on data_assay_ec50 are 71.41, 72.39, 61.93, 64.70, 67.24.
I can't understand why this happen. Maybe I make mistakes about the hyper-parameters, and they are as following:

I saw the similar question in the close issue. Can you tell me the detailed value of the hyper-parameters on each dataset (except the random seed)?

[yangnianzu0515]

Thank you very much for your interest in our work.

We think the issue might be related to your selection of hyperparameters. We notice in a recent paper accepted by NeurIPS23, titled "Learning Invariant Molecular Representation in Latent Discrete Space", the variance on the EC50 dataset was not as significant as what you have experienced.

However, it's been about a year and a half since that work was published, and we cannot recall the specific parameters. We suspect that part of the reason for the discrepancy might be due to your setting of the 'environment number' hyperparameter to 20, which seems somewhat high.

We hope our response is of some assistance to you.

[panmianzhi]

Thank you very much! PS: actually I also run the code of Learning Invariant Molecular Representation in Latent Discrete Space on DrugOOD, but the result is also not as good as their paper😂😂 Anyway, thank you for your reply!

[mxqmxqmxq]

@panmianzhi May I ask if you have read the article published in KDD 2024? It is of the same type as this one！