Instability in reproducing GUE dataset result

[Mamingqian]

Hi,

I notice a strong instability in the performance of GUE dataset result reproduction. It seems that in different iteration, the mcc score result differ a lot (causing the ranking of baselines and the model to shift). I reproduce with your exact code and GPU settings.

I wonder how you dealt with this issue, by averaging several iterations or some other solutions?

Thank you

[Zhihan1996]

Hey,

For the results I reported in the paper, I use 3 random seeds and take the average of them. I remember the results are relatively stable in most datasets excepts for the covid dataset and a few ones in EMP.