Unable to reproduce paper's results on MeSH dataset

[chensh911]

Hello,

I am reaching out to you regarding the results of your paper, where you evaluated your model's performance on both the MAG and MeSH datasets.

I have tried to reproduce your results on both datasets using your code and the same hyperparameters as mentioned in the paper, and I was able to obtain results similar to the ones reported in the paper for the MAG dataset. However, when I ran the same experiments on the MeSH dataset, I found that my results were worse than those reported in the paper. My result in MeSH is

P@1,3,5: 0.9117345916709328 , 0.7319429296414183 , 0.5970285129209607
NDCG@1,3,5: 0.9117345916709328 , 0.791101895548494 , 0.7191552905597784

I was wondering if there are any additional parameters or configurations that I need to consider while running the experiments on the MeSH dataset. Is it possible that the configuration for the MeSH dataset is different from the one used for the MAG dataset?

Thank you for your time and I look forward to your response.

[chensh911]

I use the "MATCH-NoHierarchy" model (with reg=0) to run.

[yuzhimanhua]

Hi, thank you for your interest in our work!

Previously, we tuned model hyperparameters on the MAG dataset. Then, for consistency across datasets, as far as I can recall, we used the same hyperparameters for the MeSH dataset.

In case I misremember, could you try a larger n_probes (e.g., 15)?
https://github.com/yuzhimanhua/MATCH/blob/master/configure/models/MATCH-MeSH.yaml#L6

Or could you try a smaller n_layers (e.g., 2 or even 1)?
https://github.com/yuzhimanhua/MATCH/blob/master/configure/models/MATCH-MeSH.yaml#L5

Thanks a lot!

[chensh911]

Thank you for your reply. After I change to smaller n_layers, it work better.