microsoft/Semi-supervised-learning

Cannot reproduce softmatch result on ag_news dataset

Mezbaur-Rahman opened this issue · 3 comments

Mezbaur-Rahman commented

I am running the following code:

python3 train.py --c config/usb_nlp/softmatch/softmatch_ag_news_40_0.yaml

But the accuracy I get is 0.8447368421 i.e error rate of 0.15 which is higher than what is reported in the result section (11.9±0.27). I am running the code with the exact config file from the repo. I also tried with different seeds but no improvement. What can be the issue here?

Hhhhhhao commented

Hi there.

Have a check at the log file here (from our last run): https://drive.google.com/drive/u/0/folders/1a9mWxDRiWBFbs0niAKNVAiUQyENFUghA

We merged several major pull request since then so it is possible that the parameters in the config file have been changed. You can check all the parameters we used in the log file, and also the final results.

Mezbaur-Rahman commented

I cannot find the use_amp parameter in my log file, it is also not present in the config file. Should I create a new argument named use_amp and set it to False? Also, the num_workers was set to 4 in my case.

Hhhhhhao commented

I cannot find the use_amp parameter in my log file, it is also not present in the config file. Should I create a new argument named use_amp and set it to False? Also, the num_workers was set to 4 in my case.

I believe that argument has been changed in the past commits. Now we might be using --amp, can you have a check on that? But I think both 'amp' and 'num_workers' wouldn't affect the performance too much.