No matter what method I use, the results on sketch domain (pacs dataset) are very bad

[hugddygff]

I have tuned the hyperparameters according to the paper,

but my acc is always about 70,
the acc reported in the paper is >= 75,

There's a big difference in the accuracy.

[divyat09]

Thanks for pointing this, the code in the master has not been updated. The current code is in the mia_adv_attacks branch: https://github.com/microsoft/robustdg/tree/mia_adv_attacks

The difference you observe currently is most probably due to no model selection based on validation accuracy in the master branch. For example, in the mia_adv_attacks branch, algorithm erm_match line 146 saves the model based on the best validation accuracy.

I will integrate the mia_adv_attacks branch with the master by this week and update you here. Else you may use the code from the mia_adv_attacks branch for now.

[hugddygff]

Thanks for pointing this, the code in the master has not been updated. The current code is in the mia_adv_attacks branch: https://github.com/microsoft/robustdg/tree/mia_adv_attacks

The difference you observe currently is most probably due to no model selection based on validation accuracy in the master branch. For example, in the mia_adv_attacks branch, algorithm erm_match line 146 saves the model based on the best validation accuracy.

I will integrate the mia_adv_attacks branch with the master by this week and update you here. Else you may use the code from the mia_adv_attacks branch for now.

I also try to check the best acc of validation, but it still has a gap with the results in the paper(about 72-73). Or somtimes, we got 75 in a run, the other runs give results less than 70. Therefore, the average accuracy is bad and the variance is big.

[divyat09]

Did you try it with the 'mia_adv_attacks' branch? Like I executed the following command for ERM on Sketch domain, and I am able to reproduce results and obtain good accuracies on the Sketch domain

python train.py --dataset pacs --method_name erm_match --match_case 0.01 --penalty_ws 0.0 --train_domains photo art_painting cartoon --test_domains sketch --out_classes 7 --perfect_match 0 --img_c 3 --pre_trained 1 --lr 0.01 --epochs 50 --model_name resnet18 --weight_decay 0.001

Please check once with the branch mia_adv_attacks, the code in the master needs to be updated (will merge this branch soon with master). The difference in the two branches is most probably is due to the validation strategy.

[hugddygff]

Did you try it with the 'mia_adv_attacks' branch? Like I executed the following command for ERM on Sketch domain, and I am able to reproduce results and obtain good accuracies on the Sketch domain

python train.py --dataset pacs --method_name erm_match --match_case 0.01 --penalty_ws 0.0 --train_domains photo art_painting cartoon --test_domains sketch --out_classes 7 --perfect_match 0 --img_c 3 --pre_trained 1 --lr 0.01 --epochs 50 --model_name resnet18 --weight_decay 0.001

Please check once with the branch mia_adv_attacks, the code in the master needs to be updated (will merge this branch soon with master). The difference in the two branches is most probably is due to the validation strategy.

Thanks for the reply and I will try to use this branch.

[divyat09]

The branch mia_adv_attacks has been merged with the master branch. You can run the commands directly from the master and reproduce the results. Also, you may refer to this updated notebook for reproducing the results of the paper.

Closing this issue.