HDF5 trainvaltest split for PACS

Question

HDF5 trainvaltest split for PACS

YiDongOuYang opened this issue 4 years ago · 3 comments

YiDongOuYang commented 4 years ago

Could you release the hdf5 of PACS? I would like to make several improvements to matchDG. Thanks in advance!

Answer 1 · 2020-12-14T14:19:02.000Z

You can download the hdf5 of PACS from here: https://drive.google.com/drive/folders/0B6x7gtvErXgfUU1WcGY5SzdwZVk

Answer 2 · 2020-12-22T14:22:45.000Z

Dear Divyat,

I found I cannot reproduce the results on PACS. I used the same command mentioned in #2. The first image below is ERM algorithm (95.84% mentioned in the paper for ResNet18 when the unseen domain is Photo, while I only got 86.55%). The second image is matchdg_ctr.

Thanks in advance!

Answer 3 · 2020-12-22T14:44:43.000Z

Its because of hyperparameter tuning, the learning rate for the domain Photo should be 0.0005, which is different from the default in the library. Please refer to table 6 & 7 in this paper for details regarding hyperparams: https://arxiv.org/pdf/2006.07500.pdf

Thanks, I will also update the readme command.

Note: The current code in the master version does not show accuracy using different validation criteria. If you want to obtain test accuracy under different validation mechanism, refer to the code in this branch , which will be soon merged with the master also.