The result record in train.py has two items.
BurningFr opened this issue · 6 comments
In train.py(L145), I find the code also records the best performance of test domain in logger.save_best().
Could you point out which result the paper uses, is it the best performance of test domain (test_res.max()) or the best validation performance (test_res[idx_best])?
Thanks a lot!
Hi, it's the best validation result. Fix the random seed, and run multiple times to report the average.
Hi, thank you for answering my question!
I find there are some missing files in ./models (alexnet.py). I run the 'train.py' on my server and the reproduction results cannot achieve as high as the paper released (PACS dataset on resnet18). If I miss something important? I have annotated some codes in class 'JigsawNewDataset' which are for jigsaw data generate process to run RSC model.
My reproduction results is 79.45% for Sketch and 80.22% for Art Painting.
I have uploaded my environment. Could you try to run the code in that environment? I attach some results I just got for Sketch and Art painting.
Thanks for pointing out the legacy code in class 'JigsawNewDataset'.
sketch
Dataset size: train 5446, val 616, test 3929
Step size: 24
0/42 of epoch 1/30 class : 2.189 - acc class : 7.03 [bs:128]
30/42 of epoch 1/30 class : 0.479 - acc class : 87.50 [bs:128]
Accuracies on val: class : 88.96
Accuracies on test: class : 64.77
New epoch - lr: 0.008
0/42 of epoch 2/30 class : 1.116 - acc class : 58.59 [bs:128]
30/42 of epoch 2/30 class : 0.335 - acc class : 92.19 [bs:128]
Accuracies on val: class : 92.05
Accuracies on test: class : 66.56
New epoch - lr: 0.008
0/42 of epoch 3/30 class : 0.646 - acc class : 72.66 [bs:128]
30/42 of epoch 3/30 class : 0.299 - acc class : 94.53 [bs:128]
Accuracies on val: class : 90.75
Accuracies on test: class : 69.92
New epoch - lr: 0.008
0/42 of epoch 4/30 class : 0.513 - acc class : 71.88 [bs:128]
30/42 of epoch 4/30 class : 0.434 - acc class : 83.59 [bs:128]
Accuracies on val: class : 93.51
Accuracies on test: class : 57.19
New epoch - lr: 0.008
0/42 of epoch 5/30 class : 0.324 - acc class : 89.84 [bs:128]
30/42 of epoch 5/30 class : 0.235 - acc class : 94.53 [bs:128]
Accuracies on val: class : 96.59
Accuracies on test: class : 71.21
New epoch - lr: 0.008
0/42 of epoch 6/30 class : 0.156 - acc class : 98.44 [bs:128]
30/42 of epoch 6/30 class : 0.208 - acc class : 92.97 [bs:128]
Accuracies on val: class : 93.83
Accuracies on test: class : 77.50
New epoch - lr: 0.008
0/42 of epoch 7/30 class : 0.243 - acc class : 92.97 [bs:128]
30/42 of epoch 7/30 class : 0.255 - acc class : 91.41 [bs:128]
Accuracies on val: class : 95.13
Accuracies on test: class : 72.94
New epoch - lr: 0.008
0/42 of epoch 8/30 class : 0.183 - acc class : 95.31 [bs:128]
30/42 of epoch 8/30 class : 0.173 - acc class : 94.53 [bs:128]
Accuracies on val: class : 94.48
Accuracies on test: class : 76.25
New epoch - lr: 0.008
0/42 of epoch 9/30 class : 0.107 - acc class : 98.44 [bs:128]
30/42 of epoch 9/30 class : 0.282 - acc class : 89.06 [bs:128]
Accuracies on val: class : 94.32
Accuracies on test: class : 77.76
New epoch - lr: 0.008
0/42 of epoch 10/30 class : 0.152 - acc class : 94.53 [bs:128]
30/42 of epoch 10/30 class : 0.081 - acc class : 98.44 [bs:128]
Accuracies on val: class : 95.94
Accuracies on test: class : 75.03
New epoch - lr: 0.008
0/42 of epoch 11/30 class : 0.092 - acc class : 97.66 [bs:128]
30/42 of epoch 11/30 class : 0.246 - acc class : 91.41 [bs:128]
Accuracies on val: class : 95.45
Accuracies on test: class : 78.62
New epoch - lr: 0.008
0/42 of epoch 12/30 class : 0.084 - acc class : 98.44 [bs:128]
30/42 of epoch 12/30 class : 0.047 - acc class : 100.00 [bs:128]
Accuracies on val: class : 95.45
Accuracies on test: class : 72.36
New epoch - lr: 0.008
0/42 of epoch 13/30 class : 0.084 - acc class : 98.44 [bs:128]
30/42 of epoch 13/30 class : 0.146 - acc class : 96.09 [bs:128]
Accuracies on val: class : 95.78
Accuracies on test: class : 75.44
New epoch - lr: 0.008
0/42 of epoch 14/30 class : 0.036 - acc class : 100.00 [bs:128]
30/42 of epoch 14/30 class : 0.037 - acc class : 100.00 [bs:128]
Accuracies on val: class : 94.97
Accuracies on test: class : 73.63
New epoch - lr: 0.008
0/42 of epoch 15/30 class : 0.106 - acc class : 96.09 [bs:128]
30/42 of epoch 15/30 class : 0.083 - acc class : 98.44 [bs:128]
Accuracies on val: class : 94.48
Accuracies on test: class : 79.05
New epoch - lr: 0.008
0/42 of epoch 16/30 class : 0.053 - acc class : 99.22 [bs:128]
30/42 of epoch 16/30 class : 0.035 - acc class : 100.00 [bs:128]
Accuracies on val: class : 93.99
Accuracies on test: class : 75.21
New epoch - lr: 0.008
0/42 of epoch 17/30 class : 0.064 - acc class : 98.44 [bs:128]
30/42 of epoch 17/30 class : 0.038 - acc class : 99.22 [bs:128]
Accuracies on val: class : 95.62
Accuracies on test: class : 74.62
New epoch - lr: 0.008
0/42 of epoch 18/30 class : 0.061 - acc class : 99.22 [bs:128]
30/42 of epoch 18/30 class : 0.031 - acc class : 99.22 [bs:128]
Accuracies on val: class : 94.97
Accuracies on test: class : 76.10
30/42 of epoch 19/30 class : 0.057 - acc class : 98.44 [bs:128]
Accuracies on val: class : 94.97
Accuracies on test: class : 77.22
New epoch - lr: 0.008
0/42 of epoch 20/30 class : 0.017 - acc class : 100.00 [bs:128]
30/42 of epoch 20/30 class : 0.023 - acc class : 100.00 [bs:128]
Accuracies on val: class : 95.13
Accuracies on test: class : 76.43
New epoch - lr: 0.008
0/42 of epoch 21/30 class : 0.049 - acc class : 96.88 [bs:128]
30/42 of epoch 21/30 class : 0.077 - acc class : 99.22 [bs:128]
Accuracies on val: class : 94.81
Accuracies on test: class : 79.97
New epoch - lr: 0.008
0/42 of epoch 22/30 class : 0.013 - acc class : 100.00 [bs:128]
30/42 of epoch 22/30 class : 0.100 - acc class : 98.44 [bs:128]
Accuracies on val: class : 94.16
Accuracies on test: class : 76.69
New epoch - lr: 0.008
0/42 of epoch 23/30 class : 0.013 - acc class : 100.00 [bs:128]
30/42 of epoch 23/30 class : 0.011 - acc class : 100.00 [bs:128]
Accuracies on val: class : 95.13
Accuracies on test: class : 78.72
New epoch - lr: 0.008
0/42 of epoch 24/30 class : 0.068 - acc class : 98.44 [bs:128]
30/42 of epoch 24/30 class : 0.017 - acc class : 100.00 [bs:128]
Accuracies on val: class : 95.29
Accuracies on test: class : 78.44
New epoch - lr: 0.0008
0/42 of epoch 25/30 class : 0.022 - acc class : 100.00 [bs:128]
30/42 of epoch 25/30 class : 0.027 - acc class : 99.22 [bs:128]
Accuracies on val: class : 96.59
Accuracies on test: class : 80.55
New epoch - lr: 0.0008
0/42 of epoch 26/30 class : 0.012 - acc class : 100.00 [bs:128]
30/42 of epoch 26/30 class : 0.014 - acc class : 100.00 [bs:128]
Accuracies on val: class : 96.75
Accuracies on test: class : 80.07
New epoch - lr: 0.0008
0/42 of epoch 27/30 class : 0.032 - acc class : 99.22 [bs:128]
30/42 of epoch 27/30 class : 0.059 - acc class : 98.44 [bs:128]
Accuracies on val: class : 96.27
Accuracies on test: class : 80.91
New epoch - lr: 0.0008
0/42 of epoch 28/30 class : 0.023 - acc class : 100.00 [bs:128]
30/42 of epoch 28/30 class : 0.014 - acc class : 100.00 [bs:128]
Accuracies on val: class : 96.75
Accuracies on test: class : 80.35
New epoch - lr: 0.0008
0/42 of epoch 29/30 class : 0.010 - acc class : 100.00 [bs:128]
30/42 of epoch 29/30 class : 0.047 - acc class : 99.22 [bs:128]
Accuracies on val: class : 96.75
Accuracies on test: class : 81.27
New epoch - lr: 0.0008
0/42 of epoch 30/30 class : 0.059 - acc class : 100.00 [bs:128]
30/42 of epoch 30/30 class : 0.047 - acc class : 100.00 [bs:128]
Accuracies on val: class : 96.43
Accuracies on test: class : 80.33
Best val 0.967532, corresponding test 0.812675 - best test: 0.812675, best epoch: 28
It took 557.549
art_painting
Dataset size: train 7137, val 806, test 2048
Step size: 24
0/55 of epoch 1/30 class : 3.264 - acc class : 0.78 [bs:128]
30/55 of epoch 1/30 class : 0.754 - acc class : 78.91 [bs:128]
Accuracies on val: class : 89.21
Accuracies on test: class : 76.32
New epoch - lr: 0.008
0/55 of epoch 2/30 class : 0.825 - acc class : 66.41 [bs:128]
30/55 of epoch 2/30 class : 0.510 - acc class : 87.50 [bs:128]
Accuracies on val: class : 93.42
Accuracies on test: class : 73.29
New epoch - lr: 0.008
0/55 of epoch 3/30 class : 0.423 - acc class : 85.16 [bs:128]
30/55 of epoch 3/30 class : 0.330 - acc class : 93.75 [bs:128]
Accuracies on val: class : 93.18
Accuracies on test: class : 77.54
New epoch - lr: 0.008
0/55 of epoch 4/30 class : 0.277 - acc class : 93.75 [bs:128]
30/55 of epoch 4/30 class : 0.379 - acc class : 85.94 [bs:128]
Accuracies on val: class : 91.56
Accuracies on test: class : 76.03
New epoch - lr: 0.008
0/55 of epoch 5/30 class : 0.296 - acc class : 88.28 [bs:128]
30/55 of epoch 5/30 class : 0.253 - acc class : 94.53 [bs:128]
Accuracies on val: class : 94.54
Accuracies on test: class : 77.83
New epoch - lr: 0.008
0/55 of epoch 6/30 class : 0.176 - acc class : 96.88 [bs:128]
30/55 of epoch 6/30 class : 0.198 - acc class : 96.09 [bs:128]
Accuracies on val: class : 94.67
Accuracies on test: class : 77.44
New epoch - lr: 0.008
0/55 of epoch 7/30 class : 0.242 - acc class : 90.62 [bs:128]
30/55 of epoch 7/30 class : 0.200 - acc class : 95.31 [bs:128]
Accuracies on val: class : 95.16
Accuracies on test: class : 79.54
New epoch - lr: 0.008
0/55 of epoch 8/30 class : 0.125 - acc class : 98.44 [bs:128]
30/55 of epoch 8/30 class : 0.153 - acc class : 96.09 [bs:128]
Accuracies on val: class : 92.80
Accuracies on test: class : 75.05
New epoch - lr: 0.008
0/55 of epoch 9/30 class : 0.079 - acc class : 98.44 [bs:128]
30/55 of epoch 9/30 class : 0.089 - acc class : 99.22 [bs:128]
Accuracies on val: class : 93.80
Accuracies on test: class : 75.78
New epoch - lr: 0.008
0/55 of epoch 10/30 class : 0.115 - acc class : 96.88 [bs:128]
30/55 of epoch 10/30 class : 0.131 - acc class : 95.31 [bs:128]
Accuracies on val: class : 95.41
Accuracies on test: class : 78.56
New epoch - lr: 0.008
0/55 of epoch 11/30 class : 0.091 - acc class : 99.22 [bs:128]
30/55 of epoch 11/30 class : 0.113 - acc class : 96.88 [bs:128]
Accuracies on val: class : 94.54
Accuracies on test: class : 81.25
New epoch - lr: 0.008
0/55 of epoch 12/30 class : 0.109 - acc class : 96.88 [bs:128]
30/55 of epoch 12/30 class : 0.140 - acc class : 96.88 [bs:128]
Accuracies on val: class : 95.78
Accuracies on test: class : 74.61
New epoch - lr: 0.008
0/55 of epoch 13/30 class : 0.043 - acc class : 100.00 [bs:128]
30/55 of epoch 13/30 class : 0.038 - acc class : 100.00 [bs:128]
Accuracies on val: class : 92.31
Accuracies on test: class : 80.57
New epoch - lr: 0.008
0/55 of epoch 14/30 class : 0.060 - acc class : 98.44 [bs:128]
30/55 of epoch 14/30 class : 0.123 - acc class : 95.31 [bs:128]
Accuracies on val: class : 95.41
Accuracies on test: class : 76.76
New epoch - lr: 0.008
0/55 of epoch 15/30 class : 0.082 - acc class : 96.09 [bs:128]
30/55 of epoch 15/30 class : 0.046 - acc class : 99.22 [bs:128]
Accuracies on val: class : 95.53
Accuracies on test: class : 79.88
New epoch - lr: 0.008
0/55 of epoch 16/30 class : 0.040 - acc class : 98.44 [bs:128]
30/55 of epoch 16/30 class : 0.030 - acc class : 99.22 [bs:128]
Accuracies on val: class : 95.66
Accuracies on test: class : 80.57
New epoch - lr: 0.008
0/55 of epoch 17/30 class : 0.062 - acc class : 98.44 [bs:128]
30/55 of epoch 17/30 class : 0.092 - acc class : 98.44 [bs:128]
Accuracies on val: class : 95.78
Accuracies on test: class : 78.42
New epoch - lr: 0.008
0/55 of epoch 18/30 class : 0.029 - acc class : 100.00 [bs:128]
30/55 of epoch 18/30 class : 0.042 - acc class : 99.22 [bs:128]
Accuracies on val: class : 94.04
Accuracies on test: class : 73.34
New epoch - lr: 0.008
0/55 of epoch 19/30 class : 0.079 - acc class : 96.88 [bs:128]
30/55 of epoch 19/30 class : 0.116 - acc class : 95.31 [bs:128]
Accuracies on val: class : 94.54
Accuracies on test: class : 76.81
New epoch - lr: 0.008
0/55 of epoch 20/30 class : 0.034 - acc class : 99.22 [bs:128]
30/55 of epoch 20/30 class : 0.019 - acc class : 100.00 [bs:128]
Accuracies on val: class : 96.15
Accuracies on test: class : 79.10
New epoch - lr: 0.008
0/55 of epoch 21/30 class : 0.032 - acc class : 99.22 [bs:128]
30/55 of epoch 21/30 class : 0.022 - acc class : 99.22 [bs:128]
Accuracies on val: class : 95.41
Accuracies on test: class : 76.32
New epoch - lr: 0.008
0/55 of epoch 22/30 class : 0.042 - acc class : 98.44 [bs:128]
30/55 of epoch 22/30 class : 0.024 - acc class : 100.00 [bs:128]
Accuracies on val: class : 95.41
Accuracies on test: class : 81.64
New epoch - lr: 0.008
0/55 of epoch 23/30 class : 0.034 - acc class : 99.22 [bs:128]
30/55 of epoch 23/30 class : 0.013 - acc class : 100.00 [bs:128]
Accuracies on val: class : 95.78
Accuracies on test: class : 81.84
New epoch - lr: 0.008
0/55 of epoch 24/30 class : 0.009 - acc class : 100.00 [bs:128]
30/55 of epoch 24/30 class : 0.032 - acc class : 99.22 [bs:128]
Accuracies on val: class : 95.78
Accuracies on test: class : 78.08
New epoch - lr: 0.0008
0/55 of epoch 25/30 class : 0.019 - acc class : 99.22 [bs:128]
30/55 of epoch 25/30 class : 0.010 - acc class : 100.00 [bs:128]
Accuracies on val: class : 96.28
Accuracies on test: class : 81.88
New epoch - lr: 0.0008
0/55 of epoch 26/30 class : 0.025 - acc class : 100.00 [bs:128]
30/55 of epoch 26/30 class : 0.012 - acc class : 100.00 [bs:128]
Accuracies on val: class : 96.53
Accuracies on test: class : 81.88
New epoch - lr: 0.0008
0/55 of epoch 27/30 class : 0.013 - acc class : 100.00 [bs:128]
30/55 of epoch 27/30 class : 0.009 - acc class : 100.00 [bs:128]
Accuracies on val: class : 96.40
Accuracies on test: class : 82.37
New epoch - lr: 0.0008
0/55 of epoch 28/30 class : 0.028 - acc class : 100.00 [bs:128]
30/55 of epoch 28/30 class : 0.005 - acc class : 100.00 [bs:128]
Accuracies on val: class : 96.40
Accuracies on test: class : 81.10
New epoch - lr: 0.0008
0/55 of epoch 29/30 class : 0.035 - acc class : 100.00 [bs:128]
30/55 of epoch 29/30 class : 0.009 - acc class : 100.00 [bs:128]
Accuracies on val: class : 97.02
Accuracies on test: class : 82.81
New epoch - lr: 0.0008
0/55 of epoch 30/30 class : 0.014 - acc class : 100.00 [bs:128]
30/55 of epoch 30/30 class : 0.009 - acc class : 100.00 [bs:128]
Accuracies on val: class : 96.65
Accuracies on test: class : 83.20
Best val 0.970223, corresponding test 0.828125 - best test: 0.832031, best epoch: 28
It took 562.823
Thank you for sharing your results. I fix the environment the same with your 'environment.yml' and run the PACS dataset with resnet18 for 5 times.
best val
| PACS | a | c | s | p | avg |
|---|---|---|---|---|---|
| 1 | 78.7109 | 77.4744 | 79.6386 | 94.4311 | 82.56375 |
| 2 | 80.4199 | 79.3089 | 80.4276 | 94.7305 | 83.72173 |
| 3 | 79.8828 | 78.8823 | 77.8061 | 93.9521 | 82.63083 |
| 4 | 77.9297 | 78.2423 | 74.7773 | 94.6108 | 81.39003 |
| 5 | 81.2012 | 75 | 75.6936 | 94.8503 | 81.68628 |
| avg | 79.6289 | 77.78158 | 77.66864 | 94.51496 | 82.39852 |
best test
| PACS | a | c | s | p | avg |
|---|---|---|---|---|---|
| 1 | 81.5918 | 78.4983 | 80.1985 | 94.6707 | 83.73983 |
| 2 | 82.0801 | 80.5887 | 80.6821 | 95.1497 | 84.62515 |
| 3 | 81.6406 | 79.8208 | 78.3151 | 95.2695 | 83.7615 |
| 4 | 82.1777 | 81.2713 | 81.0384 | 94.9701 | 84.86438 |
| 5 | 81.2012 | 79.029 | 82.0565 | 95.0299 | 84.32915 |
| avg | 81.73828 | 79.84162 | 80.45812 | 95.01798 | 84.264 |
I find the results is not stable with random initialization.
Hi, I run one experiment at a time in the server, and the best Val result usually appears at the last 5 epoch. It should be more stable and better.
I will try to fix the randomness anyway.
Got it! Thank you.