warmup performance good but after poor
Closed this issue · 7 comments
Hi there , just tested the model on my dataset and the performance during warm up is good, but it decreases a lot during training. Acutally the test metrics doesn;t change. Seems like the trained model is not used at all. The loss is performing good. Could you please share any ideas? Much appreciated.
Warmup:
| Building net
save noisy labels to ./data/Wafer/noise_file/clean_label_058597.json ...
Warmup Net1
Wafer:clean_label | Epoch [ 0/600] Iter[ 1/104] CE-loss: 7.5067 Penalty-loss: -7.1986 All-loss: 7.5067
Wafer:clean_label | Epoch [ 0/600] Iter[101/104] CE-loss: 1.7427 Penalty-loss: -1.7540 All-loss: 1.7427
| Test Epoch #0 Acc Net1: 0.18%, Acc Net2: 2.74% Acc Mean: 4.44%
Warmup Net1
Wafer:clean_label | Epoch [ 1/600] Iter[ 1/104] CE-loss: 1.5454 Penalty-loss: -1.7752 All-loss: 1.5454
Wafer:clean_label | Epoch [ 1/600] Iter[101/104] CE-loss: 1.0084 Penalty-loss: -1.2549 All-loss: 1.0084
| Test Epoch #1 Acc Net1: 3.45%, Acc Net2: 1.19% Acc Mean: 70.52%
Warmup Net1
Wafer:clean_label | Epoch [ 2/600] Iter[ 1/104] CE-loss: 0.7619 Penalty-loss: -0.9628 All-loss: 0.7619
Wafer:clean_label | Epoch [ 2/600] Iter[101/104] CE-loss: 0.9429 Penalty-loss: -1.0692 All-loss: 0.9429
| Test Epoch #2 Acc Net1: 2.54%, Acc Net2: 1.53% Acc Mean: 84.39%
after:
Wafer:clean_label | Epoch [ 3/600] Iter[ 82/104] Net1 loss: 0.74 Net2 loss: 0.85
Wafer:clean_label | Epoch [ 3/600] Iter[ 83/104] Net1 loss: 0.88 Net2 loss: 0.84
Wafer:clean_label | Epoch [ 3/600] Iter[ 84/104] Net1 loss: 0.73 Net2 loss: 0.94
Wafer:clean_label | Epoch [ 3/600] Iter[ 85/104] Net1 loss: 0.84 Net2 loss: 0.87
Wafer:clean_label | Epoch [ 3/600] Iter[ 86/104] Net1 loss: 0.74 Net2 loss: 0.61
Wafer:clean_label | Epoch [ 3/600] Iter[ 87/104] Net1 loss: 0.69 Net2 loss: 0.70
Wafer:clean_label | Epoch [ 3/600] Iter[ 88/104] Net1 loss: 0.76 Net2 loss: 0.87
Wafer:clean_label | Epoch [ 3/600] Iter[ 89/104] Net1 loss: 0.73 Net2 loss: 0.79
Wafer:clean_label | Epoch [ 3/600] Iter[ 90/104] Net1 loss: 0.61 Net2 loss: 0.81
Wafer:clean_label | Epoch [ 3/600] Iter[ 91/104] Net1 loss: 0.72 Net2 loss: 0.87
Wafer:clean_label | Epoch [ 3/600] Iter[ 92/104] Net1 loss: 0.92 Net2 loss: 0.92
Wafer:clean_label | Epoch [ 3/600] Iter[ 93/104] Net1 loss: 0.79 Net2 loss: 0.78
Wafer:clean_label | Epoch [ 3/600] Iter[ 94/104] Net1 loss: 0.80 Net2 loss: 0.83
Wafer:clean_label | Epoch [ 3/600] Iter[ 95/104] Net1 loss: 0.65 Net2 loss: 0.76
Wafer:clean_label | Epoch [ 3/600] Iter[ 96/104] Net1 loss: 0.64 Net2 loss: 0.57
Wafer:clean_label | Epoch [ 3/600] Iter[ 97/104] Net1 loss: 0.71 Net2 loss: 0.69
Wafer:clean_label | Epoch [ 3/600] Iter[ 98/104] Net1 loss: 0.83 Net2 loss: 0.65
Wafer:clean_label | Epoch [ 3/600] Iter[ 99/104] Net1 loss: 0.57 Net2 loss: 0.73
Wafer:clean_label | Epoch [ 3/600] Iter[100/104] Net1 loss: 0.72 Net2 loss: 0.72
Wafer:clean_label | Epoch [ 3/600] Iter[101/104] Net1 loss: 0.84 Net2 loss: 0.90
Wafer:clean_label | Epoch [ 3/600] Iter[102/104] Net1 loss: 0.70 Net2 loss: 0.50
Wafer:clean_label | Epoch [ 3/600] Iter[103/104] Net1 loss: 0.75 Net2 loss: 0.79
| Test Epoch #3 Acc Net1: 2.63%, Acc Net2: 2.63% Acc Mean: 2.63%
I had the same problem, the training during the warm-up phase didn't work, but I didn't find any bugs at the moment
I use my own code for warm-up training
I had the same problem, the training during the warm-up phase didn't work, but I didn't find any bugs at the moment
Hi friend, anyway i didn't solve this at all. It's a good paper and there's lots of sophisticated stuff in the paper. I think it just needs more time to be investigated. But given my time, I tested my datasets on a prior study named DivideMix. I think DivideMix is some sort of backbone of ProMix in some sense. The structure is simpler and i get resonable results there. Take a look if you need. Hope it's helpful.
I replicated their code by implementing it myself，emmmm
Thank you for the question！ During the warm-up phase, it's possible to observe low accuracy for Acc-Net1/Net2 while the average accuracy Acc Mean remains normal. This is because, during the warm-up, only the original head is trained and updated, while the auxiliary pseudo head remains untrained. The accuracy for Net1/Net2 is calculated based on the pseudo head (thus very low during warm-up), whereas the average accuracy Acc-Mean is calculated based on the original head (thus remains normal). You can modify the output of the accuracy for net1/net2 in the test function accordingly (we merely adopt Acc Mean for evaluation in our experiments).
Thank you for your detailed answer