some error in the traindata
Closed this issue · 5 comments
@erkil1452 I trained a model using your train.txt, but there are some warning, I guess there are some error in you training data:
Epoch: [325][651/1587] Time 0.721 (0.724) Data 0.010 (0.013) Angular 2.076 (2.421) Loss 0.0120 (0.0128) Prediction Error 2.5859 (2.5833)
Epoch: [325][652/1587] Time 0.722 (0.724) Data 0.009 (0.013) Angular 2.126 (2.421) Loss 0.0104 (0.0128) Prediction Error 2.4450 (2.5831)
Epoch: [325][653/1587] Time 0.721 (0.724) Data 0.010 (0.013) Angular 2.402 (2.421) Loss 0.0133 (0.0128) Prediction Error 2.4985 (2.5830)
WARNING:root:NaN or Inf found in input tensor.
Epoch: [325][654/1587] Time 0.721 (0.724) Data 0.010 (0.013) Angular nan (nan) Loss 0.0152 (0.0128) Prediction Error 2.5007 (2.5829)
Epoch: [325][655/1587] Time 0.721 (0.724) Data 0.010 (0.013) Angular 2.482 (nan) Loss 0.0134 (0.0128) Prediction Error 2.5017 (2.5828)
Epoch: [325][656/1587] Time 0.720 (0.724) Data 0.010 (0.012) Angular 2.701 (nan) Loss 0.0138 (0.0128) Prediction Error 2.8752 (2.5832)
Epoch: [325][657/1587] Time 0.721 (0.724) Data 0.010 (0.012) Angular 1.995 (nan) Loss 0.0106 (0.0128) Prediction Error 2.3353 (2.5828)
Epoch: [325][658/1587] Time 0.723 (0.724) Data 0.010 (0.012) Angular 2.179 (nan) Loss 0.0112 (0.0128) Prediction Error 2.4611 (2.5826)
Epoch: [325][659/1587] Time 0.723 (0.724) Data 0.010 (0.012) Angular 2.341 (nan) Loss 0.0140 (0.0128) Prediction Error 2.5880 (2.5826)
Epoch: [325][660/1587] Time 0.721 (0.724) Data 0.010 (0.012) Angular 2.273 (nan) Loss 0.0114 (0.0128) Prediction Error 2.2808 (2.5822)
Epoch: [325][661/1587] Time 0.720 (0.724) Data 0.010
It seems you are getting NaN. You may need to investigate further into your model, data loader, training parameters etc. It is not easy to find culprit in these cases. I would try plotting the gradients over the course of the training.
@erkil1452 I use the data in your train.txt, and the model predict the gaze using only single RGB image. so my training data is only within the train.txt, each line in the train.txt is a training data.
I am noticing you get NaN only for the angular error. If this is the source of your issues I suspect that you have not clamped the input values you pass into a acos or asin to range [-1,1]. Just a possibility.
-
I change something like the model and the dataset loader modules based on your code. the dataset loader only change the input of 7 images to only one image， other not changed. the nan value not appeared in every epoch, maybe is a normal value in the next epoch.
-
what the effect will be caused by the nan value,the loss will not be decrease then the model will not be the best?
-
now my training log is:
Epoch: [393][26/1587] Time 0.721 (0.795) Data 0.010 (0.084) Angular 2.163 (2.203) Loss 0.0114 (0.0115) Prediction Error 2.3297 (2.4135)
Epoch: [393][27/1587] Time 0.721 (0.792) Data 0.010 (0.081) Angular 2.161 (2.201) Loss 0.0108 (0.0115) Prediction Error 2.2852 (2.4089)
Epoch: [393][28/1587] Time 0.721 (0.790) Data 0.009 (0.079) Angular 1.797 (2.187) Loss 0.0093 (0.0114) Prediction Error 2.1193 (2.3990)
Epoch: [393][29/1587] Time 0.721 (0.787) Data 0.010 (0.077) Angular 2.118 (2.185) Loss 0.0114 (0.0114) Prediction Error 2.2670 (2.3946)
Epoch: [393][30/1587] Time 0.722 (0.785) Data 0.010 (0.074) Angular 1.970 (2.178) Loss 0.0101 (0.0114) Prediction Error 2.1648 (2.3871)
Epoch: [393][31/1587] Time 0.722 (0.783) Data 0.010 (0.072) Angular 2.221 (2.179) Loss 0.0114 (0.0114) Prediction Error 2.2715 (2.3835)
Epoch: [393][32/1587] Time 0.722 (0.781) Data 0.010 (0.071) Angular 2.109 (2.177) Loss 0.0108 (0.0114) Prediction Error 2.2226 (2.3787)
Epoch: [393][33/1587] Time 0.722 (0.780) Data 0.010 (0.069) Angular 2.022 (2.173) Loss 0.0107 (0.0114) Prediction Error 2.3832 (2.3788)
the validate log is:
Epoch: [178/213] Time 0.218 (0.234) Angular 13.9332 (16.3664) Loss 0.1301 (0.1761)
Epoch: [179/213] Time 0.217 (0.234) Angular 18.2591 (16.3769) Loss 0.1606 (0.1760)
Epoch: [180/213] Time 0.218 (0.233) Angular 15.4334 (16.3717) Loss 0.2238 (0.1762)
Epoch: [181/213] Time 0.219 (0.233) Angular 17.6182 (16.3786) Loss 0.1944 (0.1763)
Epoch: [182/213] Time 0.218 (0.233) Angular 17.1792 (16.3830) Loss 0.2205 (0.1766)
Epoch: [183/213] Time 0.219 (0.233) Angular 18.0038 (16.3918) Loss 0.2387 (0.1769)
Epoch: [184/213] Time 0.219 (0.233) Angular 15.4107 (16.3865) Loss 0.1948 (0.1770)
Epoch: [185/213] Time 0.220 (0.233) Angular 15.2042 (16.3801) Loss 0.1492 (0.1769)
Epoch: [186/213] Time 0.218 (0.233) Angular 19.2744 (16.3956) Loss 0.2371 (0.1772)
the model in that status is the best? the validate angular error is still large. Is the model overfitting? what suggestion do you give when the model overfitting?
The NaN would completely spoil the network weights and make the model unusable.
If your model is overfitting you can either try to enlarge your training set, add augmentation to your training data, add noise, Dropouts, ... or just stop the training early.