Loss not reducing
ashutosh1807 opened this issue · 4 comments
Hi I am trying to train depth former but the loss is not reducing. PFA log file
swin_nyu.log
.
I think that's a convergence issue of the direct regression model with the si-log loss. A better solution may use other losses like RMSE loss to warm up the model and then apply the silog, but I do not have enough time right now to conduct these modifications.
For now, you can just re-run the training script, since this convergence issue randomly occurs. When you see the loss in your log is around 3.0 but does not continually drop, it means the model has collapsed. Just stop it and re-run the script. Once the loss drops, there will be no other problems in the training stage.
Are you sure that is the issue? BTS also has direct regression model with the si-log loss, but the loss works for that.
Yes, definitely. It is the issue of how to "directly" regress the predictions. Please refer to these codes.
In BTS, they apply the sigmoid as (see config):
output = self.sigmoid(self.conv_depth(feat)) * self.max_depth.
In contrast, we directly regress the predictions as
output = self.relu(self.conv_depth(feat)) + self.min_depth,
since the scale_up in depthformer is set to False.
I conduct several experiments and find that there will be no convergence problem if we set scale_up = True in depthformer. In our paper, we write "directly regress", so I keep the alignment of the implementation and the paper. May I will simply set the scale_up = True or add the warmup stage in the future, but the performance is on par in these settings.
Okay, thanks. I will close this now.