JosephKJ/OWOD

I want to train the network from scratch

JiuqingDong opened this issue · 0 comments

I want to train the network from scratch instead of a pre-trained model R-50.pkl. Then I annotate the weights in the config file.
image

However, I got an error.

File "/home/multiai3/anaconda3/envs/OWOD/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/home/multiai3/anaconda3/envs/OWOD/lib/python3.7/site-packages/torch/nn/parallel/distributed.py", line 886, in forward
output = self.module(*inputs[0], **kwargs[0])
File "/home/multiai3/anaconda3/envs/OWOD/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/home/multiai3/Jiuqing/OWOD-master/detectron2/modeling/meta_arch/rcnn.py", line 517, in forward
proposals, proposal_losses = self.proposal_generator(images, features, gt_instances)
File "/home/multiai3/anaconda3/envs/OWOD/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/home/multiai3/Jiuqing/OWOD-master/detectron2/modeling/proposal_generator/rpn.py", line 437, in forward
proposals = self.predict_proposals(anchors, pred_objectness_logits, pred_anchor_deltas, images.image_sizes)
File "/home/multiai3/Jiuqing/OWOD-master/detectron2/modeling/proposal_generator/rpn.py", line 469, in predict_proposals
self.training,
File "/home/multiai3/Jiuqing/OWOD-master/detectron2/modeling/proposal_generator/proposal_utils.py", line 92, in find_top_rpn_proposals
"Predicted boxes or scores contain Inf/NaN. Training has diverged."
FloatingPointError: Predicted boxes or scores contain Inf/NaN. Training has diverged.

could you help to solve this problem or give me some suggestions?