train loss

Question

train loss

linchunmian opened this issue 5 years ago · 6 comments

Hi, I run the code trained on BDD100K dataset from scratch, but the result returns loss nan. What's the problems? Could you please give me some advice for mode training in such case?
Thanks!

Answer 1 · 2019-12-28T15:34:50.000Z

@ChunmianLin
Probably because of the variance.
I recommend you to train the model using darknet53.conv.74.
Please see this: #16 (comment)

Answer 2 · 2019-12-29T06:13:24.000Z

thanks. I try it again.
In early time, I have asked you about KITTI test. One question confused me now is that its test code is basde on MATLAB and C, so need I setup according environment and test my model?

Answer 3 · 2019-12-30T05:27:39.000Z

Thanks for response! I have restored the pretrained model, but it seems to make no effect. The learning rate I use is from 1e-4 to 1e-6, is it too large for training? I don't know what's the problem.

…

-----原始邮件----- 发件人:"Jiwoong Choi" <notifications@github.com> 发送时间:2019-12-28 23:34:51 (星期六) 收件人: jwchoi384/Gaussian_YOLOv3 <Gaussian_YOLOv3@noreply.github.com> 抄送: ChunmianLin <cmlin@buaa.edu.cn>, Mention <mention@noreply.github.com> 主题: Re: [jwchoi384/Gaussian_YOLOv3] train loss (#47) @ChunmianLin Probably because of the variance. I recommend you to train the model using darknet53.conv.74. Please see this: #16 (comment) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

Answer 4 · 2019-12-30T12:33:47.000Z

thanks. I try it again.
In early time, I have asked you about KITTI test. One question confused me now is that its test code is basde on MATLAB and C, so need I setup according environment and test my model?

Maybe, you just need to use the C/C++ code for KITTI testing. I'm not sure but I think you don't need to setup the environment.

Thanks for response! I have restored the pretrained model, but it seems to make no effect. The learning rate I use is from 1e-4 to 1e-6, is it too large for training? I don't know what's the problem.
…
-----原始邮件----- 发件人:"Jiwoong Choi" notifications@github.com 发送时间:2019-12-28 23:34:51 (星期六) 收件人: jwchoi384/Gaussian_YOLOv3 Gaussian_YOLOv3@noreply.github.com 抄送: ChunmianLin cmlin@buaa.edu.cn, Mention mention@noreply.github.com 主题: Re: [jwchoi384/Gaussian_YOLOv3] train loss (#47) @ChunmianLin Probably because of the variance. I recommend you to train the model using darknet53.conv.74. Please see this: #16 (comment) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

It's not large. If you use the pre-trained weight file ("darknet53.conv.74") and BDD cfg file which I uploaded, the model can train well.
If not, try to increase the burn-in value in the cfg file or decrease the learning rate.

Answer 5 · 2020-01-03T14:23:14.000Z

Hi, jichoi. One problem confused me is how to validate your expermental result? Would the official server save your test record, for BDD100 and KITTI evaluation server? Thanks for your reply!

…

-----原始邮件----- 发件人:"Jiwoong Choi" <notifications@github.com> 发送时间:2019-12-30 21:13:22 (星期一) 收件人: jwchoi384/Gaussian_YOLOv3 <Gaussian_YOLOv3@noreply.github.com> 抄送: ChunmianLin <cmlin@buaa.edu.cn>, Mention <mention@noreply.github.com> 主题: Re: [jwchoi384/Gaussian_YOLOv3] train loss (#47) Closed #47. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

Answer 6 · 2020-01-16T10:07:13.000Z

Thanks, sincerely! 发送自 Windows 10 版邮件应用发件人: Jiwoong Choi 发送时间: 2019年12月30日 20:33 收件人: jwchoi384/Gaussian_YOLOv3 抄送: ChunmianLin; Mention 主题: Re: [jwchoi384/Gaussian_YOLOv3] train loss (#47) thanks. I try it again. In early time, I have asked you about KITTI test. One question confused me now is that its test code is basde on MATLAB and C, so need I setup according environment and test my model? Maybe, you just need to use the C/C++ code for KITTI testing. I'm not sure but I think you don't need to setup the environment. Thanks for response! I have restored the pretrained model, but it seems to make no effect. The learning rate I use is from 1e-4 to 1e-6, is it too large for training? I don't know what's the problem. … -----原始邮件----- 发件人:"Jiwoong Choi" notifications@github.com 发送时间:2019-12-28 23:34:51 (星期六) 收件人: jwchoi384/Gaussian_YOLOv3 Gaussian_YOLOv3@noreply.github.com 抄送: ChunmianLin cmlin@buaa.edu.cn, Mention mention@noreply.github.com 主题: Re: [jwchoi384/Gaussian_YOLOv3] train loss (#47) @ChunmianLin Probably because of the variance. I recommend you to train the model using darknet53.conv.74. Please see this: #16 (comment) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe. It's not large. If you use the pre-trained weight file ("darknet53.conv.74") and BDD cfg file which I uploaded, the model can train well. If not, try to increase the burn-in value in the cfg file or decrease the learning rate. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.