taokong/RON

What is the normal train loss

Opened this issue · 1 comments

zjuDebug commented

hi, @taokong ,great work!
I want to re-implement your experiment, and I have three small issues?

  1. What is the normal train loss when the model converge well?
    mark: When I train ,the loss always 4.0+ before 8w iters, and I test at 8w Iters only got mAp=0.54(train data is 2007trainval + 2013trainval)
  2. I also notice that I can do test in a pascal Titanx , but got CUDNN_STATUS_SUCCESS (8 vs. 0) CUDNN_STATUS_EXECUTION_FAILED error in a TitanX.
  3. Training needs much time, can you give me some advices?
    Thanks in advance!
HiKapok commented

hello, i also want to re-implement this work recently, but the loss always went towards nan after several steps. after carefully check i found the objectness proir(after softmax) will all be 0. did you ever got into problems like this? BTW, what's your learning rate setting?