Why the trainning is very slow in my dateset
yushanshan05 opened this issue · 11 comments
When I use GTX1080 to train in VOC2007, It is very fast. It can iterate 100 times during 30 minites. But When I use GTX1080 to train in my dateset, It can iterate 30 times during the whole night. It seems something wrong. Do you know the reason? Thank you!
I think the problem may be at the image reading process, but not sure.
Thank you !I have find the problem.
As you thinking, when crop the image in _get_image_blob_augment function of lib\roi_data_layer\minibatch.py, the image reading may be in the loop and can't exit.
My datasets has some small objects and long-shaped objects，so it can't crop the object when the __C.TRAIN.CROPS = (1.0, 0.9, 0.8, 0.7, 0.6, 0.5, 0.4) and __C.TRAIN.GT_OVERLAP = 0.5 in config.py.
When I use other framewrok of detection,such as SSD, the ways of data augment is not include cropping. So，I want to know the augment of cropping can increase how many of the mAP. I don't find this in you paper.
Thank you!
Hi, @yushanshan05
I think you could modify the params according to your dataset. The cropping is very helpful for one stage object detector, we also mentioned the trick at our paper(4.2) .
Oh, I see. I will try to modify the params. Thanks!
@yushanshan05 When i train my datasets, it can not continue iterring. It stops in iteration 0. I don't know the crop_ratio how to change, Do you solve the problem?
@super-wcg I have solve the problem. There is __C.TRAIN.CROPS in lib\fast_rcnn\config.py.
But I find my main problem is the annotation files. In my xml, the started coordinate of bbox is 0，while the coordinate will be subtract 1 in lib\datasets\pascal_voc.py. So it caused the endless loop in the data augument.
@yushanshan05 ok ,thanks
@yushanshan05 @taokong I have a problem that how to produce the objectness prior? I don't understand the meaning in the paper. Can you help me answer it? Thanks.
@super-wcg The objectness prior is produced in RPN, which reflect the the anchors is background or foreground. So the objectness prior is only used to classify there is an onject or not. In the end, the objectness prior must be higher than a threshold, there will produce a label(not backgroud) which will participate the calculation of loss.
That's my understanding, I'm not sure it is right.
@yushanshan05 @taokong I want to show the objectness prior, what should I do?
@yushanshan05 @super-wcg Hi, When i training in KITTI, it can not continue iterring. It stops in iteration 600. Thanks for your help!