[SSD] Small object detection
Tsuihao opened this issue · 96 comments
Hi all,
I have a question regarding the configuration of SSD.
An interesting task for me is to fine-tuning the SSD_mobilenet_v1_coco_2017_11_17 with Bosch small traffic light dataset.
However, the default setting is to resize the image into 300 x 300 (image_resizer).
Here is the total loss during training.
It loss maintains around 6. (Please ignore the overlapping at 5000 steps, due to some re-launch trainign process.)
I think the trend of the total loss is okay.
However, when I stop around 12k and feed with the test dataset (around 90 images for a short try). There is nothing detected.
Personally, I have some doubts about this issue:
- Maybe the small traffic lights are too small for SSD?
- However, why the total loss curve displayed a correct "learning" process?
Can I simply change the config of image size into 512 x 512 or even larger value (1000 x 1000)?
Will this work correctly as well?
Regards,
Hao
Did you try taking 300x300 crops from the images?
You could try training it on smaller images and feed in overlapping crops of size 300x300 that tile the original image, which could be bigger. I was able to train it on 1000x600 images, and it worked on my test set which was also 1000x600. This might be slightly hard since your original set is not 300x300, but if instead you could form a dataset out of random crops of size 300x300 from your original set then maybe...
The images I am actually working with are around 12MP, and I am feeding in crops of size 1000x600. However, with 1000x600, SSD is struggling to learn the classes, but the localization error is very low.
Hi @oneTimePad,
Thanks for your reply.
I have thought about this approach too.
However, in this case, I need to take care of the annotation too right?
Did you first annotation all the images and then covert the annotations into the cropped corresponding image (with some python script I assume)?
Or you first crop them and then annotate manually on those 300x 300 images?
@Tsuihao you cropping already annotated images. SSD has issues with detecting small objects but Faster-RCNN much better at this.
Hi @Luonic,
Yes, I had successfully trained faster rcnn and obtained an accurate result.
As shown:
However, it is too slow for my use case.
That is why I want to try the fastest SSD mobilenet model :)
I have some concerns regarding the annotated information.
When you crop the annotated images, how did you "update" the information in the original annotation?
Let's say:
Original image 1280 x 720 and the annotated traffic light is :
boxes: {label: Green, occluded: false, x_max: 752.25, x_min: 749.0, y_max: 355.125, y_min: 345.125}
when you crop it into 300 x 300, the annotated image coordinate system need to be updated.
Did you manually re-annotate them or there is some crop image tool can help you do this?
Regards,
Hao
Ah, yes. Completely forgot about the annotation. In my case I have program that generates all of my training data, so I can easily change the training data image size (which will then change the annotations). However, yeah, you could write a program that converts the bounding box coordinates as you mentioned, but as mentioned I am still struggling with getting the classification accuracy up.
An idea I had, was to first train mobilenet base network, fine tuning from the checkpoint trained on the coco dataset or a classification checkpoint, to just classify small crops of the the objects of interest. In your case, crops of traffic lights classifying their color. Then go back to SSD and fine-tune the model from these weights trained to classify. I haven't tried this yet, but it might help mostly with the classification accuracy.
You mentioned mobilenet(s); have you tried a different base network?
Hi @oneTimePad,
Thanks for the reply.
So there is one way I could do is: crop the traffic light image and then re-annotate all the images
I was trying to avoid this since the manual crop and re-annotate will take few days I assume :p.
In my case, I also used the pre-trained SSD mobilenet on coco dataset and fine tuning with the traffic light dataset.
There are two assumptions I made (please correct me if I am wrong):
-
during the image_resize to 300 x 300, Tensorflow will also resize the annotation in "tf.record" data: In my case, it does not work just because the original images 1280 x 720 resize into 300 x 300, the small traffic light just nearly vanishes. I suspect that is the reason I could not have the correct result.
-
I assume that the release Tensorflow SSD mobilenet is under SSD300 architecture, not SSD500 architecture : And this is why I was trying to change the image_resizer into larger value (512 x 512); however, it still not worked.
Maybe the last way is really like what you say, crop and re-annotate everything. that will be a lot overhead.
If you want to train an SSD512 model, you need to start from scratch. The pre-trained model can only be fine-tuned as SSD300 model.
Hi @Tsuihao Did you successfully train the SSD model on small objects? If so how did you get around it?
My original images are 512x512 I am thinking about cropping them to 300x300 around the areas of interest and create the TFrecords file from the cropped ones. Would this be ok?
Hi @augre,
I have not tried it yet.
I am also thinking about the same approach as you described and will try it as long as I have time.
I am not sure how the performance will be of cropping training images.
Maybe you can share your experience later :)
Hi @Tsuihao
Could you share your trained model(faster-rcnn)?
And what framework did you use for training, caffe or tensorflow?
Thanks
@Tsuihao Any progress on this method ?
I'm having the same issue, do you have any interesting findings that you remember you could share ?
Thanks !
Try this paper
S3FD: Single Shot Scale-invariant Face Detector
https://arxiv.org/abs/1708.05237 They modified SSD OHEM and IOU criterion to be more sensitive to small object like faces
Hi, I'm interested in training ssd500 mobilenet from scratch, can someone give me some hints?
Thank you.
Hi @Tsuihao
I have a problem with ssd_mobilenet_v2_coco. My images are 600x600 size but with resizing in the config file 300x300. Is there any possibility to work 600x600 in this case? Do my training images have to be 300x300? How did you solved small object problem?
@sapjunior : Have you used the implementation on some application other than faces?
@jungchan1 sorry I could not provide my trained work. I was using TensorFlow
@cyberjoac Nope, I did not go further on this topic; however, I am still looking forward to see if anyone can share the experience in this community :)
@elifbykl 600X600 for me sounds acceptable to resize into 300x300; however, it also depends on the relative object size you are working on. Based on the above discussion, you training image will resize inito 300x 300 due to the fixed architecture SSD provided by Tensorflow. I am still not solving the small object detection with SSD yet.
I trained a model capable of recognizing 78 German traffic signs. I used Tensorflow's Object Detection API for the training. The model can recognize the characters at a signsof about 15 meters. Here you can download the model and try it out.
Model: http://eugen-lange.de/download/ssd-4-traffic-sign-detection-frozen_inpherence_graph-pb/
@Tsuihao i had a similar problem and i needed to slice the image into smaller tiles/crops. however i already labelled my dataset and i was not sure what size of tiles were suitable for training.
So i wrote a python script that slices the image in a giving size and recalculates the annotations for you in separate .xml files per tile/image it creates.
Here is the code, its far from perfect but i needed a quick solution.
https://github.com/julianklumpers/slice_image_with_annotations/blob/master/slice_image_with_annotations.py
It uses openCV rather then PIL because i tested both and openCV was much quicker with sliceing and saving the images.
It creates tiles with coordinates from the original image as a name, this way i can stich the image back together.
Feel free to adjust it to your needs. i will probably make a library some day
The function creates 2 rows and 2 columns. so if you have a image that is 1000x1000 and you need 500x500 tiles. you just put size=(2,2) 1000 / 2 = 500.
Hello @Tsuihao,
have you tried the stock SSD_mobilenet_v1_coco_2017_11_17 without training and see the result visually?
My situation is the performance from stock SSD_inception_v2_coco_2017_11_17 is better than my trained-with-kitti model on car detection.
I am still working on this and hopefully can get back to you ASAP.
Best,
Hi @willSapgreen,
Yes, I have tried to use the pure SSD_mobilenet_v1_coco_2017_11_17 to do the traffic light detection. And the result is better than my trained SSD with traffic light dataset.
However, this result can be foreseen due to the fact that SSD_mobilenet_v1_coco_2017_11_17 trained with the COCO dataset. In my case, I need a more details about the detected traffic lights e.g. red, green, yellow, red left, etc.
In your case, you wanted to detect car, I believed that car in the image is much bigger than the traffic light; therefore, you should not have the same issue (traffic light is too small) as mine.
I will suggest you to:
- Check your tensorboard report (see whether training result is good or bad)
- Change with different model e.g. faster_rcnn (see whether your data/label is valid)
Regards,
Hey, I read that you struggled with resizing/cropping and then labeling again. I had the same problem so I made some scripts that I am trying to turn into a library. Why don't you check them https://github.com/lozuwa/impy
There is a method called reduceDatasetByRois() that takes in an offset and produces images of size (offset)X(offset) which contain the annotations of the original image.
I'm finding several problems in obtaining a good detection on small objects.
My images are 640x480 and the objects size are typically around 70x35 - 120x60.
I'm using the typical ssd_mobilenet config file, and I train from ssd_mobilenet_v2 pretrained model. I'm interested in a good accuracy with a great speed, so I need SSD architecture. Maybe is better to move to SSD inception v2? Or can I change some parameters, like anchors and fixed_shape_resizer (but... how?)
Thank you for any advice,
@eumicro how did you edit the config file to obtain that good detection?
Hi, i have a problem related with this, but it's a little different. I want to train a model to detect my hand, yes only one class and run the model on my phone. But the speed is a little slow ,about 400ms. I want to resize the image to smaller size like 100*100, the speed is much fast, but the presicion is very bad. I guess i need to train the ssd from scratch, is that right ? @izzrak
OK i will try 224*224
I have same problem with detecting small objects, my input 660x420 and the objects are about 25x35. I consider my objects medium size but SSD mobilenet v1 gives low accuracy and the training time is long. I did try to make my input 660x660 (width:heigh = 1:1) as recommended by @oneTimePad to see how the resizing step to 300x300 of SSD make any improvement but the answer is yes, but not much.
@AliceDinh, for long training time, what do you mean? How many steps? Which learning rate? Do you change anchors values?
- Training time is long, means to get loss~=1.0, the numbers of step are more than 200K. (With FasterRCNN, after 2K steps I get loss ~=0.02)
- Where to check the learning rate? Is that from the Tensorboard? I trained on server without Internet so I could not launch the Tensorboard from there.
- Change the anchors values? What specific values I should change?
Learning rate is defined inside the configuration. If you haven't change or edit anything inside the config file, it would be like below for SSD Mobilenet v1
learning_rate: { exponential_decay_learning_rate { initial_learning_rate: 0.0002 decay_steps: 800720 decay_factor: 0.95 }
Can you please share some experiences over training with SSD Mobilenet v1?
In my case, my object size varies from let's say 6 X 6 pixel to 20 X 20 pixel. My train and test image size is 200 X 200 pixel. Each of the images contains on average around 40 objects. I used (for now) 300 images as train image and 50 images as test image. In the last week, I trained till 77K+ steps, but the total loss didn't go under 3.5. In result of that, I didn't got much better detection.
Should I pay more attention on to increase the number of train images or to let it train more steps with same images to get good detection?
I would like to know, did you get better detection after being trained more than 200K steps??
My original image is bigger than 1000 X 1000 pixel, so I cropped these to 200 X 200 pixel, then labelled it with a tool and trained. After training, I observed it detects comparatively good in 200 X 200 train images than the 1000 X 1000 original images. Could you please mention some logical reasons behind that?
Thanks in advance.
@ashraful100
my training image: 660x416, 16-19 objects with size 25x35, I consider my objects medium size, not as small as yours. I mostly used default parameters of SSD mobilenet V1 to train. I modified only the PATH_TOBE_CONFIGURE, the number of class. After 15K steps, the total loss dropped to 0.8 but stay there forever. The model can detect most of the objects, but not all as my expectation. However, whatever it detected, it labeled correctly.
I am training with other input-sizes to see any differences coz the resizing step of SSD matters to others. I'll let you know the result later on.
@Tsuihao @Luonic
Hi Hao Tsui and Luonic
I am trying train on a dataset with large image (2040X1563) containing small object too (~25X25).
I am trying to use fast -rcnn (using the config file faster_rcnn_inception_v2_pets.config). I am having trouble to detect anything. Would you guys be able to share some experience on how you train your dataset with faster-rcnn for this kind of dataset? (speed is not my concern here).
I am very new to this so any suggestion is very appreciated. I think my problem currently is the RPN doesn't give any proposed box ( I see none in tensor board).
Note: If I crop the images down to ~600X400, its seem it does work, but I am wondering is it just impossible or I need to do some configuration changes.
I am also very confused about the
height_stride: 16
width_stride: 16
in first_stage_anchor_generator, from what I read, I should change them to a smaller value (since mu object is only around 25x25 insize).
But when I change them to 8/4, I do not see any improvement when I did not crop the image. when I use dataset with cropped image, oth original 16/16 give the best result.
Hi guys, I've successfully finetuned a pre-trained ssd-mobilenet with some tips I read in this conversation. I still have to try if the same method works for really small objects too (<20 pixels).
First, my input image was 1920x1080 so I decided to take crops of 400x400 to train the neural network. I don't think it is important how big the crops you use for training are, as long as you don't lose to much detail and the have the same aspect ratio of the input volume of the model you are finetuning (300x300 in my case). I did not change any hyperparameter.
After that, to perform the detection, I split the image (which can be of whatever size) in square crops. I've tried with multiple sizes but it seems that taking 6-8 crops is enough for me. If I try to predict using the whole image (without cropping), it detects some objects but the result is less accurate, however, it is much faster (300ms vs 800ms CPU only).
So, basically, it is a trade-off here. If I have time, I would like to train the model using distorted images (I mean, with a different aspect ratio of the input volume), to see if I can detect objects without cropping.
You can see the result here if you want: https://github.com/fdiazgon/cone-detector-tf
I would try to comment again if I get some interesting results.
PD: really interesting conversation, by the way.
i'm working on similar models but i'm quiet new. Could you elaborate when you say that you take crops to train the network.
I'm not sure to understand how it helps to identify small objects. By using cropped images, it is like a zoom in right? so i understand it would reduce the training losses but during prediction, won't the problem be the same for tiny objects?
Also for the cropping, what is good strategy? crop directly the bounding boxes, or center image cropping or random area?
thanks for the explanations!
By using cropped images, it is like a zoom in right?
Yes. The thing is, you have a 1280*720 image with an object of size 10x10 in it. But, the input volume of the NN you are finetuning is of size 300x300, so the NN has to squeeze the image before processing it and, consequently, the object you are trying to detect is no more 10x10, it is much smaller and maybe impossible to detect.
but during prediction, won't the problem be the same for tiny objects?
No if you take crops during prediction (at the expense of increasing the time to detect).
Also for the cropping, what is good strategy?
I 'scanned' the whole image. Make the crops overlap a few pixels in case the crop cuts an object in half. I did this for an image of 1280x720 with crops of 300x300, and I needed to take eight crops (you can increase the size of the crop).
ok i see! did you use special routines to make the cropping task or internal tensorflow methods? i saw some were available in TF apparently.
For prediction, if you do cropping too, i understand it will take more time for sure! still you preferred doing cropping with ssd mobilenet than using faster rcnn and a single image prediction? it still save processing time using mobilenet+multiple cropped images?
it still save processing time using mobilenet+multiple cropped images?
That's a good question. I haven't tried. The problem that I have is that I need the smallest and least resource-consuming network possible and, from a quick read, mobilenet-ssd seemed to be the best choice (maybe I am wrong). But it would be awesome if I could avoid cropping.
Hi guys, I found that different input image sizes donot affect the accuracy much, considered similar (for my dataset). I highly recommend SSD-FPN which improves the small object detection significantly
@AliceDinh thank you for the insight, I am looking to try SSD-FPN. Please could you explain further what your mean by
SSD anchor generator by somehow creates crops automatically and randomly?
@david-macleod Let's try SSD-FPN, training with TensorFlow Object Detection API version 2 (which replace train.py by model_main.py), you will see the improvement.
About the cropping, I found that for my data which is quite sensitive, I will try with other data then post here later on.
Hi @Tsuihao
In your result that you posted a while ago:
How to display the time in milisecond and the total detected objects as you did? Please show me! Thanks
@AliceDinh Did you try the training with SSD-FPN.If yes did you find any improvement over the normal SSD
I was Training with SSD_resnet_fpn on Bosch Small TL dataset and hit this issue during Training.My losses just spiked up at step number 6K.
Should I continue Training,My regularization loss does not come down.From the original config file i just modified 2 Things:
Batch size 32 instaed of 64
Warmup steps/total steps 800/18000 instead of 2000/25000
@eumicro what model and how did you fine-tune the model to get accurate prediction?
@eumicro what model and how did you fine-tune the model to get accurate prediction?
Hi, sorry my English is not that good. I described how I fine tuned and trained the SSD MobileNet here (only in German, sorry): http://eugen-lange.de/german-traffic-sign-detection/
the main "tuning steps" are:
- generated my own data set (see my homepage for more details), I think it was the most important "step" ^^...
- removed 2 first layers from the MobileNet
- used grayscale pictures
@giridhar13
SSD-FPN that I tried is SSD Mobilenet V1 FPN and the accuracy is very good:
Comparing to SSD Mobilenet V1, I got total loss arround 0.8 and stay there forever
What sizes of imagery is your data? the problem with SSD is it doesn't work for large images.
660x416 is my image size, object size is about 25x35
@AliceDinh doesn't SSD FPN require fixed shape image resizing? Do you pad your images?
I've tried 640x640 SSD FPN and got poor results.
I got good result without padding
@AliceDinh thank you for the insight. What do you suggest i use in terms of model arch- from your experience:
Image size: 1200x854
Object size: 40x40
So if i downsize my images, object size will be really small.
I tried taking 256x256 patches but accuracy was not good. How can i train SSD FPN with large imagery?
Sorry @aysark, I am not sure about your situation. I think trying to train with different resolutions, image sizes to see the accuracy of SSD FPN model, then decided the size later on.
@aysark Try the following：
image_resizer {
fixed_shape_resizer {
height: 854
width: 1200
}
}
Hi everybody,
I'm trying to reproduce the SSD_mobilenet_V1 results, but I'm having some troubles.
I am using TensorFlow Object-detection API
I use the config file that was uploaded with the frozen model, so i expected to get similar results to what was presented on the website.
The resizing that is being done is to 300x300, yet my goal is to train on bigger size (I plan to do so whenever i will be sure that everything is running well on the basic example).
It looks like some of you succeeded in the task , so I'll appreciate any help/insights/code...
BTW, are you working with the legacy scripts (Train.py / eval.py) of the model_main.py script which is newer? i get different results for both of them...
What happens if we decrease the min_score_thresh? We can get more detections by lowering this threshold though.
https://github.com/tensorflow/models/blob/master/research/object_detection/utils/visualization_utils.py#L354
Yeah,
the visualization will essentially allow to draw any boxes with confidence score above or equal to your min_score_thresh on to your image. A good min_threshold is usually around 0.5 but it also depends on what you're trying to see.
Maybe the last way is really like what you say, crop and re-annotate everything. that will be a lot overhead.
I might have misunderstood. Why is cropping and transforming the annotation into the cropped images so difficult? It should be just a few lines of python codes.
Hi @NightFury10497 ,
Lately I've worked with google object detection API and had my own struggles with it, something that might help you in the training process:
The part of the train_config -
rms_prop_optimizer { learning_rate { exponential_decay_learning_rate { initial_learning_rate: 0.004000000189989805 decay_steps: 800720 decay_factor: 0.949999988079071 } }
- initial_learning_rate - for me this learning rate was too high, and in addition look at the decay_steps, it is 800720 which actually says that it stays the same for 800k steps and only than decreases,
i recommend to lower decay_steps to 1000 / 2000 / 5000 (something of this order).
You can also start with the given learning rate to see how it goes and if the model doesn't seem to converge, try to lower it to 0.001
Let me summarize this discussion... please correct me if i am wrong ..:-)
If i am training my own custom dataset using legacy train.py the best strategy is to crop a bigger resolution image into smaller chunks to have many images.
Or if i am using the version 2 model_main.py that process is automatically taken care of ?
Please reply.
@tmyapple @Tsuihao @oneTimePad @Luonic @izzrak @augre @fdiazgon
do i need to resize and create the crops of objects detected as per the above comments as well?
or is it fine in detecting only one single class using Object Detection API (tf) of MobilenetV2 ssd?
by changing the Learning rate will it work fine?
- Changing the learning may help, because the one exists now in the pipeline.config is probably not what you need and it the one that was used for the training that was done from scratch.
- Regarding crops... - I guess it depends on the image resolution you have... I would try first to continue as you did - meaning work with 512x512 Fixed resizer and compare it to results you get on 300x300.
3.More thoughts: take in mind the sized of the objects to intend to detect.
The model is trained to have 6 output branches with 6 anchors per pixel (except for the first branch which has 3 anchors) -
anchor_generator { ssd_anchor_generator { num_layers: 6 min_scale: 0.2 max_scale: 0.95 aspect_ratios: 1.0 aspect_ratios: 2.0 aspect_ratios: 0.5 aspect_ratios: 3.0 aspect_ratios: 0.3333 }
- do you really need these 6 output branches?
- i guess you can even remove the two last aspect ratios (3:1, 1:3) - because face tends to be more "boxy" -
-
after you will see that the model starts to learn something - this it another thing you may want to tune . min scale defines the anchors scale relative to the image an the first output branch, max scale --> anchors scale relative to the image at the last layer (it is interpolated in all the output layers inbetween)
min_scale: 0.2 max_scale: 0.95
reducing the scales may help to find smaller objects...
@tmyapple @Tsuihao @oneTimePad @Luonic @izzrak @augre @fdiazgon`
1st I trained mobilenetV2 ssd on Wider Face Dataset
2nd Annotated them using yolo then after converting it to xml(voc) verified each in labelImg
3rd I created tfrecords trained using the pipeline
#NOTE the size of the images are 300300 with the annotated objects (with the fixed image resizer 300300) and with the same learning rates you mentioned before i changed them in the pipeline as well as decay_steps to 5000
The ISSUE i am facing is below:
Model is detecting the faces but in a very small manner that that bounding boxes detected are way too small.
- Changing the learning may help, because the one exists now in the pipeline.config is probably not what you need and it the one that was used for the training that was done from scratch.
- Regarding crops... - I guess it depends on the image resolution you have... I would try first to continue as you did - meaning work with 512x512 Fixed resizer and compare it to results you get on 300x300.
3.More thoughts: take in mind the sized of the objects to intend to detect.
The model is trained to have 6 output branches with 6 anchors per pixel (except for the first branch which has 3 anchors) -
anchor_generator { ssd_anchor_generator { num_layers: 6 min_scale: 0.2 max_scale: 0.95 aspect_ratios: 1.0 aspect_ratios: 2.0 aspect_ratios: 0.5 aspect_ratios: 3.0 aspect_ratios: 0.3333 }
- do you really need these 6 output branches?
- i guess you can even remove the two last aspect ratios (3:1, 1:3) - because face tends to be more "boxy" -
min_scale: 0.2 max_scale: 0.95
reducing the scales may help to find smaller objects...
######
Do you mean i should alter:
anchor_generator {
ssd_anchor_generator {
num_layers: 6
min_scale: 0.20000000298023224
max_scale: 0.949999988079071
aspect_ratios: 1.0
aspect_ratios: 2.0
aspect_ratios: 0.5
aspect_ratios: 3.0
aspect_ratios: 0.33329999446868896
}
}
to
anchor_generator {
ssd_anchor_generator {
num_layers: 6
min_scale: 0.2
max_scale: 0.95
}
}
- Did your loss function seemed to converge ?
- What tool do you use for visualization ? -- i'm not sure how you've plotted this image - but I recommend to open tensorboard (in case you didn't) - the events are written there periodically an you will get also some images from your validation set with their detections.
- At start - in order to find out everything works as expected it is a common practice to try overfit on one image - instead of one image you can just put the test.record path as your training also... it would help you to diagnose your work.
btw, i attach an example of the Tensorboard layout ---
- Changing the learning may help, because the one exists now in the pipeline.config is probably not what you need and it the one that was used for the training that was done from scratch.
- Regarding crops... - I guess it depends on the image resolution you have... I would try first to continue as you did - meaning work with 512x512 Fixed resizer and compare it to results you get on 300x300.
3.More thoughts: take in mind the sized of the objects to intend to detect.
The model is trained to have 6 output branches with 6 anchors per pixel (except for the first branch which has 3 anchors) -
anchor_generator { ssd_anchor_generator { num_layers: 6 min_scale: 0.2 max_scale: 0.95 aspect_ratios: 1.0 aspect_ratios: 2.0 aspect_ratios: 0.5 aspect_ratios: 3.0 aspect_ratios: 0.3333 }
- do you really need these 6 output branches?
- i guess you can even remove the two last aspect ratios (3:1, 1:3) - because face tends to be more "boxy" -
min_scale: 0.2 max_scale: 0.95
reducing the scales may help to find smaller objects...
######
Do you mean i should alter:
anchor_generator {
ssd_anchor_generator {
num_layers: 6
min_scale: 0.20000000298023224
max_scale: 0.949999988079071
aspect_ratios: 1.0
aspect_ratios: 2.0
aspect_ratios: 0.5
aspect_ratios: 3.0
aspect_ratios: 0.33329999446868896
}
}
to
anchor_generator {
ssd_anchor_generator {
num_layers: 6
min_scale: 0.2
max_scale: 0.95
}
}
you shouldn't remove all the anchors, you can try something like this:
anchor_generator {
ssd_anchor_generator {
num_layers: 6
min_scale: 0.2
max_scale: 0.95
aspect_ratios: 1.0
aspect_ratios: 2.0
aspect_ratios: 0.5
}
}
but if you ask me you should start with the basic and tune it from there later on..
@tmyapple @Tsuihao @oneTimePad @Luonic @izzrak @augre @fdiazgon
- Changing the learning may help, because the one exists now in the pipeline.config is probably not what you need and it the one that was used for the training that was done from scratch.
- Regarding crops... - I guess it depends on the image resolution you have... I would try first to continue as you did - meaning work with 512x512 Fixed resizer and compare it to results you get on 300x300.
3.More thoughts: take in mind the sized of the objects to intend to detect.
The model is trained to have 6 output branches with 6 anchors per pixel (except for the first branch which has 3 anchors) -
anchor_generator { ssd_anchor_generator { num_layers: 6 min_scale: 0.2 max_scale: 0.95 aspect_ratios: 1.0 aspect_ratios: 2.0 aspect_ratios: 0.5 aspect_ratios: 3.0 aspect_ratios: 0.3333 }
- do you really need these 6 output branches?
- i guess you can even remove the two last aspect ratios (3:1, 1:3) - because face tends to be more "boxy" -
min_scale: 0.2 max_scale: 0.95
reducing the scales may help to find smaller objects...
######
Do you mean i should alter:
anchor_generator {
ssd_anchor_generator {
num_layers: 6
min_scale: 0.20000000298023224
max_scale: 0.949999988079071
aspect_ratios: 1.0
aspect_ratios: 2.0
aspect_ratios: 0.5
aspect_ratios: 3.0
aspect_ratios: 0.33329999446868896
}
}
to
anchor_generator {
ssd_anchor_generator {
num_layers: 6
min_scale: 0.2
max_scale: 0.95
}
}
you shouldn't remove all the anchors, you can try something like this:
anchor_generator {
ssd_anchor_generator {
num_layers: 6
min_scale: 0.2
max_scale: 0.95
aspect_ratios: 1.0
aspect_ratios: 2.0
aspect_ratios: 0.5
}
}but if you ask me you should start with the basic and tune it from there later on..
Issue is not there in training again, Please specify what all changes i should do in the pipeline of MobilenetV2_ssd for images with 300*300 for detection of small object.
@oneTimePad , @izzrak .. do you guys have any idea about this...
Hey guys, A quick hijack of the post here. By now, (thanks to experiments by @AliceDinh ) we know that FPN as a feature extractor matched with SSD helps increase accuracy on small objects. But here is another issue that I'm facing.
Problem Statement: Objects very similar to each other with the distinguishing feature between them being very small.
Example:
-
Watches -> I trained regular Mobilenet SSD on one specific watch (LG Watch). It works great. But the problem is, it detects any watch. I cannot possibly train on all watch brands/types all over the world to avoid them during detection obviously.
-
Cars -> Attached below is a Chrysler car rear view. This is a 200 S. I have a dataset of the rear view of the car. Now problem is, the entire car rear looks same for all tiers. For example, the difference between the 200 S (in the pic) and 200 C would be.. the S and C in the badging on the car.
Ideas ->
-
I just had an idea reading this discussion here where I can do weird annotations. For example, first annotate the car to localize it from the environment. And then differentiate between cars using annotations on the character like 'S' or 'C'. This way SSD-FPN would help because the small objects like 'S' / 'C' are retained because of FPN and SSD in general can just handle the rear view of car from rest of the environment. Do you guys think this will help?
-
I have no clue on how to approach the problem with the watches though. Let's say I have 10 specific type of watch classes. How would I go about annotating this dataset and what kind of a model can be used with this. The watches are similar to each other except very minute changes in details.
For Idea-2, here's what I already know and have. I trained with vanilla Mobilenet-SSD and it didn't seem to help. My logical guess is because the object looks similar in more than 90% of the pixels, the annotations between the 2 objects is not different by much. But based on idea-1, if I instead of annotating the entire watch for detecting that one class, I just annotate much less area of the watch where the difference between the classes is high (meaning more than 60% of the pixels in the annotated region is different between 2 different watches), then it will do a better detection? Ofc, now it becomes a small object detection because the number of pixels will be small, hence using SSD-FPN. Would this help in any way?
Side Questions:
Apart from those questions above, a couple of questions which I always confuse myself with:
- I currently have around 1500 pictures for each watch class that I collected for my school project. Is this enough dataset per class or do I need more pictures? I have 10 classes that I'm working with. I do know, the amount data required is proportional to the architecture parameter count. I'm talking about SSD-FPN with resnet50 or mobilenet. Also, Faster-RCNN.
- For background class images, do I have to match the images per class (say 1500 per above question) or do I need more or do I just get by with a 100 images or so? Also, when we say background classes, can it be any images? Can I randomly pull data from other datasets and call it background class?
- Will retaining the aspect ratio of the dataset help? I collected the watch dataset with the image size at 2592x1944 (4:3) and I RESIZE it to 640x480 (4:3) as input image to the neural network. I don't want to use the high resolution because it uses a lot of memory to train and inference is slow and I'm looking for an alternate for cropping my image data. Resizing sounds like a default option otherwise? I'm assuming this is better than resizing it to a 1:1 aspect ratio because it preserves the integrity of the object compared to changing the aspect ratio? Or does it not matter of how the anchor boxes and basically how SSD works?
Thanks for the help.
@Deep-Sek
Isn't it a better idea to have some other tricks to distinguish between different types of those similar cars? for example, using OCR techniques to read the letters and decide whether it is a "C" series car or an "S" series car.
I had some experience classifying similar classes before though, e.g. different type of cars( different brand, year etc.) and different birds. It is indeed a hard problem, and I think you can have a look at paper in this domain, such as:
http://openaccess.thecvf.com/content_cvpr_2017/papers/Fu_Look_Closer_to_CVPR_2017_paper.pdf
@Ekko1992 I skipped OCR techniques all together because I thought since this is "OCR in the wild" where we don't control the environment, the performance would not be good. I'll give it a try asap and keep everyone updated on how it works out. Maybe I can do some affine transformations and control the text density and structure a bit.
Also, will take a look at the paper and try that too. Thanks a lot for the resources. I'll provide an update as soon as I can.
I did try this: http://vis-www.cs.umass.edu/bcnn/docs/bcnn_iccv15.pdf
Basically, took this network architecture idea as a feature extractor and replicated it using MobileNet with bilinear connection and then plugged in the regular SSD for detection network after. Can you tell me what you think of that paper? The idea sounds like it should give amazing results.
But not sure if I did have enough data to substantiate training this huge network with double the parameters. It just took way too long to converge. I'll probably re-attempt too at a later time after trying out your suggestions.
您好@oneTimePad，
谢谢回复。
所以我可以做的一种方法是：裁剪交通灯图像，然后重新注释
我试图避免这种情况的所有图像，因为手动裁剪和重新注释需要几天我假设：p。就我而言，我还在coco数据集上使用了预先训练过的SSD mobilenet，并使用交通灯数据集进行了微调。
我做了两个假设（如果我错了，请纠正我）：
- 在image_resize到300 x 300期间，Tensorflow还将调整“tf.record”数据中的注释：在我的情况下，它不起作用只是因为原始图像1280 x 720调整为300 x 300，小交通灯几乎消失。我怀疑这是我无法获得正确结果的原因。
- 我假设发布的Tensorflow SSD mobilenet属于SSD300架构，而不是SSD500架构：这就是为什么我试图将image_resizer更改为更大的值（512 x 512）; 然而，它仍然没有奏效。
也许最后一种方式真的像你说的那样，裁剪并重新注释一切。这将是一个很大的开销。
Even if the image is cropped and re-annotated during training, the image is still so large when detected that cropping seems to be of little use.
Hello,
I am also facing a problem of recognizing small objects on the image.
In my case I need to be able to detect multiple numbers (0-9) as well as tiny logos on the image. Let's say we have an advertisement billboard of a more or less standard shape which contains 3-4 lines of small logos with digits in front.
For example:
DHL - 1248265
UPS - 7623652
FedEx - 3726565
The real size of a billboard is pretty big, but we need to detect numbers from a distance, so the numbers would actually become small, although you could still easily recognize them on the phone screen.
I am wondering if the following approach would work with SSD mobilenet V1/V2 models:
I will create a dataset consisting of individual numbers, logos and the whole billboard.
Then we will detect the whole billboard at first. Since its pretty large relative to the image. After getting it's bounding box, I will crop the image based on that, maybe enlarge it a bit and then feed the result back to the model to detect logos and numbers
So we would actually run the detector twice on the same image. I assume this would be anyway faster than running ResNet or Faster-RCNN on mobile device.
Does anyone know if that would make any improvements for detecting process with SSD mobilenet?
Tensorflow is crap and below-par piece of shitty library written for the benefit of Google cloud.
Thank you.
For those who are visiting... let me break down the entire story for you.
comment the following in your pipeline.config file. There are bugs depending upon which version of tensorflow your using that is why if your working on new version this problem should not come in your way. For the old version:
#data_augmentation_options {
#random_horizontal_flip {
#}
#}
#data_augmentation_options {
#ssd_random_crop {
#}
#}
@dexception Which version of tensorflow you're reffering to as the old version? And since which version this bug is fixed?
Thanks.
OK i will try 224224
@hengshanji Did training with 224224 MobilenetSSD V2 solve the issue?
Here is something I tried that I haven't seen anyone else try here.
My problem is my camera input is 1280x960 and I'm looking for small labels. To keep the height from becoming to distorted when the image is fit into the 300x300 input space I kept the aspect ratio but fit the image into the same linear space. e.g.
300 * 300 = 90e3,
Y = X * 960/1280,
90e3=X * X * 960/1280 = X^2 * 960/1280,
X = sqrt(90e3 * 1280/960) = 346.41,
Y = 259.81._
Rounding X and Y to integers to keep X * Y<90e3 with minimal wasted bytes finds the optimal new size to be 346x260 with 40 * 3 wasted bytes. img.shape = (260,346,3)
image_resizer {
fixed_shape_resizer {
height: 260
width: 346
}
}
Retraining a SSD with inception v2, I should keep the meat of what the model has learned with minimal trouble.
This converged to a loss of 1.8 after 86000 steps.
@tcrockett Preserving aspect ratio should not really affect your training in anyway. If your camera input is 4:3 (1280x960) and you resize your input image to 1:1 (300x300) and you're always consistent with this. Then it shouldn't matter. For example, after you train your network by resizing your pics from 4:3 to 1:1.. as long as you do the same during inference time (post training) and convert your camera input from 4:3 to 1:1, the distortion that you do on the image is consistent and the neural network doesn't care much about that. I can see that the network having trouble with detections if you used a different aspect ratio to capture raw data (before resizing) and then resized that to 1:1. But preserving aspect ratio doesn't really do anything.
In SSD, the prior boxes have different aspect ratios which is why the aspect ratio of the input image doesn't really matter because the prior boxes will pick up the aspect ratio variation of the objects.
left is 300x300, right is 260x346
Without aspect ratio adaption the width of the logo will be represented in the 300x300 space by fewer pixels reducing the horizontal detail.
Hallo Tsuihao,
is the loss in your graph for the traffic light detection in percent? Or I must multiply the values with 100?
My problem is the same, because I get values between 1 and 2.
Thanks for your answer
Sigg
Can anyone suggest something about Retraining a Object Detection model. i.e -
Suppose i train tensorflow faster Rcnn_inception on any custom data having 10 classes like ball, bottle, Coca etc.. and its performing quite well. Now later i got some new data of 10 more classes like Paperboat, Thums up etc and I want my model to trained on these too. Is there any method so that i can retrain my generated model for these 10 new classes too to upgrade it for 20 classes, rather starting training from scratch.
Hi guys, here are my 2 cents: in my scenario I want to detect UI elements (buttons, checkbox, etc) from screenshots of 800x800 using ssd_mobile_net_v2. The dimensions of the objects range from 80px to 400px.
-
The input is are 800x800 images and the preprocessing step is fixed_shape_resizer set on 800x800.
-
I found extremely useful to modify the ssd_anchor_generator min_scale and max_scale based on the dimensions of the objects (0.1 and 0.5).
-
Another improvement was to modify the file ssd_mobilenet_v2_feature_extractor.py to use layer_15/expansion_output as first feature map and the rest are all new layers (no more layer_19).
Lastly in my case I also have the need for an augmentation that creates an effect of zoom-in zoom-out for simulating projects at different scales and positions. For this I modify the preprocessor as in the pull request #8043 and used the configuration
data_augmentation_options {
ssd_random_crop_pad_fixed_aspect_ratio{
aspect_ratio: 1.0
min_padded_size_ratio: [0.5,0.5]
max_padded_size_ratio: [2, 2]
operations {
random_coef: 0.5
overlap_thresh: 1.0
clip_boxes: false
min_object_covered: 1.0
min_aspect_ratio: 0.25
max_aspect_ratio: 4
min_area: 0.1
max_area: 1.0
}
}
}
On Stack Overflow someone explained how to test the augmentation. This is the adapted script to visualize the effect of the above operation
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function
import functools
import os
import cv2
from absl.testing import parameterized
import numpy as np
import tensorflow as tf
from scipy.misc import imsave, imread
from object_detection import inputs
from object_detection.core import preprocessor
from object_detection.core import standard_fields as fields
from object_detection.utils import config_util
from object_detection.utils import test_case
FLAGS = tf.flags.FLAGS
tf.disable_eager_execution()
class DataAugmentationFnTest(test_case.TestCase):
def test_apply_image_and_box_augmentation(self):
# Put here your augmentation
data_augmentation_options = [
(preprocessor.ssd_random_crop_pad_fixed_aspect_ratio, {
'min_object_covered': [1.0],
'aspect_ratio': 1.0,
'aspect_ratio_range': [(0.25, 4)],
'area_range': [(0.1, 1.0)],
'overlap_thresh': [0.999999],
'clip_boxes': [False],
'random_coef': [0.0],
'min_padded_size_ratio': (0.25, 0.25),
'max_padded_size_ratio': (2, 2)})
]
data_augmentation_fn = functools.partial(
inputs.augment_input_data,
data_augmentation_options=data_augmentation_options)
tensor_dict = {
fields.InputDataFields.image:
# lena.png is the image reference
tf.constant(imread('lena.png').astype(np.float32)),
fields.InputDataFields.groundtruth_boxes:
# just a ground truth box element in normalized coordinates [y1,x1,y2,x2]
tf.constant(np.array([[ 0.5, 0.5, 0.53 , 0.53]], np.float32)),
fields.InputDataFields.groundtruth_classes:
tf.constant(np.array([1.0], np.float32))
}
# This is the size of the resizer
final_image_size= (800, 800)
augmented_tensor_dict = data_augmentation_fn(tensor_dict=tensor_dict)
with self.session() as sess:
for x in range(100):
augmented_tensor_dict_out = sess.run(augmented_tensor_dict)
final_image_shape=augmented_tensor_dict_out[fields.InputDataFields.image].shape
print("Final Shape "+ str(x) + ": ", final_image_shape)
print("Final Boxes "+ str(x) + ": ", augmented_tensor_dict_out[fields.InputDataFields.groundtruth_boxes])
final_image=augmented_tensor_dict_out[fields.InputDataFields.image]
if augmented_tensor_dict_out[fields.InputDataFields.groundtruth_boxes].shape[0] > 0:
point_x=augmented_tensor_dict_out[fields.InputDataFields.groundtruth_boxes][0][1]
point_y=augmented_tensor_dict_out[fields.InputDataFields.groundtruth_boxes][0][0]
point_x2=augmented_tensor_dict_out[fields.InputDataFields.groundtruth_boxes][0][3]
point_y2=augmented_tensor_dict_out[fields.InputDataFields.groundtruth_boxes][0][2]
final_image = cv2.rectangle(final_image, (int(point_x*final_image_shape[1]),int(point_y*final_image_shape[0])), (int(point_x2*final_image_shape[1]),int(point_y2*final_image_shape[0])), (255,0,0), 2)
else:
print("Boxes is empty")
imsave('test/lena_out'+str(x)+'.jpeg',cv2.resize(final_image,final_image_size))
if __name__ == '__main__':
tf.test.main()
@eumicro what model and how did you fine-tune the model to get accurate prediction?
Hi, sorry my English is not that good. I described how I fine tuned and trained the SSD MobileNet here (only in German, sorry): http://eugen-lange.de/german-traffic-sign-detection/
the main "tuning steps" are:
- generated my own data set (see my homepage for more details), I think it was the most important "step" ^^...
- removed 2 first layers from the MobileNet
- used grayscale pictures
from which file you removed first two layers ?
@sky5media have you been able to solve your issue? If yes, how? I also try to use object detection for OCR but I have 14 classes and can only detect 9 of them with model_main. Train.py loss does something weird doing great for the first epoch and then goes expotentially to billioons.
Quite a same issue i am facing with ssd_mobilenet_v2_coco_2018_03_29 pre-trained model. Localisation loss is fluctuating and loss is quite high even after 50K steps. Trying to train model with 7 classes (Pedestrian;Truck;Car;Van;Bus;MotorBike;Bicycle). I know the same classes are already available in the pre-trained model but i am feeding my own images. Any idea whats wrong?
@sainisanjay Your learning rate(LR) is too high I guess. Try setting a scheduled decay of LR.
Check whether your objects are correctly annotated and easy to disntinguish from the background.
Check the exif orientation of your pictures as well.
@synergy178, I have following parameters:
initial_learning_rate: 0.001
decay_steps: 40000
decay_factor: 0.95
I am not really sure how to check the the exif orientation of your pictures. But i have visualised my TF records with tfrecord-viewer. This tools gives my same results as original annotation. As can be seen attached image.
Further, i have checked the image orientation with following two options. Both has gave me same orientation:
Option 1: Example from exif
import matplotlib.pyplot as plt
import image_to_numpy
img = image_to_numpy.load_image_file("my_file.jpg")
plt.imshow(img)
plt.show()
Option 2: Normal matplotlib lib.
from matplotlib import image
from matplotlib import pyplot
image = image.imread("my_file.jpg")
print(image.dtype)
print(image.shape)
pyplot.imshow(image)
pyplot.show()
Since both libraries are giving same orientation so i assumed orientation of images are correct. Problem is something else?
@synergy178 unfortunately no, I couldn't solve it.
Here is something I tried that I haven't seen anyone else try here.
My problem is my camera input is 1280x960 and I'm looking for small labels. To keep the height from becoming to distorted when the image is fit into the 300x300 input space I kept the aspect ratio but fit the image into the same linear space. e.g.300 * 300 = 90e3,
Y = X * 960/1280,
90e3=X * X * 960/1280 = X^2 * 960/1280,
X = sqrt(90e3 * 1280/960) = 346.41,
Y = 259.81._Rounding X and Y to integers to keep X * Y<90e3 with minimal wasted bytes finds the optimal new size to be 346x260 with 40 * 3 wasted bytes. img.shape = (260,346,3)
image_resizer { fixed_shape_resizer { height: 260 width: 346 } }Retraining a SSD with inception v2, I should keep the meat of what the model has learned with minimal trouble.
This converged to a loss of 1.8 after 86000 steps.
It is not a good idea to have different height and width for the image resizer in case you want to convert it to uff to run on edge devices. Because you need to manually put the ratios in the uff config file. and the function that is used to calculate the ratios take only one variable as input. so for 300x300, the ratios would be calculated for 300. but for your case 260x346, if you input either 260 or 346, the resulting bounding boxes generated by the tensorrt model in the edge device will be different than the ones generated by the tensorflow model in your pc.
@preronamajumder Did you use transfer learning or you train the model from scratch? I believe, If you change the height and width you can not use the pre-trained model (300x300) for weight initialization.
https://github.com/DetectionTeamUCAS/FPN_Tensorflow
This project based Faster rcnn + FPN, which is accurate to detect small objects. But I was not able to deploy the project on Openvino, sinice the merge function in "fusion_two_layer" is limited on Openvino.
Hi all.
I'm trying to train an SSD on a custom barcode detection task. The issue is that the dataset images are all different sizes and keep aspect ratio resizer doesn't seem to be working with ssd resnet 50. Is it required for the input images to be the same sizes in 1:1 ratio as in the fixed resizer?
@preronamajumder Did you use transfer learning or you train the model from scratch? I believe, If you change the height and width you can not use the pre-trained model (300x300) for weight initialization.
I used transfer learning with ssd_mobilenet_v2_coco. fixed image resizer can be changed. But I started setting it to 300x300.
Hi all.
I'm trying to train an SSD on a custom barcode detection task. The issue is that the dataset images are all different sizes and keep aspect ratio resizer doesn't seem to be working with ssd resnet 50. Is it required for the input images to be the same sizes in 1:1 ratio as in the fixed resizer?
Why dont you try to pad the images? It will maintain the aspect ratio of the ground truth boxes and will also give the appropriate size required by the detection model.
Hi @Luonic,
Yes, I had successfully trained faster rcnn and obtained an accurate result.
As shown:
However, it is too slow for my use case.
That is why I want to try the fastest SSD mobilenet model :)I have some concerns regarding the annotated information.
When you crop the annotated images, how did you "update" the information in the original annotation?
Let's say:
Original image 1280 x 720 and the annotated traffic light is :
boxes: {label: Green, occluded: false, x_max: 752.25, x_min: 749.0, y_max: 355.125, y_min: 345.125}when you crop it into 300 x 300, the annotated image coordinate system need to be updated.
Did you manually re-annotate them or there is some crop image tool can help you do this?Regards,
Hao
Hey @Tsuihao could you share the repo you use for the faster-RCNN please? Thanks in advance!
I'm finding several problems in obtaining a good detection on small objects. My images are 640x480 and the objects size are typically around 70x35 - 120x60.
I'm using the typical ssd_mobilenet config file, and I train from ssd_mobilenet_v2 pretrained model. I'm interested in a good accuracy with a great speed, so I need SSD architecture. Maybe is better to move to SSD inception v2? Or can I change some parameters, like anchors and fixed_shape_resizer (but... how?)
Thank you for any advice,
@eumicro how did you edit the config file to obtain that good detection?
@darkdrake88 @sainisanjay
He removed the first two layers of the architecture in my opinion.
I thought a bit about it and I'm sure these layers are excluded:
scientific paper (https://arxiv.org/abs/1801.04381):
224x244x3 conv2d, output_channels=32, stride=2
112x112x3 bottleneck, output_channels=16, stride=1
TF OD API (https://github.com/tensorflow/models/blob/master/research/slim/nets/mobilenet/mobilenet_v2.py):
op(slim.conv2d, stride=2, num_outputs=32, kernel_size=[3, 3])
op(ops.expanded_conv, expansion_size=expand_input(1, divisible_by=1),num_outputs=16)Questions about it:
- Is it necessary to rerun the protoc command (refer to the TensorFlow Installation guide) or just comment these two lines an start training?
- Why this change increase the ability of the model do detect smaller objects, which are more far away?