Problem about Energy Based Unknown Identifier
Closed this issue · 12 comments
Hello @JosephKJ , thanks for your source code.
I am puzzled by how the Energy Based Unknown Identifier mentioned in the paper is implemented in the code.
As described in the paper, the energy function is used to replace the classification head of Faster R-CNN (I think), but the classification head I see in the code is a Linear module named self.cls_score in the FastRCNNOutputLayers class. Thanks!
Hi @kendyChina, as explained in the paper, "The learned distributions can be used to label a prediction as unknown." (Sec 4.3 last line)
This is done here: https://github.com/JosephKJ/OWOD/blob/master/detectron2/evaluation/pascal_voc_evaluation.py#L94
Please let me know if this clarifies your doubt.
@JosephKJ
Thank you for your prompt response, I mistakenly thought EBUI was working during training.
I also have a question, why is EBUI used in the val.yaml file and not in the test.yaml file?
@kendyChina, this is because the energy distribution will be fit during val, saved to file, and then read from the file during test.
I noticed that the OWOD.SKIP_TRAINING_WHILE_EVAL in the val.yaml file is false. Does it mean that backpropagation occurs during the val phase?
Thats true, we just learn the weibull distribution parameters from the validation data. Please refer to the code below: https://github.com/JosephKJ/OWOD/blob/master/detectron2/engine/train_loop.py#L181
I'm still a little confused, will the parameters of the model (such as Backbone and RPN) be updated when this line of code is executed in the val phase?
OWOD/detectron2/engine/train_loop.py
Line 325 in 6eaa496
Kindly see this line where we skip training: https://github.com/JosephKJ/OWOD/blob/master/detectron2/engine/train_loop.py#L144
run_step will be called only in L147, and we continue before that.
Kindly reopen this if you have any further confusion. Thank you!
Thank you for your patient reply!
I noticed that in t1_val.yaml, you explicitly set SKIP_Training_WHILE_EVAL to False, which should not skip L147, causing the model's parameters to be updated during the val phase:
OWOD/configs/OWOD/t1/t1_val.yaml
Line 17 in 6eaa496
It's not clear if my understanding is wrong.
You are correct that model parameters to be updated during the val phase, but please note that this model is not used any further. The final model used for testing, is what we had without doing validation. See line 20 here.
This is done just to extract the feature vectors to fit the Weibull distribution. Hope it clarifies, if not, please revert back, I am happy to explain.
You are correct that model parameters to be updated during the val phase, but please note that this model is not used any further. The final model used for testing, is what we had without doing validation. See line 20 here.
This is done just to extract the feature vectors to fit the Weibull distribution. Hope it clarifies, if not, please revert back, I am happy to explain.
I think the model is still used. Because the model after validation was output to the same folder and there is only one model_final.pth. And during evaluation, it is used.