[Resolved] Discrepancy in t*_ft.txt

Question

[Resolved] Discrepancy in t*_ft.txt

Closed this issue 3 years ago · 12 comments

Thanks for your great work !

However, I run the released code but the performance is behind the reported performance in the paper. The first row is the reported performance, the second is my performance. I didn't change any code.

Meanwhile, I find the data split in your logs is different from the released data split in the google drive.
For task 2 finetune, the first image is data split in your logs, the second is that in the google drive. They are apparently different.

Looking forwards your reply.

Answer 1 · 2021-05-11T05:09:06.000Z

Dear @PeizeSun : I cannot thank you enough for noticing this discrepancy. Apparently, t2_ft.txt was a version that I had used to run an ablation. I have updated the same now. Can you please try now? Unfortunately, I dont have the exact same file with me (the machine on which I ran these is now decommissioned, and I had missed to backup it.), but the current files would suffice.

Please do try it out and let me know.

Thanks,
Joseph

Answer 2 · 2021-05-11T06:15:41.000Z

@JosephKJ i cannot find t1_ft.txt. but in run.sh ,t1_ft is nessary

Answer 3 · 2021-05-11T06:19:06.000Z

There is no fine-tuning step required for Task 1. There is nothing like forgetting there. We are learning all the VOC classes in Task 1, and learning to identify unknowns. Please let me know why you think that t1_ft is necessary.

Answer 4 · 2021-05-11T06:33:19.000Z

Ah, you might be referring to the old version of run.sh, @LoveIsAGame. Kindly pull the latest tip.

Answer 5 · 2021-05-11T06:37:35.000Z

Yes, I didn't notice you made a change to run.sh two hours ago. Thanks @JosephKJ

Answer 6 · 2021-05-12T13:02:07.000Z

@JosephKJ In Task1, I am using 1 GPU, so I set BASE_LR to 0.0025 and here is my result.I don't know what went wrong.
The configuration files are executed in the order t1_train t1_val t1_test.

Answer 7 · 2021-05-12T14:55:34.000Z

Hi @LoveIsAGame : You need to run it for 8x more iterations as you are running on a single GPU. Please refer to the official Detectron 2 spec. This issue might also be helpful.

Answer 8 · 2021-05-17T05:00:09.000Z

Hi, the new t2_ft.txt is still different from that used in models_backup/t2_ft.The prior contains 1743 images and the latter contains 386. Could you provide the t2_ft.txt which is used to train the model in models_backup? Thx

Answer 9 · 2021-05-17T05:10:30.000Z

Can you copy over <owod_folder>/datasets/OWOD_imagesets/ to <owod_folder>/datasets/VOC2007/ImageSets/Main and try? Please ensure that you are on the latest commit of the code. Thanks!

Answer 10 · 2021-05-17T05:22:56.000Z

Can you copy over <owod_folder>/datasets/OWOD_imagesets/ to <owod_folder>/datasets/VOC2007/ImageSets/Main and try? Please ensure that you are on the latest commit of the code. Thanks!
Yes, I have updated the latest t2_ft.txt which has 1743 images, is that right? but the log.txt of t2_ft in your google drive owod_backup shows there are only 386 images.

Answer 11 · 2021-05-17T05:28:23.000Z

There is some confusion here:

t2_ft.txt in owod_backup contains 1665 images and not 386 (Please refer to the screenshot in the first comment in this thread by PeizeSun)
I believe 386 is what you got after running the previous version of t2_ft.txt
If you run the latest t2_ft.txt, you will get 1743 images. Please find my log below:

Answer 12 · 2021-05-21T16:42:47.000Z

Kindly reopen if the issue persists.