Clarification regarding iOD task with ORE

[IsHYuhi]

Thank you for your amazing works!!!!

Although I would like to reproduce the red box part of the table, I am assuming that the provided config is to reproduce "ORE - (CC + EBUI)", is that correct?

The reason is that I thought the same as mentioned in #50,

I check the hyperparameters on the YAML files and find that the hyperparameters ENABLE_CLUSTERING and COMPUTE_ENERGY are set to FALSE on all three YAML files.

Do I need to change these parts? Also, in your response, you said "we don't need validation data in iOD setting", but how and for which data is EBUI used?

If it's needed a different setting not provided to reproduce the red box part, could you please provide it?

Thank you.

related issues:
#24 #8

[BrendaKuo]

Hi @JosephKJ and @IsHYuhi, I also have a question about reproducing the results. I run the codes in the following order.

1. python tools/train_net.py --num-gpus 4 --config-file ./configs/OWOD/iOD/19_p_1/base_19_train.yaml SOLVER.IMS_PER_BATCH 8 SOLVER.BASE_LR 0.01
2. python tools/train_net.py --num-gpu 4 --config-file ./configs/OWOD/iOD/19_p_1/next_1_train_with_ud.yaml SOLVER.IMS_PER_BATCH 8 SOLVER.BASE_LR 0.01
3. python tools/train_net.py --num-gpu 4 --config-file ./configs/OWOD/iOD/19_p_1/ft_with_unk.yaml SOLVER.IMS_PER_BATCH 8 SOLVER.BASE_LR 0.01
   When I was running the third command, it said FileNotFoundError: [Errno 2] No such file or directory: 'datasets/VOC2007/ImageSets/Main/train_ft_1.txt'. I couldn't find the corresponding txt file in the repository. Could you give me a direction or provide the corresponding file? I would really appreciate it.

[IsHYuhi]

As you may know, I think you can use ./OWOD/datasets/coco_utils/balanced_ft.py to create *ft*.txt file.
However, for the sake of accuracy, I was wondering if the authorr could provide it as well.

[JosephKJ]

Unfortunately, I missed uploading ft.txt, and I dont have them with me anymore.

Do I need to change these parts?

You can change the flags in the config files.

how and for which data is EBUI used?

EBUI is required only when we explicitly want to label some instances as unknown. This is not required in iOD, hence validation data is not required.

If you are serious about only iOD, you may have to consider this PR paper, this CVIU paper and this WACV paper too. This field is fast moving.

[BrendaKuo]

Hi @IsHYuhi,

As you may know, I think you can use ./OWOD/datasets/coco_utils/balanced_ft.py to create *ft*.txt file.
However, for the sake of accuracy, I was wondering if the authorr could provide it as well.

Thanks a lot for your answer! I managed to generate the required files. But instead of reproducing ORE 19+1 results in the article, mine dropped more than thirty points. I modified line 48-57 in ./datasets/coco_utils/balanced_ft.py as follows.

VOC_CLASS_NAMES_19 = [
    "aeroplane", "bicycle", "bird", "boat", "bottle", "bus", "car", "cat",
    "chair", "cow", "diningtable", "dog", "horse", "motorbike", "person",
    "pottedplant", "sheep", "sofa", "train"
]

VOC_CLASS_NAMES_p_1 = ["tvmonitor"]

# Change this accodingly for each task t*
known_classes = list(itertools.chain(VOC_CLASS_NAMES_19, VOC_CLASS_NAMES_p_1))
train_files = ['./datasets/VOC2007/ImageSets/Main/trainval.txt',
               './datasets/VOC2007/ImageSets/Main/trainval.txt']

annotation_location = './datasets/VOC2007/Annotations'
items_per_class = 20
dest_file = './datasets/VOC2007/ImageSets/Main/' + 'train_ft_1.txt'

Then I tested the accuracy of base 19 final model and found that Prev class AP50 was 76.78 which is about 7 points higher than the results in the article (19 +1 setting mAP). I don't know what the problem is. I only changed lr from default 0.02 to 0.01 and batchsize from default 16 to 8 to run the experiment on 4 GPUs. Did you change anything else when running the experiment? Do you mind sharing your experience reproducing the results of ORE - (CC + EBUI)? Thanks~