A problem about step2:creat_index for XD.

[KirinHao]

Hello, thank you for your work. I encountered a problem while reproducing the experiment:
As mentioned in your supplymentary, for XD datasets, using blip2-flan-t5-xxl directly will get the best results, but in slurm/xd_violence/02_create_index.sh, there are still five model generated captions loaded at the same time, how can I modify the code?
Looking forward to your reply, thank you!

# in slurm/xd_violence/02_create_index.sh
cap_model_names=(
    "$xd_violence_dir/captions/raw/Salesforce/blip2-opt-6.7b-coco/"
    "$xd_violence_dir/captions/raw/Salesforce/blip2-opt-6.7b/"
    "$xd_violence_dir/captions/raw/Salesforce/blip2-flan-t5-xxl/"
    "$xd_violence_dir/captions/raw/Salesforce/blip2-flan-t5-xl/"
    "$xd_violence_dir/captions/raw/Salesforce/blip2-flan-t5-xl-coco/"
)

# in src/models/create_index.py
for captions_dir in captions_dirs:
        captions_dir = Path(captions_dir)
        cap_model_name = captions_dir.name
        assert cap_model_name in CAP_MODEL_NAMES  # five models

Can I temporarily delete the other models directly in the code?
I am a student, and the graphics card resources are very tight, so I dare not try too much. Looking forward to your reply, thank you very much!

# in slurm/xd_violence/02_create_index.sh
cap_model_names=(
    "$xd_violence_dir/captions/raw/Salesforce/blip2-flan-t5-xxl/"
)

[lucazanella]

Hi, and thank you for your interest! As you correctly pointed out, the current script creates an index containing captions from all blip2 models, but for XD-Violence, you should only use captions from blip2-flan-t5-xxl. Indeed, index_name is set to flan-t5-xxl in slurm/xd_violence/03_clean_captions.sh, which ensure it uses the index created using blip2-flan-t5-xxl captions only.

The snippet of code you suggest:

cap_model_names=(
    "$xd_violence_dir/captions/raw/Salesforce/blip2-flan-t5-xxl/"
)

is correct.

Thank you for pointing that out!

[KirinHao]

Thank you very much!