thuiar/ch-sims-v2

Make Acoustic and Visual Cues Matter: CH-SIMS v2.0 Dataset and AV-Mixup Consistent Module

CH-SIMS v2.0

Official codes for paper "Make Acoustic and Visual Cues Matter: CH-SIMS v2.0 Dataset and AV-Mixup Consistent Module" (ICMI 2022)

• Dataset HomePage

• ICMI 2022 Paper

1. Illustration of CH-SIMS v2.0 Data

2. Data Download

1. CH-SIMS v2(s) - Supervised data:

• Google Drive
• Baiduyun Drive Code: icmi

2. CH-SIMS v2(u) - Unsupervised data:

• Google Drive
• Baiduyun Drive Code: icmi

3. Baselines Results

4. Run Experiments

1. Download dataset and set correct path in:

config/config.py --> line 32  --> "root_dataset_dir"

2. If you want to run the AV-MC framework:

python run.py --is_tune Flase --modelName v1

3. If you want to run the AV-MC(Semi) framework

python run.py --is_tune Flase --modelName v1_semi

5. Citation

If you find this paper or dataset useful, please cite us at:

@misc{liu2022make,
      title={Make Acoustic and Visual Cues Matter: CH-SIMS v2.0 Dataset and AV-Mixup Consistent Module}, 
      author={Yihe Liu and Ziqi Yuan and Huisheng Mao and Zhiyun Liang and Wanqiuyue Yang and Yuanzhe Qiu and Tie Cheng and Xiaoteng Li and Hua Xu and Kai Gao},
      year={2022},
      eprint={2209.02604},
      archivePrefix={arXiv},
      primaryClass={cs.MM}
}

6. Contact Us

For any questions, please email Yihe Liu or Ziqi Yuan