declare-lab/MISA

MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis

MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis

Code for the ACM MM 2020 paper MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis

Setup the environment

We work with a conda environment.

conda env create -f environment.yml
conda activate misa-code

Data Download

• Install CMU Multimodal SDK. Ensure, you can perform from mmsdk import mmdatasdk.
• Option 1: Download pre-computed splits and place the contents inside datasets folder.
• Option 2: Re-create splits by downloading data from MMSDK. For this, simply run the code as detailed next.

Running the code

1. cd src
2. Set word_emb_path in config.py to glove file.
3. Set sdk_dir to the path of CMU-MultimodalSDK.
4. python train.py --data mosi. Replace mosi with mosei or ur_funny for other datasets.

Citation

If this paper is useful for your research, please cite us at:

@article{hazarika2020misa,
  title={MISA: Modality-Invariant and-Specific Representations for Multimodal Sentiment Analysis},
  author={Hazarika, Devamanyu and Zimmermann, Roger and Poria, Soujanya},
  journal={arXiv preprint arXiv:2005.03545},
  year={2020}
}

Contact

For any questions, please email at hazarika@comp.nus.edu.sg