/MISA

MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis

Primary LanguagePythonMIT LicenseMIT

MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis

Code for the ACM MM 2020 paper MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis

Setup the environment

We work with a conda environment.

conda env create -f environment.yml
conda activate misa-code

Data Download

  • Install CMU Multimodal SDK. Ensure, you can perform from mmsdk import mmdatasdk.
  • Option 1: Download pre-computed splits and place the contents inside datasets folder.
  • Option 2: Re-create splits by downloading data from MMSDK. For this, simply run the code as detailed next.

Running the code

  1. cd src
  2. Set word_emb_path in config.py to glove file.
  3. Set sdk_dir to the path of CMU-MultimodalSDK.
  4. python train.py --data mosi. Replace mosi with mosei or ur_funny for other datasets.

Citation

If this paper is useful for your research, please cite us at:

@article{hazarika2020misa,
  title={MISA: Modality-Invariant and-Specific Representations for Multimodal Sentiment Analysis},
  author={Hazarika, Devamanyu and Zimmermann, Roger and Poria, Soujanya},
  journal={arXiv preprint arXiv:2005.03545},
  year={2020}
}

Contact

For any questions, please email at hazarika@comp.nus.edu.sg