/EmotionDetection

Dimensional Emotion Detection from Categorical Emotion Annotation

Primary LanguagePython

Dimensional Emotion Detection from Categorical Emotion Annotation

Paper | Dataset Preparation | Traning | Results | Citation | Contact

Abstract: We present a model to predict fine-grained emotions along the continuous dimensions of valence, arousal, and dominance (VAD) with a corpus with categorical emotion annotations. Our model is trained by minimizing the EMD (Earth Mover's Distance) loss between the predicted VAD score distribution and the categorical emotion distributions sorted along VAD, and it can simultaneously classify the emotion categories and predict the VAD scores for a given sentence. We use pre-trained RoBERTa-Large and fine-tune on three different corpora with categorical labels and evaluate on EmoBank corpus with VAD scores. We show that our approach reaches comparable performance to that of the state-of-the-art classifiers in categorical emotion classification and shows significant positive correlations with the ground truth VAD scores. Also, further training with supervision of VAD labels leads to improved performance especially when dataset is small. We also present examples of predictions of appropriate emotion words that are not part of the original annotations.

This repository contains the source code for the paper Toward Dimensional Emotion Detection from Categorical Emotion Annotations, which is accepted by EMNLP 2021. In this work, we present a model to predict fine-grained emotions along the continuous dimensions of valence, arousal, and dominance (VAD) with a corpus with categorical emotion annotations. We use pre-trained RoBERTa-Large and fine-tune on three different corpora with categorical labels and evaluate on EmoBank corpus with VAD scores. Please refer to the paper for more details.

Overview of approach: Overview of approach

Dataset Preparation

In this work, we use four existing datasets consisting of text and corresponding emotion annotations. Three of them have categorical emotion labels, and the last is annotated with VAD scores. We summarize the key information of these datasets and provide the links to download these datasets if they are directly accessible.

Dataset Size # Emotions Type Download Link
SemEval 10,983 11 multi-labeled classification link
ISEAR 7,666 7 single-labeled classification link
GoEmotions 58,009 28 multi-labeled classification link
EmoBank 10,062 V,A,D regression link

Traning

Dependencies Requirements

pip install -r requirements.txt

Running Baseline Model (Pretrained RoBERTa-Base)

Classification

python src/main.py --config configs/baseline/semeval_cat_classification.json

Regression

python src/main.py --config configs/baseline/emobank_dim_regression.json

Running Our Model

Classification & Zeroshot VAD prediction

python src/main.py --config configs/semeval/vad_from_categories.json

Regression

python src/main.py --config configs/semeval/vad_regression.json

Results

Performance of EmoBank VAD score prediction

With fine-tuning pre-trained RoBERTa-Large, we show significant positive correlations with VAD scores using only the categorical emotion annotations. If those models are continuously fine-tuned on EmoBank, it outperforms all SOTA VAD regression models.

Model Scheme V(r) A(r) D(r)
Baseline, SemEval Zero-shot .715 .319 .319
Baseline, ISEAR Zero-shot .611 .083 .242
Baseline, GE Zero-shot .630 .277 .311
AAN Supervised .424 .352 .265
Ensemble Supervised .635 .375 .277
SRV-SLSTM Semi-super .620 .508 .333
RoBERTa-Large Supervised .829 .569 .513
Ours, EB←SemEval Supervised .838 .570 .518
Ours, EB←ISEAR Supervised .836 .568 .536
Ours, EB←GE Supervised .835 .573 .529

Performance of categorical emotion classification

With fine-tuning pre-trained RoBERTa, we show comparable performance to SOTA models in classification.

Dataset (Model) Macro F1 Micro F1 Acc.
ISEAR (MT-CNN) - .668 -
ISEAR (Baseline) .754 .755 -
ISEAR (Ours) .752 .753 -
SemEval (NTUA-SLP) .528 .701 .588
SemEval (Seq2Emo) - .709 .592
SemEval (RoBERTa-Large) .574 .725 .607
SemEval (Ours) .566 .725 .607
GoEmotions .640 - -
GoEmotions (Baseline) .618 .691 .659
GoEmotions (Ours) .611 .686 .657

Citation

@article{park2019toward,
  title={Toward dimensional emotion detection from categorical emotion annotations},
  author={Park, Sungjoon and Kim, Jiseon and Jeon, Jaeyeol and Park, Heeyoung and Oh, Alice},
  journal={arXiv preprint arXiv:1911.02499},
  year={2019}
}

Contact

Please contact {sungjoon.park, jiseon_kim, vano1205}@kaist.ac.kr or raise an issue in this repository.