/dntai_fg20_affwild2_challenges

Affective Behavior Analysis in-the-wild Challenge in FG 2020 Conference

Primary LanguageJupyter Notebook

Affective Expression Analysis in-the-wild using Multi-Task Temporal Statistical Deep Learning Model

Challenges: The First Affective Behavior Analysis in-the-wild (ABAW) Competition
Homepage: https://ibug.doc.ic.ac.uk/resources/fg-2020-competition-affective-behavior-analysis/
Team Name: CNU_ADL
Team Members:
(1) Nhu-Tai Do, donhutai@gmail.com
(2) Tram-Tran Nguyen Quynh, tramtran2@gmail.com
(3) Soo-Hyung Kim
Affiliation: Chonnam National University, South Korea

Our paper:

Affective Expression Analysis in-the-wild using Multi-Task TemporalStatistical Deep Learning Model
Link: https://arxiv.org/abs/2002.09120

@article{Do2020,
	archivePrefix = {arXiv},
	arxivId = {2002.09120},
	author = {Do, Nhu-Tai and Kim, Soo-Hyung},
	eprint = {2002.09120},
	month = {Feb},
	title = {{Affective Expression Analysis in-the-wild using Multi-Task Temporal Statistical Deep Learning Model}},
	url = {http://arxiv.org/abs/2002.09120},
	year = {2020}
}

How to run

  1. Download and setup Anaconda3
  2. Run setup_envs.sh to install conda environments with Python 3.7, keras, tensorflow, etc.
  3. Unzip two pandas index files of Aff-Wild2 dataset: affwild2_cropped_aligned_frames_v1.zip and affwild2_cropped_frames_v1.zip in [data/AffWild2/data] folder
  4. Download and setup Aff-Wild2 dataset:
    1. Extract annotations.zip and copy 3 folder AU_Set, EXPR_Set, VA_Set to data/AffWild2/data/annotations folder
    2. Extract ccropped_aligned.zip to data/AffWild2/data/cropped_aligned folder
    3. Extract ccropped.zip and merge batch 1&2 folder to data/AffWild2/data/cropped folder
    4. Extract videos.zip and merge batch 1&2 folder to data/AffWild2/data/cropped_aligned folder
  5. Download weight files and copy to folder submit1/weights from https://drive.google.com/drive/folders/1rJB2viPCxw93qFSaga3uqC6OfWMKRHn2?usp=sharing
  6. Open JupyterLab and run *.ipynb in submit folder to output the results (
    • Run sel_t[xx].ipynb to output the prediction files(modify params parameter if neccessary)
    • Run sel_t[xx]_submit.ipynb to output the result folder (modify params parameter if neccessary)

Proposed model

alt text

Aff-Wild2 dataset

  • Overview cropped_aligned image frames in different videos alt text

  • Overview cropped_aligned image frames in the same videos alt text

  • Data Distribution in Basic Emotion Recognition Track on Training and Validation alt text

  • Data Distribution in Valence-Arousal Regression Track on Training and Validation alt text

Result

  • List Models
    alt text

  • List Results
    alt text

  • Fusion Results on Validation: E xpr. Score = 0.533, Valence-Arousal Score = 0.5126

  • Submission results: Track 1 Valence-Arousal Challenge on Validation: 0.484 (1), 0.534 (2), 0.514 (3), and 0.527 (4)
    Track 2 Basic Emotion Recognition Challenge on Validation: 0.501 (1), 0.492 (2), 0.478 (3), and 0.543 (4)

Baseline paper: 

@misc{kollias2020analysing,
    title={Analysing Affective Behavior in the First ABAW 2020 Competition},
    author={Dimitrios Kollias and Attila Schulc and Elnar Hajiyev and Stefanos Zafeiriou},
    year={2020},
    eprint={2001.11409},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}