AKMNet for Micro-Expression

Recognizing Micro-Expression in Video Clip with Adaptive Key-Frame Mining

Purpose

The existing representation based on various deep learning techniques learned from a full video clip is usually redundant. In addition, methods utilizing the single apex frame of each video clip require expert annotations and sacrifice the temporal dynamics. In our paper, we propose a novel end-to-end deep learning architecture, referred to as adaptive key-frame mining network (AKMNet). Operating on the video clip of micro-expression, AKMNet is able to learn discriminative spatio-temporal representation by combining spatial features of self-learned local key frames and their global-temporal dynamics.

Citation

Peng, Min, Chongyang Wang, Yuan Gao, Tao Bi, Tong Chen, Yu Shi, and Xiang-Dong Zhou. "Recognizing Micro-expression in Video Clip with Adaptive Key-frame Mining", arXiv preprint arXiv:2009.09179 (2020).

Platform and dependencies

Ubuntu 14.04 Python 3.7 CUDA8.0 CuDNN6.0+
pytorch==1.0.0

Data Preparation (option)

Download the dataset
CASMEI: http://fu.psych.ac.cn/CASME/casme-en.php
CASMEII: http://fu.psych.ac.cn/CASME/casme2-en.php
SAMM: http://www2.docm.mmu.ac.uk/STAFF/m.yap/dataset.php
SMIC: https://www.oulu.fi/cmvs/node/41319
preprocessing
1.you can also use the data in cropped fold to conduct the experiment. For SAMM dataset, the face detect and align method same as the paper CASME II: An Improved Spontaneous Micro-Expression Database and the Baseline Evaluation
2. for phase-based video magnification method, please ref to http://people.csail.mit.edu/nwadhwa/phase-video/
In our method, we do not need the frame normalization, the design of all modules in the AKMNet is independent on the length of input video clip

Method

Experiment

Comparison Experiment

Methods	CASMEI	CASMEII	SMIC	SAMM
`LBP-TOP`	0.6618	0.3843	0.3598	0.3899
`LBP-SIP`	0.6026	0.4784	0.3842	0.5220
`STCLQP`	0.6349	0.5922	0.5366	0.5283
`HIGO`	0.5781	0.5137	0.3720	0.4465
`FHOFO`	0.6720	0.6471	0.5366	0.6038
`MDMO`	0.6825	0.6314	0.5793	0.6164
`Macro2Micro`	0.6772	0.6078	-	0.6436
`MicroAttention`	0.6825	0.6431	-	0.6489
`ATNet`	0.6720	0.6039	-	0.6543
`STSTNet`	0.6349	0.5529	0.5488	0.6289
`STRCN-G`	0.7090	0.6039	0.6280	0.6478
# AKMNet	0.7566	0.6706	0.7256	0.7170

Justification of the Adaptive Key-Frame Mining Module

Methods	CASMEI	CASMEII	SMIC	SAMM
`AKMNetva-all`	0.6618	0.3843	0.3598	0.3899
`AKMNetva-random`	0.6138	0.6118	0.5427	0.6289
`AKMNetva-norm16`	0.6667	0.6314	0.5976	0.6604
`AKMNetva-norm32`	0.6825	0.6392	0.6434	0.6478
`AKMNetva-norm64`	0.7090	0.6392	0.6463	0.6164
`AKMNetva-norm128`	0.6984	0.6431	0.6646	0.6792
`AKMNet`	0.7566	0.6706	0.7256	0.7170

Ablation Experiment

Methods	CASMEI	CASMEII	SMIC	SAMM
`AKMNet-s12`	0.6984	0.6392	0.6463	0.6667
`AKMNet-s13`	0.7354	0.6431	0.6463	0.6604
`AKMNet-s23`	0.7249	0.6549	0.6707	0.6918
`AKMNet-s123`	0.7566	0.6706	0.7256	0.7170

Annotated Apex Frame VS ‘Most Informative’ Frame

Methods	CASMEI	CASMEII	SMIC	SAMM
`Resnet18`	Apex frame	0.6772	0.6078	0.6436
`Resnet18`	Max-key frame	0.6825	0.6392	0.6486
`VGG-11`	Apex frame	0.6667	0.6235	0.6277
`VGG-11`	May-key frame	0.6931	0.6353	0.6649

how to use: for each LOSO exp:
first： set list_file_train and list_file_test in main.py properly, each of them is a list file, contents in file like this:
/home/XXX/fold/sub01/EP01_12__alpha15 19 3
...
where /home/XXX/fold/sub01/EP01_12__alpha15 is a fold which contain a image sequence of a micro-expression, 19 is the len of the clips, 3 is the label
second: set premodel in main.py if you have the pretrained model
third: run python main.py in your terminal.

If you have questions, post them in GitHub issues.