/Facial-Expression-Recognition

Primary LanguagePython

Facial expression Recognition using ensemble of CNNs

Table of Contents

Benchmarks

FER2013

Model Accuracy
VGG19 70.77
EfficientNet_b2b 70.83
Googlenet 71.97
Resnet34 72.42
Inception_v3 72.72
Resnet50 72.86
Cbam_Resnet50 72.95
Bam_Resnet50 73.14
Densenet121 73.16
Resnet152 73.22
Resnet101 74.06
ResMaskingNet 74.14
ResMaskingNet + 6 76.82

JAFFE

Model Accuracy
EfficientNet_b2b 90
Resnet18 91.42
Resnet101 92.86
Resnet50 94.28
Cbam_Resnet50 94.29
ResMaskingNet 97.1
Ensemble of above CNNs 98.2

CK+

Model Accuracy
Resnet18 96.631
Densenet121 97.573
VGG19 98.058
Resnet101 98.544
ResNet50_pretrainedvgg 98.544
ResMaskingNet 98.87

Installation

  • Install PyTorch by selecting your environment on the website and running the appropriate command.
  • Clone this repository and install package prerequisites below.
  • Then download the dataset by following the instructions below.

Prerequisites

Datasets

  • FER2013 Dataset (locate it in saved/data/fer2013 like saved/data/fer2013/train.csv)
  • JAFFE Dataset (locate it in saved/data/jaffe like saved/data/jaffe/train.csv)

Training on FER2013

  • To train network, you need to specify model name and other hyperparameters in config file (located at configs/*) then ensure it is loaded in main file, then run training procedure by simply running main file, for example:
python main_fer.py  # Example for fer2013_config.json file

  • The best checkpoints will chosen at term of best validation accuracy, located at saved/checkpoints

  • The TensorBoard training logs are located at saved/logs, to open it, use tensorboard --logdir saved/logs/

  • By default, it will train alexnet model, you can switch to another model by editing configs/fer2013\_config.json file (to resnet18 or cbam\_resnet50 or my network resmasking\_dropout1.

    Follow similar process for JAFFE dataset

Evaluation

Below is an example for generating a striking confusion matrix writing things in latex.

(Read this article for more information, there will be some bugs if you blindly run the code without reading).

python ./Visualization/gen_confusion_matrix.py

Below is an example, to generate salient maps for jaffe images to find most import parts of the image used by the model for prediction.

python ./Visualization/gen_salientmap.py

Below is an example, to generate gradCAM visualization to find most import parts of the image used by the model for prediction.

python ./Visualization/gradCAM_resmasking.py

Ensemble method

I used no-weighted sum avarage ensemble method to fuse 7 different models together, to reproduce results, you need to do the following steps:

  1. Download all needed trained weights and located on ./saved/checkpoints/ directory. Link to download can be found on Benchmarking section.
  2. Edit file gen_results and run it to generate result offline for each model.
  3. Run gen_ensemble.py file to generate accuracy for example methods.

Documentation

documentation

References