Several models were built and they were trained on the FER-2013 dataset and their performances were compared. Further, these models were used to build a web application for real-time facial emotion recognition with optimized frame rate.
Preprocessing: Images are resized to 128 X 128 or 224 X 224 for some models. Training, validation and testing sets are created.
Training: Models are trained on GPU and hyperparameters are tuned accordingly.
Face detection: Haarcascade frontal face classifier or opencv DNN face detector is used to detect faces in the image.
Emotion Recognition: Predictions are made on the detected faces.
- Install all dependencies using requirements.txt like so:
pip install -r requirements.txt-
Git LFS - (Can be skipped) To clone and use this repository, you'll need Git Large File Storage (LFS).
-
Note that if you want to follow the notebook, you will have to make appropriate changes to the directory paths which should be trivial.
- Use the following command to run the app.py file
python app.py- Open http://127.0.0.1:5000/ on any browser
Model weights are provided in the folders of respective models. Their architectures can be found in the notebook.
- models 1 and 2 were trained locally on my PC with GPU (NVIDIA GeForce MX130). Due to hardware limitations, they couldn't be trained to their optimal state.
- Model3, built from scratch, is a decent model that has relatively fewer parameters (654,335). It was also trained locally and various hyperparameters were tuned so that its accuracy reaches its optimum. An accuracy of nearly 50% was achieved on the testing set. The following emotions can be easily detected with this model: happy, neutral, angry, and fear.
- The models, model_mobilenet and model_resnet_FineTuned_Large, could achieve over 60% accuracy on the testing set. These models were trained on a Kaggle notebook using GPU and their optimal weights were saved and downloaded. The notebook can be found here
- Due to hardware limitations, the Resnet50 based model couldn't be tested in real-time. It should have a performance comparable to that of the MobileNetV2 based model.
- The MobileNetV2 based model can easily detect most emotions in real-time. It also has the highest testing accuracy (60.67%) among all the models I tested.
Further analysis is provided in the notebook.
- Due to RAM limitations of Kaggle, I wasn't able to augment the training data. With more RAM, or perhaps, with a more memory-efficient code, one can train the models on an augmented training set which could substantially improve the accuracy.
- I have also tuned the learning rate while testing the models but perhaps, more meticulous tuning of other hyperparameters may result in improved accuracy.
- For the resnet based model, one can try freezing fewer or more layers and see if it improves the accuracy.
Following some or all of the above suggestions, I believe the accuracies of model_mobilenet and model_resnet_FineTuned_Large could easily go up to 65-70%
Easily accessible demo for real-time emotion recognition.
Distributed under the MIT License. See LICENSE for more information.
- This repository provided helpful resources for this project.