The project aims to train a model using tensorflow for facial emotion detection and used the trained model as predictor in android facial expression recongnition app.
The model is trained using tensorflow python framework and used in android application where the basic langauge is java.
Basically tensorflow provides a c++ api, that can be used in android application. The trained model by python langauge can be integrated with android project after inclduing tensorflow c++ framework dependencies and using native interface the model can be loaded and called in java class. This is the whole thing.
The total work of this project is divided into two parts
-
Devlop a model in tensoflow using python langauge
KaggleFaceEmotionRecognition folder contains the work
-
Devlop an android appication for facial expression recongtion
FaceEMR folder contains the work
In this work , I have used a simple Convolutional Neural Network Architecture to train a facial expression dataset.
1. DataSet: The dataset is collected from Facial xpression recognition challenge in kaggle The challenge link https://www.kaggle.com/c/challenges-in-representation-learning-facial-expression-recognition-challenge/
The data consists of 48x48 pixel grayscale images of faces.The dataset contains facial expression of seven categories (0=Angry, 1=Disgust, 2=Fear, 3=Happy, 4=Sad, 5=Surprise, 6=Neutral
2. Model:
In this work I have used the below CNN model
input_image->conv2d->pooling->conv2d->pooling->conv2d->pooling->dropout->softmax
The code fragment
```
x_image = tf.reshape(x, [-1, 48, 48, 1])
#48*48*1
conv1 = tf.layers.conv2d(x_image, 64, 3, 1, 'same', activation=tf.nn.relu)
#48*48*64
pool1 = tf.layers.max_pooling2d(conv1, 2, 2, 'same')
#24*24*64
conv2 = tf.layers.conv2d(pool1, 128, 3, 1, 'same', activation=tf.nn.relu)
#24*24*128
pool2 = tf.layers.max_pooling2d(conv2, 2, 2, 'same')
#12*12*128
conv3 = tf.layers.conv2d(pool2, 256, 3, 1, 'same', activation=tf.nn.relu)
#12*12*256
pool3 = tf.layers.max_pooling2d(conv3, 2, 2, 'same')
#6*6*256
flatten = tf.reshape(pool3, [-1, 6*6*256])
fc = tf.layers.dense(flatten, 1536, activation=tf.nn.relu)
dropout = tf.nn.dropout(fc, keep_prob)
logits = tf.layers.dense(dropout, 7)
outputs = tf.nn.softmax(logits, name=output_node_name)
```
3. Result: I have used 5000 iterations with batch size 100 and restore the model in protocal buffer file
I have used Android Studio for this application.
Integrating tensorflow dependency in android is really a tedious thing. the good news is that the latest news that android studio manages all dependencis related to tensorflow after adding the dependencies in build.gradle(Module:app) file
dependencies {
compile 'org.tensorflow:tensorflow-android:+'
}
The final dependency part looks like
dependencies {
implementation fileTree(dir: 'libs', include: ['*.jar'])
implementation 'com.android.support:appcompat-v7:26.1.0'
implementation 'com.android.support.constraint:constraint-layout:1.1.0'
testImplementation 'junit:junit:4.12'
androidTestImplementation 'com.android.support.test:runner:1.0.1'
androidTestImplementation 'com.android.support.test.espresso:espresso-core:3.0.1'
compile 'org.tensorflow:tensorflow-android:+'
}
1. Designing the UI Components
Home Screen look like this
After taking a picture
And the final result
2. Interacting with the Tensorflow Native Api
The org.tensorflow.contrib.android.TensorFlowInferenceInterface handles all necessary operation to interact with native api. See more details in https://github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/android
3. Finalizing the work
There are lots of options for improvement.
- Adding some cool features and increasing model performance using another models
- Tuning the hyperparameters of used cnn model