Automatic Image Captioning in facial emotion dataset

Description

This is the code repo for UCD final year project: Automatic Image Captioning in facial emotion dataset

The project develops an appropriate model for images containing human faces that provides a more accurate and impact caption.

The project aims to:

Preparing a group of suitable data from dataset.
Developing a model to generate new caption from the image.
Developing a model to extract emotion from the human face.
Developing a model to produce improved captions from caption and emotion generated.
Evaluating and analysing the performance of the model.

For more details, please check the project report.

No prerequisite required.

Clone or download the repo

git clone https://csgitlab.ucd.ie/nanwu/automatic-image-captioning-in-facial-emotion-dataset.git

Runs should follow the following order:

data-cleaning.ipynb

Create flickr5k dataset folder from flickr8k
(optional) flickr5k-analysis.ipynb

Only used for dataset analysis
image-feature.ipynb

Create flickr5k_features.pkl storing image features in flickr5k folder
tokenize-caption.ipynb

Create flickr5k_tokenizer.pkl storing caption tokenizer in flickr5k folder
image-captioning-model.ipynb

Create flickr5k_model.h5 storing the model in flickr5k folder

Run:

emotion-and-evaluation.ipynb

This script requires flickr5k_model.h5 and evaluation images