This repository contains the code for the Final Project of the Natural Language Processing course at Tsinghua University, Spring 2021.
In this project, we have developed a Deep Learning model to generate a description of a short video clip using a single sentence in English. The model architecture consists of a Visual CNN encoder and a pre-trained VGGish audio encoder, followed by Soft Attention for multi-modal fusion and a LSTM sentence decoder. The design choices are inspired by RecNet. The output is a sequence of words that represent the content of the input video. The full architecture is illustrated below:
Please refer to the project report for more specific details about the architecture, training procedures and results.
First, clone the repository in your local machine by running this command:
git clone https://github.com/hmartelb/multimodal-video-captioning
cd multimodal-video-captioning
Make sure that you have Python 3 installed in your system. Also, Pytorch 1.5 or above needs to be installed. Check the official installation guide to set it up according to your system requirements and CUDA version.
It is recommended to create a virtual environment to install the dependencies. Open a new terminal in the master directory, activate the virtual environment and install the dependencies from requirements.txt
by executing this command:
(venv) pip install -r requirements.txt
Download the MSVD and MSR-VTT datasets and place them in the datasets/
folder. The code expects the datasets to have following structure:
<dataset name>/
audios/
video1.wav
...
videoN.wav
features/
audio/
video/
metadata/
train.csv
val.csv
test.csv
videos/
video1.mp4
...
videoN.mp4
Some folders inside features/*
will be empty, and its contents will be generated after the feature extraction process.
In case of the MSVD dataset, it is necessary to run the
download_youtube.py
script before to obtain the audio data.
After donwloading the data, we need to compute the features for each video and audio clip.
(venv) python extract_features.py --dataset <path_to_dataset_root>
[--gpu <device_id>]
When the feature extraction finishes, the folders features/audio
and features/video
should contain 1 .npy
file for each video in the dataset, with the same name as the original.
To train a model, run the script train.py
with the appropriate command line arguments, as follows:
(venv) train.py --dataset 'MSVD' or 'MSR-VTT'
[--epochs <integer> (default, 50)]
[--batch_size <integer> (default, 128)]
[--lr <float> (default, 1e-4)]
[--gpu <device_id>]
Modify the experiment configuration inside of the script and adapt it to your own needs. An array of experiments can also be used to run sequentially, one after the other, for conveninence.
When the training has finished, evaluate the results by loading an existing model checkpoint, as follows:
- Execute the notebook in
notebooks/predict_captions.py
. - The results will be saved to the folder
results/<dataset_name>
in.csv
format with 1 file per model, containing the generated captions and ground truth captions. - Run the evaluation of
pycocoevalcap/
on the generated captions to obtain the objective scores for the model.
-
The code borrows the LSTM and reconstructor architectures from the original RecNet implementation, which has been adapted to our multi-modal problem.
-
The VGGish model used for audio feature extraction is from this PyTorch implementation.
This code implementation is licensed under the terms of the MIT License.
MIT License
Copyright (c) 2021 Héctor Martel, Chua Khang Hui
Master in Advanced Computing, Tsinghua University
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.