Introduction

Partial implementation of Hierarchical Boundary-Aware Neural Encoder for Video Captioning. The C3D part is implemented but not work well.

Difference from original paper: In the original paper, the author sampled every 5 frames, while I sampled fix number of frames using np.linspace.

Requirements

Pretrained Model

VGG16 pretrained on ImageNet [PyTorch version]: https://download.pytorch.org/models/vgg16-397923af.pth
ResNet50 pretrained on ImageNet [PyTorch version]: https://s3.amazonaws.com/pytorch/models/resnet50-19c8e357.pth
C3D pretrained on Sports1M [ported from Keras]: http://imagelab.ing.unimore.it/files/c3d_pytorch/c3d.pickle

Datasets

Packages

torch
torchvision
numpy
scikit-image
nltk
h5py
pandas
future # python2 only
tensorboard_logger # for using tensorboard to view training loss

You can use:

    (sudo) pip2 install -r requirements.txt

to install all the above packages.

Usage

Preparing Data

Firstly, we should make soft links to the dataset folder and pretrained models. For example:

    mkdir datasets
    ln -s YOUR_MSVD_DATASET_PATH datasets/MSVD
    mkdir models
    ln -s YOUR_RES50_MODEL_PATH models/

somes detail can be found in args.py.

Note: If you use the MSR-VTT dataset, there are some extra steps. The MSR-VTT dataset split the train_val and test video data into two zip files, so as the annotations. So please merge the two parts of video data (annotations) into one directory (json file), and modify the msrvtt_video_root and msrvtt_anno_json_path variables in args.py

Then we can:

Prepare video feature:
```
 python2 video.py
```
Prepare caption feature and dataset split:
```
 python2 caption.py
```

Training

Before training the model, please make sure you can use GPU to accelerate computation in PyTorch. Some parameters, such as batch size and learning rate, can be found in args.py.