Introduction

PyTorch implementation for ACM MM2023 paper:

Joint Searching and Grounding: Multi-Granularity Video Content Retrieval

The codes are modified from TVR and MS-SL

Environments

python 3.10
pytorch 1.13.1
torchvision 0.14.1
tensorboard 2.10.0
tqdm 4.64.1
easydict 1.10
h5py 3.7.0
cuda 11.7

Data Preparation

The data split files, textual RoBERTa features and visual features of Charades-STA dataset can be downloaded from Baidu Cloud Disk or Google Drive. The directory structure is expected to be the following:

data
|-- charades
|   |-- TextData
|   |-- charades_i3d_rgb_lgi.hdf5

Training and Evaluation

Training

Run the following scripts to train JSG on the corresponding dataset.

#Add project root to PYTHONPATH (Note that you need to do this each time you start a new session.)
source setup.sh

EXP_ID=train_jsg
GPU_DEVICE_ID=0
ROOTPATH=$HOME/data

./charades.sh $EXP_ID $GPU_DEVICE_ID $ROOTPATH

Evaluation

The model is placed in the directory $ROOTPATH/$DATASET/results/$MODELDIR after training. Run the following script to evaluate it(Suppose the model is trained on Charades-STA):

DATASET=charades
EVALID=eval_jsg
ROOTPATH=$HOME/data
MODELDIR=xxx

./test.sh $DATASET $EVALID $ROOTPATH $MODELDIR

Model Performance

Video-level retrieval performance

On Charades-STA

	R@1	R@5	R@10	R@100	SumR
JSG	2.4	7.7	12.8	49.8	72.7

Event-level retrieval performance