CoRe

Created by Xumin Yu*, Yongming Rao*, Wenliang Zhao, Jiwen Lu, Jie Zhou

This is the PyTorch implementation for ICCV paper Group-aware Contrastive Regression for Action Quality Assessment arXiv.

We present a new Contrastive Regression (CoRe) framework to learn the relative scores by pair-wise comparison, which highlights the differences between videos and guides the models to learn the key hints for action quality assessment.

Pretrained Model

Our pretrained CoRe model for MTL-AQA is available at [Tsinghua Cloud] [Google Drive]

Usage

Requirement

Python >= 3.6
Pytorch >= 1.4.0
torchvision >= 0.4.1
torch_videovision

pip install git+https://github.com/hassony2/torch_videovision

Download initial I3D

We use the Kinetics pretrained I3D model from the reposity kinetics_i3d_pytorch

Dataset Preparation

MTL-AQA

Please download the dataset from the repository MTL-AQA. The data structure should be:

$DATASET_ROOT
├── MTL-AQA/
    ├── new
        ├── new_total_frames_256s
            ├── 01
            ...
            └── 09
    ├── info
        ├── final_annotations_dict_with_dive_number
        ├── test_split_0.pkl
        └── train_split_0.pkl
    └── model_rgb.pth

The processed annotations are already provided in this repo. The prepared dataset is coming soon (in BaiDuYun). If you want to prepare the data by yourself, please see MTL_helper for some helps. We provide codes for processing the data from an online video to the frames data.

AQA-7

Download AQA-7 Dataset:

mkdir AQA-Seven & cd AQA-Seven
wget http://rtis.oit.unlv.edu/datasets/AQA-7.zip
unzip AQA-7.zip

The data structure should be:

$DATASET_ROOT
├── Seven/
    ├── diving-out
        ├── 001
            ├── img_00001.jpg
            ...
        ...
        └── 370
    ├── gym_vault-out
        ├── 001
            ├── img_00001.jpg
            ...
    ...

    └── Split_4
        ├── split_4_test_list.mat
        └── split_4_train_list.mat

The prepared dataset is coming soon (in BaiDuYun).

JIGSAWS

Please download the dataset from JIASAWS. You are required to complete a form before you use this dataset for academic research.

The training and test code for JIGSAWS is on the way.

Training and Evaluation

To train a CoRe model:

bash ./scripts/train.sh <GPUIDS>  <MTL/Seven> <exp_name>  [--resume]

For example,

# train a model on MTL
bash ./scripts/train.sh 0,1 MTL try 

# train a model on Seven
bash ./scripts/train.sh 0,1 Seven try --Seven_cls 1

To evaluate a pretrained model:

bash ./scripts/test.sh <GPUIDS>  <MTL/Seven> <exp_name>  --ckpts <path> [--Seven_cls <int>]