LIGHTEN-Learning-Interactions-with-Graphs-and-hierarchical-TEmporal-Networks-for-HOI

Introduction

This repository contains code for LIGHTEN HOI detection pipeline, proposed in the ACM MM'20 paper: LIGHTEN: Learning Interactions with Graph and Hierarchical TEmporal Networks for HOI in videos.

Installation

LIGHTEN is implemented in Pytorch1.4 with CUDA-10.1 in python3.8. Other python packages can be installed using :

pip install -r requirements.txt

Setting up the codebase

Datasets

Download RGB frames for CAD120 videos from CAD120 dataset page
Download COCO image directory (traintest2017) from COCO website

Pre-trained models

Download the pretrained models from the following folders : i) CAD120 checkpoints ii) V-COCO checkpoints

Configuration

Set the corresponding paths to data and pre-trained models in config.py file. Hyper-paramters and model configurations can be set from this file. The directory structure after setting up looks like :

LIGHTEN-Learning-Interactions-with-Graphs-and-hierarchical-TEmporal-Networks-for-HOI/
  CAD120/
    checkpoints/
      checkpoint_GCN_frame_detection.pth
      checkpoint_GCN_segment_detection.pth
    data/
      training_data.p
      testing_data.p
    models/
  V-COCO/
    checkpoints/
    data/
      training_data.p
      testing_data.p
      action_index.json
      Test_Faster_RCNN_R-50-PFN_2x_VCOCO_Keypoints.pkl
    models/

Running the code

CAD120

Resnet frame-wise features can be precomputed and stored beforehand as :

cd CAD120/
python compute_RoI_feats.py

This will create two new files at CAD120/data/, which contain image features from backbone module. Alternately, the precomputed feature files can be downloaded from this google drive folder.
Training and Testing the LIGHTEN model for CAD120 can be done as follows :

cd CAD120/
python train_CAD120.py
python test_CAD120.py

V-COCO

Resnet image features can be precomputed and stored using :

cd V-COCO/
python compute_RoI_feats.py

This will store resnet features in the directory : V-COCO/data/
Training, Validation and Testing of LIGHTEN on V-COCO (image-setting) can be done as :

cd V-COCO/
python train_VCOCO.py
python eval_VCOCO.py
python test_VCOCO.py

Note that eval_VCOCO.py evaluates only action label detection, and uses ground-truth object detections similar to train_VCOCO.py. However, test_VCOCO.py evaluates the model using faster-RCNN-FPN object detections, and computes the final mAP score as per the evaluation script available at VSRL Repository

praneeth11009/LIGHTEN-Learning-Interactions-with-Graphs-and-Hierarchical-TEmporal-Networks-for-HOI