SHG-VQA

Learning Situation Hyper-Graphs for Video Question Answering [CVPR 2023]

Aisha Urooj Khan, Hilde Kuehne, Bo Wu, Kim Chheu, Walid Bousselhum, Chuang Gan, Niels Da Vitoria Lobo, Mubarak Shah

Website | Paper | BibTeX

Official Pytorch implementation and pre-trained models for Learning Situation Hyper-Graphs for Video Question Answering (coming soon).

Abstract

Answering questions about complex situations in videos requires not only capturing the presence of actors, objects, and their relations but also the evolution of these relationships over time. A situation hyper-graph is a representation that describes situations as scene sub-graphs for video frames and hyper-edges for connected sub-graphs and has been proposed to capture all such information in a compact structured form. In this work, we propose an architecture for Video Question Answering (VQA) that enables answering questions related to video content by predicting situation hyper-graphs, coined Situation Hyper-Graph based Video Question Answering (SHG-VQA). To this end, we train a situation hyper-graph decoder to implicitly identify graph representations with actions and object/human-object relationships from the input video clip. and to use cross-attention between the predicted situation hyper-graphs and the question embedding to predict the correct answer. The proposed method is trained in an end-to-end manner and optimized by a VQA loss with the cross-entropy function and a Hungarian matching loss for the situation graph prediction. The effectiveness of the proposed architecture is extensively evaluated on two challenging benchmarks: AGQA and STAR. Our results show that learning the underlying situation hypergraphs helps the system to significantly improve its performance for novel challenges of video question-answering task.

Proposed SHG-VQA Architecture

Datasets

We report results on two reasoning-based video question answering datasets: AGQA and STAR.

Instructions to preprocess AGQA and STAR (to be added)

AGQA

Download train and val split for the balanced version of AGQA 2.0 we used in our experiments from here.

Compilation Instructions

Download our data from above.
Download the Charades videos and frames.
Check the README.md files in each directory (AGQA/data and AGQA/annotations) to see where each file should be kept.
Open AGQA/src/tasks/agqa_data.py and locate the "todo"s. Change the paths of the root and annotations directory to match yours.

Training

For the full model, run agqaHGQA.py. For the question-only and video-question-only models, run agqaQ.py and agqaVQA.py respectively.

The following flags are used for the full model:

--train train --valid valid --llayers 5 --xlayers 2 --rlayers 5 \
--noCaps --crossAttnType cross --batchSize 32 --optim bert --lr 1e-5 --taskHGQA \
--fromScratch --LossHGPerFrame --epochs 100 --tqdm --output path/to/output/files \
--augmentType rand_aug --backbone slow_r50 --multiGPU

For the question-only or video-question-only models:
- Change the --taskHGQA to --taskQ or --taskVQA
For AQGA's test splits, models are trained as follows:
- Indirect References: This split is trained using the full model
- Novel Compositions: Add a --novelComp flag
- More Compositional Steps: Add a --compSteps flag

Evaluation

The following flags are used for evaluating the full model:

--test test --llayers 5 --xlayers 2 --rlayers 5 \ 
--noCaps --LossHGPerFrame --crossAttnType cross --batchSize 8 --optim bert --lr 1e-5 --taskHGQA \
--fromScratch --indirectRef --epochs 100 --tqdm --output path/to/output/files \
--augmentType no_aug --backbone slow_r50 --load path/to/saved/model --multiGPU

To test AQGA's different test splits:
- Indirect References: Keep the --indirectRef flag, load the full model's .pth
- Novel Compositions: Change to --novelComp, load the .pth from training the novelComp model
- More Compositional Steps: Change to --compSteps, load the .pth from training the compSteps model
- Note: --indirectRef will evaulate both the full model and the indirect references split together

Pre-Processing

Questions:

We randomly sampled 10% of questions from the training set to be used for the validation set

Train/Valid/Test files are formatted as a list of dictionaries, containing all information given by AGQA.

Format:

[{
"question": "Before starting to open a bag, what was the person taking?",
 "answer": "paper", 
 "video_id": "3ZUVI", 
 "global": ["obj-rel"], 
 "local": "v023", 
 "ans_type": "open", 
 "steps": 4, 
 "semantic": "object", 
 "structural": "query", 
 "novel_comp": 0, 
 "more_steps": 0, ...}]

Answers:

We iterated through all ground-truth answers in the training/validation set to get all answer choices.
All possible answers are mapped to an index and stored in the answer vocab.
- Format: {"answer1": 0, "answer2": 1, ...}

Video Frames:

Download the videos and their frames from the Charades dataset.
We randomly sampled a fixed clip-len from these frames.
- trimmed_frame_ids.json file:
  - Contains pre-trimmed and sampled frames of clip length 16
  - Format:
```
{
vid_id1: [frame#, frame#,...],
vid_id2: [frame#, frame#,...],
...}
```

Triplets:

AGQA provides "scene graphs" and object/relationships/action classes.
We iterated through the given scene graphs for each video's frame and created triplets in the form of: (person, relation, object)
- frameTriplets.json:
  - Format:
```
{
vid_id: {frame#: [(triplet), (triplet)], 
         frame#: [(triplet), (triplet)]},
         ...}
```

For the given action classes, we list all actions found in each video frame.

frameActions.json:

Format:

             vid_id: {frame#: [(action), (action)], 
                      frame#: [(action)]
                      },
          ...}

All unique triplets and actions are mapped to an index for easy reference in the code.
- relationship_triplets.json: {(triplet): 0, (triplet): 1, ...}
- action_dictionaries.json: {action: 0, action: 1, ...}

Citation

If this work and/or its findings are useful for your research, please cite our paper.

@misc{khan2023learning,
      title={Learning Situation Hyper-Graphs for Video Question Answering}, 
      author={Aisha Urooj Khan and Hilde Kuehne and Bo Wu and Kim Chheu and Walid Bousselham and Chuang Gan and Niels Lobo and Mubarak Shah},
      year={2023},
      eprint={2304.08682},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

Questions?

Please contact 'aishaurooj@gmail.com'

LimGeunTaekk/SHG-VQA