/METEOR

Code for the METEOR dataset containing data loading scripts along with pre-trained models for 2D object detection and action-behavior prediction

Primary LanguageJupyter Notebook

METEOR

This repository contains the official code for the paper: METEOR: A Massive Dense & Heterogeneous Behavior Dataset for Autonomous Driving.

Rohan Chandra, Mridul Mahajan, Rahul Kala, Rishitha Palugulla, Chandrababu Naidu, Alok Jain, and Dinesh Manocha

Under Review at ICRA 2022 | Preprint | Project Page

Pre-requisites

This repository is based on MMDetection and MMAction2.

Download the Dataset

Download the dataset here.

Dataset Preparation

  1. Run the following command to extract frames from the videos:

    python videos2frames.py --videos_directory /path/to/videos_dir --output_directory /path/to/output_dir
  2. Run the following command to reorganize the dataset into COCO format for object detection:

    python xml2coco.py --dataset_directory /path/to/dataset_dir --output_directory /path/to/output_dir
  3. Run the following command to reorganize the dataset into rawframe annotation format for behavior prediction:

    python xml2rawframe.py --dataset_directory /path/to/dataset_dir --output_directory /path/to/output_dir

Config Preparation

  1. Download the pre-trained DETR model from link.

  2. Download the pre-trained TSN model from link.

  3. Update the paths to the base config file, train annotations file, the test annotations file, and the pre-trained model in detr_config.py and tsn_config.py.

Training

  1. Run the following command to initiate the training process for DETR:

    python mmdetection/tools/train.py detr_config.py
  2. Run the following command to initiate the training process for TSN:

    python mmaction2/tools/train.py tsn_config.py

Pre-trained Models

Testing

  1. Run the following command to initiate the testing process for DETR:

    python mmdetection/tools/test.py detr_config.py detr_pretrained.pth --eval bbox mAP recall
  2. Run the following command to initiate the testing process for TSN:

    python mmaction2/tools/test.py tsn_config.py tsn_pretrained.pth --eval mmit_mean_average_precision

Swin Detection (Scripts/Swin-Transformer-Object-Detection)

  1. Installation (https://github.com/open-mmlab/mmdetection/blob/master/docs/en/get_started.md)

  2. DATA (Json file): Change the frame path in json file to your own path: e.g. "file_name": "/scratch0/xijunwang/data/METEOR_Dataset/Frame_XML_Annotations/REC_2020_10_12_01_26_57_F/frame_000000.JPG" -> "file_name": "/xxx/REC_2020_10_12_01_26_57_F/frame_000000.JPG"

  3. Train:

    tools/dist_train.sh configs/swin/mask_rcnn_swin_tiny_patch4_window7_mstrain_480-800_adamw_1x_metor.py 8 --cfg-options model.pretrained=<PRETRAIN_MODEL>
  4. Reference