RoadScene2Graph (RS2G)

Description

This repository contains the code for our paper titled "RS2G: Data-Driven Scene-Graph Extraction and Embedding for Robust Autonomous Perception and Scenario Understanding"

The repository is based on the structure and code from roadscene2vec, available here.

The main changes are the addition of the RS2G model, the RS2G_Trainer, and the RS2G execution script.

Configuration parameters for the RS2G model as well as the baselines are available in config/. All hyperparameter tuning can be done via changes to these config files.

Please cite our paper if you find our code or paper useful for your research:

@article{malawade2023rs2g,
      title={RS2G: Data-Driven Scene-Graph Extraction and Embedding for Robust Autonomous Perception and Scenario Understanding}, 
      author={Arnav Vaibhav Malawade and Shih-Yuan Yu and Junyao Wang and Mohammad Abdullah Al Faruque},
      year={2023},
      journal={arXiv preprint arXiv:2304.08600}
}

Installation

Installation follows the same procedure as roadscene2vec. Please follow the installation instructions for that library located here. At a minimum, you will need Python 3, PyTorch, and PyTorch Geometric.

Datasets

Our synthetic datasets can be downloaded at the following link: https://drive.google.com/drive/folders/1Zpzfvt_4jlgEiI8eE0HdICu4xkoug5jq?usp=sharing. We do not have permission to publicly share the HDD dataset or the Traffic-Anomaly dataset. If you would like to use these datasets or others, please follow the instructions from roadscene2vec for real-image scene-graph extraction.

Model Training and Evaluation

The execution script can be run as follows for each model:

#Rule-Based MRGCN baseline
python 3_train_model.py --yaml_path ../config/rule_graph_risk_config.yaml  

#CNN+LSTM baseline
python 3_train_model.py --yaml_path ../config/image_learning_config.yaml

#RS2G
python 6_train_rs2g_model.py --yaml_path ../config/rs2g_graph_risk_config.yaml

Transfer Learning

To run transfer learning experiments, the following commands can be used: