Task-Relevant Failure Detection for Trajectory Predictors in Autonomous Vehicles

This repository contains code for the following paper: https://arxiv.org/pdf/2207.12380.pdf

Installation

Clone the submodules as well when cloning the repository:

git clone --recurse-submodules <repository cloning URL>

There are two ways to use the code, one is by creating a docker image and the other is installation from source. The docker image is recommended and will require less manual setup.

Docker Image (easier)

Navigate to the docker folder: cd docker
Build the docker file: docker build -t <docker-image-name:tag> .. Typically, this will take a while.
The docker can now be run in an interactive session: docker run -it <docker-image-name:tag> /bin/bash

Installation from Source (more involved)

Create a conda environment with python=3.9
Install pytorch conda install pytorch=1.9.0 torchvision=0.2.2 torchaudio=0.9.0 cudatoolkit=10.2 -c pytorch
Install pip install torchmetrics==0.6.0
Install nuplan-devkit=0.1.0
- Clone nuplan-devkit git clone https://github.com/motional/nuplan-devkit.git
- Checkout commit 3c4f5b8dcd517b27cfd258915ca5fe5c54e3cb0c for the appropriate version
- Comment out pytorch related requirements (torch==* and torchvision*) from requirements.txt, they throw an error and are not necessary
- run pip install -e .
- run pip install protobuf==3.20.0
conda install -c conda-forge firefox geckodriver
Navigate back to this repo and pip install -r requirements.txt
Upgrade pathos to 0.2.9: pip install pathos==0.2.9
Clone Rewards-Deterioration-Detection github into ./ and rename as drd
- Comment out L33 of StatsCalculator.py
Clone hj reachability code into ./detection
Make directory target_values in ./detection and saves in ./ and ./nuPlan
Downgrade shapely to 1.8.2: pip install shapely==1.8.2

Setting up Data

Process the dataset on the host as explained here. If using docker, then mount it as a volume using -v flag while running the docker. If building from source, then place the data in the appropriate path as explained below.

nuScenes data
- Download Map Expansion v>=1.2 and Full dataset trainval (only metadata is required) from nuScenes.org. Extract the downloaded zip file's contents and place them in the ./nuScenes/process_data/ directory. Copy the contents of the extracted maps folder into the ./nuScenes/process_data/maps folder. Finally, process them into a data format that our model can work with.
```
cd nuScenes/process_data

# For the mini nuScenes dataset, use the following
python process_data.py --data=./v1.0-mini --version="v1.0-mini" --output_path=../processed

# For the full nuScenes dataset, use the following
python process_data.py --data=./v1.0-trainval_meta --version="v1.0-trainval" --output_path=../processed
```
- Place processed data .pkl files into a folder ./nuScenes/processed/ and maps and icons folders into ./nuScenes/
nuPlan data
- Download Maps and Metadata for all splits (v0.2) from nuScenes.org
- Place .db file and maps folder into a folder ./nuPlan/dataset/

Template command for mounting the data as a volume on the docker image:

docker run \
  -v /on/host/path/to/nuScenes/processed:/home/p-qad/p-qad/nuScenes/processed \
  -v /on/host/path/to/nuScenes/maps:/home/p-qad/p-qad/nuScenes/maps \
  -v /on/host/path/to/nuScenes/icons:/home/p-qad/p-qad/nuScenes/icons \
  -v /on/host/path/to/nuPlan/dataset:/home/p-qad/p-qad/nuPlan/dataset \
  -it <docker-image-name:tag> \
  /bin/bash

If building from source, then place the data in the approriate path as described in the file structure below.

Instructions for reproducing results

We have provided .npy files in the ./saves/ folder from our runs that were used to generate results for the paper. The results can be generated by running process_data.py as mentioned above with num_seeds=5. To simply reproduce the results in the paper, directly jump to step 4 below. To generate your own save files, go through all the steps.

In ./detection/, run python hj_reachability.py to generate the target values
In ./nuScenes/, run python main.py --exp exp --seed seed to save anomaly detection results on nuScenes fixed scenes with exp=1 and seed as desired.
In ./nuPlan/, run python main.py --exp exp --seed seed to save anomaly detection results on nuPlan fixed and reactive scenes with the following changes:
- exp=1 runs anomaly detection on reactive nuPlan scenarios from mini dataset
- exp=2 runs anomaly detection on fixed-plan nuPlan scenarios
- exp=3 runs anomaly detection on reactive nuPlan scenarios from full dataset
- exp=4 runs anomaly detection for adaptive replanning rate
In ./detection/, run python process_data.py --exp exp --num_seeds num_seeds to generate the ROC plot and information in the tables with the following changes:
- exp=1 generates info in Table 3
- exp=2 generates Fig 6 and info in Table 4
- exp=3 generates Fig 3 and info in Table 2
- exp=4 generates info in Table 1

Resulting file structure and explanation of files/folders

./detection/:
- anomaly_detector.py: Utility functions which run anomaly detection
- cost_functions.py: Cost functions
- cost_util.py: Class for computing planning and fixed plan costs
- dynSimpleCarAvoid.py: Class for relative dynamics used in hj_reachability.py
- hj_reachability.py: Computes target values for HJ reachability detection
- process_data.py: Ploting utilities (used to make ROC figure in paper and compute FPR/FNR)
- query_utils.py: Utilities for getting states of the ego/other agents
- query.py: Class for types of states of ego/other agents
- plots/: Location of images saved from process_data.py
- target_values/: Location of saved target values from hj_reachability.py
- hj_reachability/: Clone of hj reachability code
./drd: Renamed detecting reward deterioration code
./models/: Trajectron++ model
./nuPlan/:
- labels.py: Hand labels for nuPlan reactive scenarios
- main.py: Runs anomaly detectors and plotting for various scenes (saves into ../saves)
- nuBoard.py: Visualization utility (used to make gifs of nuPlan scenarios)
- nuplan_trajectron_utils.py: Utilities which handle conversion between nuPlan and trajectron
- nuplan_utils.py: Default values for nuPlan
- planner.py: Class for planning nuPlan scenarios (can run both reactive planner and expert plan)
- plot.py: Plotting utilities (used to generate images for hand-labeling)
- primitives.py: Class for generating motion primitive tree for planner
- process_scenarios.py: Loads in scenes from the database and processes them into a simple format
- nuplan-devkit/: https://github.com/motional/nuplan-devkit (can be located anywhere)
- override/: Classes/functions which override nuPlan-devkit classes and functions
- plots/: Location of saved gifs/images
- dataset/: Location to place nuPlan data
  - maps/: nuPlan maps
  - nuplan_v0.2.db: Database of nuPlan scenarios
  - nuplan_v0.2_mini.db: Mini database of nuPlan scenarios
./nuScenes/
- labels.py: Hand labels for nuScenes validation data
- main.py: Runs anomaly detectors and plotting for various scenes (saves into ../saves)
- plot.py: Plotting utilities (used to generate images with Trajectron++ predictions as contour plots)
- icons/: Images used in plotting
- maps/: nuScenes maps used in plotting and trajectory prediction
- plots/: Location of saved gifs/images
- process_data/: Contains scripts for processing nuScenes data. Borrowed from a variation of Trajectron++
  - main.py
  - kalman_filter.py
  - helper.py
  - evaluate.py
  - devkit/: nuScenes devkit submodule checked out at commit 1050c3d11b5413a1fc5ca4e73a9e426747263297
- processed/: Processed nuScenes for use with Trajectron++, files are .pkl
./saves: Location of saved anomaly detections (used by ./detection/process_data.py)
./trajectron/: Trajectron++ source code
adav.yml: YML file for checking installation and debugging

License

The source code is released under the NSCL licence. The preprocessed dataset and pretrained models are under the CC BY-NC-SA 4.0 licence.

NVlabs/pred-fail-detector