A Baseline for 3D Multi-Object Tracking

This repository contains the official python implementation for "A Baseline for 3D Multi-Object Tracking". If you find this code useful, please cite our paper:

@article{Weng2019_3dmot, 
  archivePrefix = {arXiv}, 
  arxivId = {1907.03961}, 
  author = {Weng, Xinshuo and Kitani, Kris}, 
  eprint = {1907.03961}, 
  journal = {arXiv:1907.03961}, 
  title = {{A Baseline for 3D Multi-Object Tracking}}, 
  url = {https://arxiv.org/pdf/1907.03961.pdf}, 
  year = {2019} 
}

Overview

News
Introduction
Dependencies
3D Object Detection
3D Multi-Object Tracking
Acknowledgement

News

Aug. 21, 2019: Python 3.5 (3.6?, 3.7?) supported.
Aug. 21, 2019: Results on KITTI "pedestrian" and "cyclist" categories released.
Aug. 19, 2019: A minor bug in orientation correction fixed.
Jul. 9, 2019: Code and results on KITTI "car" category released.

Introduction

3D multi-object tracking (MOT) is an essential component technology for many real-time applications such as autonomous driving or assistive robotics. However, recent works for 3D MOT tend to focus more on developing accurate systems giving less regard to computational cost and system complexity. In contrast, this work proposes a simple yet accurate real-time baseline 3D MOT system. We use an off-the-shelf 3D object detector to obtain oriented 3D bounding boxes from the LiDAR point cloud. Then, a combination of 3D Kalman filter and Hungarian algorithm is used for state estimation and data association. Although our baseline system is a straightforward combination of standard methods, we obtain the state-of-the-art results. To evaluate our baseline system, we propose a new 3D MOT extension to the official KITTI 2D MOT evaluation along with two new metrics. Our proposed baseline method for 3D MOT establishes new state-of-the-art performance on 3D MOT for KITTI, improving the 3D MOTA from 72.23 of prior art to 76.47. Surprisingly, by projecting our 3D tracking results to the 2D image plane and compare against published 2D MOT methods, our system places 2nd on the official KITTI leaderboard. Also, our proposed 3D MOT method runs at a rate of 214.7 FPS, 65 times faster than the state-of-the-art 2D MOT system.

Dependencies:

This code has been tested on python 2.7 and 3.5, and also requires the following packages:

scikit-learn==0.19.2
filterpy==1.4.5
numba==0.43.1
matplotlib==2.2.3
pillow==5.2.0
opencv-python==3.4.3.18
glob2==0.6

One can either use the system python or create a virtual enviroment (virtualenv for python2, venv for python3) specifically for this project (https://www.pythonforbeginners.com/basics/how-to-use-python-virtualenv). To install required dependencies on the system python, please run the following command at the root of this code:

$ pip install -r requirements.txt

To install required dependencies on the virtual environment of the python (e.g., virtualenv for python2), please run the following command at the root of this code:

$ pip install virtualenv
$ virtualenv .
$ source bin/activate
$ pip install -r requirements.txt

3D Object Detection:

For convenience, we provide the 3D detection of PointRCNN on the KITTI MOT dataset at "./data/KITTI/" for car, pedestrian and cyclist splits. Our detection results follow the format of the KITTI 3D Object Detection Challenge (format definition can be found in the object development toolkit here: http://www.cvlibs.net/datasets/kitti/eval_object.php?obj_benchmark=3d) except that the order is switched. We show an example of detection as follows:

Frame	Type	2D BBOX (x1, y1, x2, y2)	Score	3D BBOX (h, w, l, x, y, z, rot_y)	Alpha
0	2 (car)	726.40, 173.59, 917.48, 315.07	13.85	1.56, 1.58, 3.48, 2.57, 1.57, 9.72, -1.56	-1.82

3D Multi-Object Tracking

Inference

To run our tracker on the KITTI MOT validation set with the provided detection:

$ python main.py car_3d_det_val
$ python main.py ped_3d_det_val
$ python main.py cyc_3d_det_val

To run our tracker on the KITTI MOT test set with the provided detection:

$ python main.py car_3d_det_test
$ python main.py ped_3d_det_test
$ python main.py cyc_3d_det_test

Then, the results will be saved to "./results" folder. In detail, results in "./results/data" folder are used for MOT evaluation, which follow the format of the KITTI Multi-Object Tracking Challenge (format definition can be found in the tracking development toolkit here: http://www.cvlibs.net/datasets/kitti/eval_tracking.php). On the other hand, results in "./results/trk_withid" folder are used for visualization only, which follow the format of KITTI 3D Object Detection challenge except that we add an ID in the last column.

Note that, please run the code when the CPU is not occupied by other programs otherwise you might not achieve similar speed as reported in our paper.

3D MOT Evaluation

To reproduce the quantitative results of our 3D MOT system using the proposed KITTI-3DMOT evaluation tool, please run:

$ python evaluation/evaluate_kitti3dmot.py car_3d_det_val
$ python evaluation/evaluate_kitti3dmot.py ped_3d_det_val
$ python evaluation/evaluate_kitti3dmot.py cyc_3d_det_val

Then, the results should be exactly same as below, except for the FPS which might vary across individual machines. Note that the results for car are a little bit better than the results in the paper. Also, we add results for pedestrian and cyclist which are not present in the paper.

Category	AMOTA (%)	AMOTP (%)	MOTA (%)	MOTP (%)	MT (%)	ML (%)	FRAG	FPS
Car	39.48	74.67	76.57	79.16	70.04	7.27	50	207.4
Pedestrian	25.21	49.69	61.19	67.00	36.53	39.52	63	436.6
Cyclist	20.05	59.29	58.47	75.25	56.76	27.03	5	1168.5

Visualization

To reproduce the qualitative results of our 3D MOT system shown in the paper:

Threshold the trajectories using a proper threshold
Draw the remaining 3D trajectories on the images (Note that the opencv3 is required by this step, please check the opencv version if there is an error)

$ python trk_conf_threshold.py car_3d_det_test
$ python visualization.py car_3d_det_test_thres

Visualization results are then saved to "./results/car_3d_det_test_thres/trk_image_vis". If one wants to visualize the results on the entire sequences, please download the KITTI MOT dataset http://www.cvlibs.net/datasets/kitti/eval_tracking.php and move the image_02 (we have already prepared the calib data for you) data to the "./data/KITTI/resources" folder.

In addition, one can check out our demo for viusualization in full_demo.mp4

2D MOT Evaluation

To reproduce the quantitative results of our 3D MOT system using the official KITTI 2D MOT evaluation server for car category shown in the paper, please compress the folder below and upload to http://www.cvlibs.net/datasets/kitti/user_submit.php

$ ./results/car_3d_det_test_thres/data

Then, the results should be similar to our entry on the KITTI 2D MOT leaderboard:

Category	MOTA (%)	MOTP (%)	MT (%)	ML (%)	IDS	FRAG	FPS
Car	83.84	85.24	66.92	11.38	9	224	214.7

Acknowledgement

Part of the code is borrowed from "SORT"

doctorwk007/AB3DMOT