Single-camera and Inter-camera Vehicle Tracking and 3D Speed Estimation Based on Fusion of Visual and Semantic Features (AI City Challenge Workshop in CVPR 2018)
This repository contains our source code of Track 1 and Track 3 at the AI City Challenge Workshop in CVPR 2018.
The source code of Track 1 is built in MATLAB and C++, with our trained YOLOv2 model provided.
The source code of Track 3 is developed in Python and C++.
Both Track 1 and Track 3 have been tested on Linux and Windows. Dependencies include CUDA, cuDNN and OpenCV.
Our UW team members include Zheng(Thomas) Tang, Gaoang Wang, Hao(Alex) Xiao, Aotian Zheng.
[Paper], [Slides], [Poster], [Project Page], [2018 NVIDIA AI City Challenge]
Sorry about the inconvenience, but the datasets for the AI City Challenge Workshop at CVPR 2018 is no longer available to the public. Instead, the 2019 CVPR AI City Challenge Workshop has just been launched. This year they have a unique city-scale dataset for multi-camera vehicle tracking as well as image-based re-identification. They also have a new dataset for traffic anomaly detection. The scale of the dataset and the number of vehicles that are being used for evaluation are both unprecedented.
To access the new datasets, each participant will need to download a data release form at the AI City Challenge website and participate in the challenge. Please forward your inquiry to aicitychallenge2019@gmail.com for questions and comments.
The NVIDIA AI City Challenge Workshop at CVPR 2018 will specifically focus on ITS problems such as
- Estimating traffic flow characteristics, such as speed
- Leveraging unsupervised approaches to detect anomalies caused by crashes, stalled vehicles, etc. This is the only way to get the humans in the loop pay attention to meaningful visual information
- Multi-camera tracking, and object re-identification in urban environments
Our team participated in 2 out of 3 tracks:
- Track 1 - Traffic Flow Analysis - Participating teams submit results for individual vehicle speed for a test set containing 27 1-minute videos. Performance is evaluated based on ground truth generated by a fleet of control vehicles that were driven during the recording. Evaluation for Challenge Track 1 is based on detection rate of the control vehicles and the root mean square error of the predicted control vehicle speeds.
- Track 3 - Multi-camera Vehicle Detection and Reidentification - Participating teams identify all vehicles that are seen passing at least once at all of 4 different locations in a set of 15 videos. Evaluation for Challenge Track 3 is based on detection accuracy and localization sensitivity for a set of ground-truth vehicles that were driven through all camera locations at least once.
Detailed information of this challenge can be found here.
Our team achieves rank #1 in both Track 1 and Track 3. The demo videos can be viewed here.
In SCT, our loss function consists of motion, temporal and appearance attributes. Especially, a histogram-based adaptive appearance model is designed to encode long-term appearance change for enhanced robustness. The change of loss is incorporated with a bottom-up clustering strategy for the association of tracklets. Robust 2D-to-3D backprojection is achieved with EDA optimization applied to camera calibration for speed estimation.
The proposed appearance model together with DCNN features, license plates, detected car types and traveling time information are combined for the computation of cost function in ICT.
Under the ./Track1/ folder, there are 6 software packages:
VDO2IMG_IPL: Converting each video file to a folder of frame imagesCAM_CAL_IPL: Manual camera calibration based on minimization of reprojection error and EDA optimizationYOLO_VEH_IPL: Extension of the YOLOv2 object detector with our trained model for vehicle detection/classificationTC_tracker: Proposed tracklet-clustering-based tracking methodAPP_MDL_IPL: Extraction of histogram-based adaptive apperance models and their comparisonSPD_EST_IPL: Speed estimation based on input of tracking results and camera parameters
Detailed description of each package is given in each subfolder.
Under the ./Track3/ folder, there are 3 software packages:
Multi-Camera Vehicle Tracking and Re-identification: Multi-camera vehicle tracking based on a fusion of histogram-based adaptive appearance models, DCNN features, detected car types and traveling time informationYOLO_LP_IPL: Detection of license plate from each cropped vehicle imageLP_COMP_IPL: Comparison of license plates under low resolution
Detailed description of each package is given in each subfolder.
The output of Multi-Camera Vehicle Tracking and Re-identification is the similarity score between each pair of vehicles for comparison. We can convert it into a distance score by inverse proportion. The output of LP_COMP_IPL is the distance score between each two license plates. The final distance score between two vehicles is the multiplication of the above two distance scores. Several vehicle pairs that enjoy low distance scores and appear in all 4 camera locations are submitted to Track 3 for evaluation.
Please cite these papers in your publications if it helps your research:
@inproceedings{tang2018vehicle,
author = {Zheng Tang and Gaoang Wang and Hao Xiao and Aotian Zheng and Jenq-Neng Hwang},
booktitle = {CVPR Workshop (CVPRW) on the AI City Challenge},
title = {Single-camera and Inter-camera Vehicle Tracking and 3D Speed Estimation Based on Fusion of Visual and Semantic Features},
year = {2018},
pages = {108--115}
}
@inproceedings{tang2017vehicle,
author = {Zheng Tang and Gaoang Wang and Tao Liu and Young-Gun Lee and Adwin Jahn and Xu Liu and Xiaodong He and Jenq-Neng Hwang},
booktitle = {arXiv preprint arXiv:1708.06831},
title = {Multiple-kernel Based Vehicle Tracking Using 3D Deformable Model and Camera Self-Calibration},
year = {2017}
}
For any question you can contact Zheng (Thomas) Tang.