Code for Hierarchical Multi-modal Fusion Tracker in CVPR2022 paper Visible-Thermal UAV Tracking: A Large-Scale Benchmark and New Baseline, which is a strong baseline for RGB-T tracking.
Three complementary modules are introduced for multi-modal fusion
- Complementary Image Fusion(CIF): CIF aims to use a shared backbone to extract complementary information and Kullback–Leibler divergence loss is introduced to unify the feature distribution.
- Discriminative Feature Fusion(DFF): DFF aims to build individual representations for both modalities and learns a channel-wise modality weight to fuse them.
- Adaptive Decision Fusion(ADF): ADF is to adaptively provide the final response by considering the results of two branches and the modality confidence.
The pretrained model are available at [GoogleDrive] and [BaiduDisk]
conda create -n HMFT python=3.6
conda activate HMFT
cd $Path_to_HMFT$
bash install.sh
cd $Path_to_HMFT$/mfDiMP/pytracking
python demo.py
cd $Path_to_HMFT$/mfDiMP/pytracking
python run_VTUAV.py
python run_GTOT.py
python run_RGBT210.py
python run_RGBT234.py
The results can be found at [GoogleDrive] and [BaiduDisk]
If you find our work useful, please cite
@InProceedings{Zhang_CVPR22_VTUAV, author = {Zhang Pengyu and Jie Zhao and Dong Wang and Huchuan Lu and Xiang Ruan}, title = {Visible-Thermal UAV Tracking: A Large-Scale Benchmark and New Baseline}, booktitle = {Proceedings of the IEEE conference on computer vision and pattern recognition}, year = {2022} }