/RGBT-Tracking-Results-Datasets-and-Methods

A personal investigation for RGBT tracking.

Apache License 2.0Apache-2.0

RGBT-Tracking-Results-Datasets-and-Methods

An investigation for RGBT tracking. Hopefully, it can help other researchers become familiar with multi-modal tracking as soon as possible. This repository is started on 02/07/2023, and will keep on updating.


This repository will give a detail investigation of the RGBT tracking community, including the Datasets, Results, and the Methods.

  • Datasets
  • Results
  • Methods
  • Competitions
  • ...

Survey Papers

  • Multi-modal visual tracking: Review and experimental comparison. Zhang, Pengyu, Dong Wang*, and Huchuan Lu. Computational Visual Media 2024. [Paper]
  • RGBT tracking: A comprehensive review. Mingzheng Feng, Jianbo Su*. Information Fusion 2024. [Paper]
  • Review and Analysis of RGBT Single Object Tracking Methods: A Fusion Perspective. Zhihao Zhang, Jun Wang, Zhuli Zang, Lei Jin, Shengjie Li, Hao Wu*,Jian Zhao, Zhang Bo. ACM TOMM 2024. [Paper]
  • Object fusion tracking based on visible and infrared images: A comprehensive review. Xingchen Zhang, Ping Ye, Henry Leung, Ke Gong, Gang Xiao*. Information Fusion 2020. [Paper]
  • A Survey for Deep RGBT Tracking. Zhangyong Tang, Tianyang Xu, and Xiao-Jun Wu*. Arxiv 2022. [Paper]

Datasets

Real Data

Dataset Publish GitHub Introduction
OTCBVS CVIU'2007 OTCBVS 9 videos
LITIV CVIU'2012 LITIV 6 videos
GTOT TIP'2016 GTOT 50 Video pairs, about 1.5W frames
RGBT210 ACM MM'2017 RGBT210 210 Video pairs
RGBT234 PR'2018 RGBT234 234 Video pairs, the extension of RGBT210
VOT-RGBT19 VOT Community:2019 VOT-RGBT2019 CODE:TZYD 60 Video pairs, about 2W frame pairs, a sub-set of RGBT 234
VOT-RGBT20 VOT Community: 2020 VOT-RGBT2020 CODE:TZYD 60 Video pairs, its data is the same with VOT-RGBT2019, but with different evaluation mechanism
LasHeR TIP'2021 LasHeR train/test split, imaged in the head-up view, above 70W image pairs in total; 245 videos in the test set
VTUAV CVPR'2022 VTUAV imaged in the top-down view by uav, long-term;176 videos in the test-st, which has around 3300 frames in average
MV-RGBT Arxiv'2024 MV-RGBT 122 video pairs. The data in this benchmark mainly focus on the modality validity.

Generated Data

Dataset Publish GitHub Introduction
LSS-Dataset(from RGB) TCSVT'2021 LSS-Dataset ,code(Ye5Q) Generated from VID
LSS-Dataset(from TIR) TCSVT'2021 LSS-Dataset,code(IHws) Generated from TIR

Results

GTOT RGBT210 RGBT234 LasHeR
Methods Venue Speed PR SR PR SR PR SR PR NPR SR
TBSI-E TPAMI'2024 58.3/3080Ti 91.0 67.0 75.5 71.5 59.6
S2OTformer ACMTOMM'2024 40/3090Ti 83.1 70.2 68.4 47.7 39.8 29.5
STMT Arxiv'2024 39.1/3090Ti 83.0 59.5 86.5 63.8 67.4 63.4 53.7
MFATrack PR'2024 25/3090 81.7 57.5 63.3 49.8
CFBT Arxiv'2024 87.7 63.0 89.9 65.9 73.2 69.5 58.4
SiamSCR IEEE Sensor Journal'2024 37.2 91.7 73.7 79.1 57.5 52.2 40.1
SiamMGT The Journal of Supercomputing'2024 90.9 73.9 81.1 60.4 54.8 41.6
AINet arxiv'2024 37.5/4090 87.5 64.8 89.2 67.3 74.2 70.1 59.1
MambaVT ARxiv'2024 31.1/A100 95.2 78.6 88.5 64.4 90.7 67.5 72.7 69.0 57.5
AMAtrack TIM'2024 68.3/RTX3070 82.6 61.2
CAFormer ARxiv'2024 83.6/RTX3080Ti 91.8 76.9 85.6 63.2 88.3 66.4 70.0 66.1 55.6
VLCTrack NeuroComputing'2024 90.9/RTX4090 90.1 76.8 81.1 59.1 56.4 51.8 43.7
MixRGBX NeuroComputing'2024 85.3 63.2 67.2 53.2
TGTrack TCSVT'2024 87.2 64.1 89.3 67.0 73.2 69.1 57.9
CKD ACMMM'2024 96.4/4090 92.5 75.1 88.4 65.2 90.0 67.4 73.2 69.4 58.1
IIMF ACMMM'2024 85.6 62.4 86.8 64.4 72.4 68.4 58.1
MMSTC TIP'2024 94.1 77.9 88.6 65.7 89.8 67.3 72.3 68.6 57.4
MELT ACMTOMM'2024 20 75.3 54.6 57.3 52.7 45.0
TIH PR'2024 72 93.5 77.0 89.4 66.4 72.0 67.7 57.2
AMRT electronics'2024 32.31 90.0 73.0 84.4 60.2 46.8 32.4
XTrack Arxiv'2024 - - - 84.8 62.2 65.5 52.5
SeqTrackv2 Arxiv'2024 - 68.5 92.3 76.7 61.0
KSTrack TCSVT'2024 - 95.5 74.1 85.0 58.7
MINet IVC'2024 - 84.3 62.9 65.9 52.9
BAT AAAI'2024 - - - 86.8 64.1 70.2 56.3
TATrack AAAI'2024 26.1/RTX3090 - - 85.3 61.8 87.2 64.4 70.2 66.7 56.1
GMMT AAAI'2024 18/RTX3090Ti 93.6 78.5 - - 87.9 64.7 70.7 67.0 56.6
LMINet IEEE Sensors Journal'2024 5.6/V100 91.4 73.6 - - 83.8 59.2 49.0 43.3 34.8
CAT++ TIP'2024 14/2080Ti 91.5 73.3 82.2 56.1 84.0 59.2 50.9 44.4 34.8
AMNet TCSVT'2024 14/2080Ti 92.9 77.4 - - 87.7 65.5 70.2 66.0 55.9
MPLKD Arxiv'2024 81.8/RTX3090 92.4 77.3 - - 87.3 65.1 71.4 - 56.7
QueryTrack< TIP'2024 27/RTX A6000 92.3 75.9 - - 84.1 60.0 66.0 - 52.0
MCTrack TCSVT'2024 - 94.1 77.6 86.4 63.7 87.5 65.6 71.6 67.6 57.1
SDSTrack CVPR'2024 21/RTX3090Ti - - - - 84.8 62.5 - 66.5 53.1
OneTracker CVPR'2024 - - - - - 85.7 64.2 - 67.2 53.8
UnTrack CVPR'2024 - - - - - 84.2 62.5 66.7 - 53.6
UStrack IJCAI'2024 84.2 93.4 78.3 87.4 65.8
AFter Arxiv'2024 23/RTX 4090 91.6 78.5 87.6 63.5 90.1 66.7 70.3 65.8 55.1
CSTNet Arxiv'2024 - - - 86.0 63.5 88.4 65.2 71.5 67.9 57.2
CAFF JSEE'2024 - 85.2 72.6 71.5 49.2 74.0 50.6 - - -
ASFNet 2024 - 91.7 73.9 - * 84.1 57.3 51.1 - 35.9
IPL Arxiv'2024 - - - 88.3 65.7 69.4 65.6 55.3
MPT TCSVT'2024 - - - - - 76.9 55.0 35.5 - 31.3
MIGTD PR'2024 - - - - - 87.6 65.8 72.0 68.1 57.2
Proformer TCSVT'2024 22 - - 86.8 62.2 89.9 65.7 67.4 63.0 53.3
AWCM TOMM'2024 5/1080Ti 89.6 73.8 - - 79.6 56.9
JTPMA INFFUS'2023 15 90.7 75.1 80.3 56.2 53.8 48.2 37.3
FDAFT PRCV'2023 88.2 66.4 68.9 - 55.3
DFAT INFFUS'2023 22/RTX2080Ti 0.893 0.723 - - 0.758 0.552 - - 0.336
ViPT CVPR'2023 - - - - - 0.835 0.617 0.651 - 0.525
CMD CVPR'2023 30/RTX1080Ti 0.892 0.734 - - 0.824 0.584 0.590 0.546 0.464
TBSI CVPR'2023 36/RTX3080Ti - - 0.853 0.625 0.871 0.637 0.692 0.657 0.556
BD2Track SPL'2023 - - - - - - - 0.560 - 0.432
EANet ICMV'2023 - - 83.5 58.4 50.6 - 36.7
SiamTDR TICPS'2023 127/RTX3090 88.5 71.4 - - - - - - -
SiamAFTS ScienticReport'2023 - 84.9 77.7 87.3 56.4 89.0 60.2 - - -
UMT AIEA'2023 3/GTX1080Ti 89.2 73.0 84.2 60.3 49.3 43.0 36.9
MPDMT ICPR'2023 40/RTX3080 90.6 75.7 80.6 58.3
RSFNet SPL'2023 83/RTX3090 92.1 75.3 - - 86.3 62.2 65.9 - 52.6
QAT ACMMM'2023 22/ 91.5 75.5 86.8 61.9 88.4 64.3 64.2 59.6 50.1
PRO CCDC'2023 - - 68.0 47.8 - - - - -
DFNet TIST'2023 88.1 71.9 - - 77.2 51.3 - - -
DTAN Cognitive Computation'2023 0.5 88. 71.5 - - 78.2 55.1 57.5 51.6 41.0
DECFNet ICMA'2023 27.6 90.4 70.8 - - - - 49.6 - 35.0
DMSTM TIM'2023 92.9 75.9 - - 78.6 56.2 54.1 49.4 40.0
SiamFEA JVCIR'2023 92.0 76.6 - - 83.7 61.7 64.5 - 50.9
HATFNet Applied Intelligence'2023 9.8 91.8/td> 77.3 - - 80.7 59.5 - - -
MACFT Sensors'2023 31 -/td> - - - 85.7 62.2 65.3 - 51.4
LCMIT SCIENCE CHINA Information Sciences'2023 93.9/td> 74.1 - - 83.1 57.8 - - -
MMMPT IPT'2023 90.3/td> 73.6 - - 79.8 54.6 53.0 - 40.0
MPDMT ICPR'2023 42 90.6/td> 75.7 - - 80.7 58.3 53.0 - 40.0
FSBNet Neural Computing and Applications'2023 27 91.3/td> 72.1 - - 82.4 56.2 53.7 - 42.5
MTNet ICME'2023 - 93.5/td> 76.0 - - 85.0 61.9 60.8 - 47.4
SiamMFF CACML'2023 - 90.5/td> 73.6 - - 81.2 57.3 60.8 - 47.4
LSAR TCSVT'2023 - 85.5/td> 70.3 77.5 54.9 78.4 55.9 46.0 - 38.5
CSMMA Sensors Journal'2023 20 89.5/td> 73.0 - - 77.8 53.2 - - -
MPLT Arxiv'2023 -/td> - 86.2 63.0 88.4 65.7 72.0 - 57.1
CRSP ICIP'2023 - 90.4 77.2 74.0 63.2 - - - - -
GABBP TNNLS'2023 - 90.3 77.2 68.6 68.5 - 59.7 - - -
PLASSO-ADSPF KBS'2023 63 - - - - 82.7 59.2 54.7 - 47.4
SRCDCF IPT'2023 53 83.0 65.3 66.3 47.5 66.9 47.8 - - -
MFENet Multimedia Tools and Applications'2023 1.43 89.8 71.9 - - 83.2 57.1 51.7 - 37.4
RMFNet Applied Sciences'2023 10 91.1 76.9 - - 79.4 58.5 - - -
M2GCI NPL'2023 90.9 73.7 - - 79.3 56.9 - - -
SiamIVFN JMLC'2023 147 91.5 79.3 - - 81.1 63.2 - - -
STRT Applied Intelligence'2023 15 91.3 74.4 80.2 56.8 83.6 60.4 63.6 59.4 50.0
SGF_MDNet+RGBT IET Image Processinge'2023 73.8 56.3
SiamCAF Remote Sensing'2023 90.6 73.0 77.1 53.7
TEFNet PRCV'2023 25.5 63.1 59.1 50.1
DAPF Multimedia Tools and Applications'2023 1.16 - - 86.0 62.0 71.1 51.6
UDT-FF Multimedia Tools and Applications'2023 71.9 79.3 65.1 - 56.1 41.4
UMT AIEA'2023 89.2 73.0 - 84.2 60.3 49.3 43.0 36.9
CSRDCF_RGBT IEEE SENSORS JOURNAL'2023 - 70.7 51.4
APFNet AAAI'2022 - 0.905 0.739 - - 0.827 0.579 0.500 - 0.362
TAAT arxiv'2022 39/RTX2080Ti 0.858 0.696 0.710 0.486 - - - - -
DMCNet TNNLS'2022 2.38/RTX2080Ti 0.909 0.733 0.797 0.555 0.839 0.593 0.490 0.431 0.355
MFGNet TMM'2022 3.37/Tesla P100 0.889 0.707 0.749 0.494 0.783 0.535
HMFT CVPR'2022 30/GTXTitan 0.912 0.749 0.786 0.535 0.788 0.568 0.460 0.413 0.326
ProTrack ACMMM'2022 - - - - 78.6 58.7 50.9 42.1
AGMINet TIM'2022 8.8 91.7 73.4 - - 84.0 59.2 48.8 42.9 34.3
CMC2R IET Image Processing'2022 - - 73.2 52.6
DuSiamRT The Visual Computer'2022 116 76.6 62.8 - - 56.7 38.4
SiamCSR IGARSS'2022 140 88.2 70.9 - - 75.4 53.2
LRMWT KBS'2022 24.6 91.1 75.3 80.6 59.2 82.5 61.6
MIRNet ICME'2022 90.9 74.4 81.6 58.9
MBAFNet IEEE Sensors Journal'2022 15.2 91.6 76.7 80.1 58.5
RSF Science China Information Sciences'2022 86.4 70.7 66.6 48.4
CF-LG IVC'2022 32.29 94.7 65.5 71.5 49.0 71.7 49.3
MFNet IVC'2022 19 90.7 73.5 71.5 49.0 84.4 60.1 59.7 55.4 46.7
RLTN JEI'2022 1.06 89.3 73.8 77.1 55.5
TFNet TCSVT'2022 20 88.6 72.9 77.7 52.9 80.6 56.0
SCA_MMA ELECTRONICS'2022 1.3 90.5 73.2 80.2 56.9
SiamMFF Machine Vision and Applications'2022 85.8 71.1
JMMAC TIP'2021 4/RTX2080Ti 90.2 73.2 - - 79.0 57.3 - - -
MANet++ TIP'2021 - 90.1 72.3 - - 80.0 55.4 46.7 40.4 31.4
CBPNet TMM'2021 - 88.5 71.6 - - 79.4 54.1 - - -
CEDiMP Sensors'2021 - 88.6 73.1 - - 80.8 56.1 - - -
FANet TIV'2021 19/RTX2080Ti 89.1 72.8 - - 78.7 55.3 44.1 38.4 30.9
ADRNet IJCV'2021 25/RTX2080Ti 90.4 73.9 - - 80.9 57.1 - - -
M^5L TIP'2021 - 0.896 0.710 - - 0.795 0.542 - - -
SiamCDA TCSVT'2021 - 0.877 0.732 - - 0.760 0.569 - - -
EStaple JCSC'2021 44/AMD3950 - - - 0.648 0.470 - - -
AFCF TITS'2021 93.1 64.0 65.7 46.9 66.4 47.2 - - -
HDINet IEEE Sensors Journal'2021 88.8 71.8 78.3 55.9 - - -
NRCMR TNNLS'2021 7.0 83.7 66.4 72.9 50.2 - - -
CMMP Neurocomputing'2021 35.0 86.9 71.1 75.1 49.1 - - -
MaCNet Sensors'2020 - 0.880 0.714 - - 0.790 0.554 0.482 0.420 0.350
CMPP CVPR'2020 1.3/RTX2080Ti 0.926 0.738 - - 0.823 0.575 - - -
CAT ECCV'2020 - 0.889 0.717 0.792 0.533 0.804 0.561 0.450 0.395 0.314
DSiamMFT 2020 14.7/RTX1080Ti - - 0.642 0.432 - - - - -
LDUA-STRCF JVCIR'2020 12.65 - - 76.0 54.6 78.3 58.1 - - -
MECF ICMCCE'2020 11 - - 69.0 50.2 - - -
mfDiMP ICCVW'2019 28/GTXTitan - - 0.786 0.555 0.785 0.559 0.448 0.395 0.343
MANet ICCVW'2019 - 0.894 0.724 - - 0.777 0.539 0.457 - 0.330
DAPNet ACM MM'2019 - 0.882 0.707 - - 0.766 0.537 0.431 - 0.314
DAFNet ICCVW'2019 - 0.891 0.712 - - 0.796 0.544 0.448 0.390 0.311
SiamFT IEEE Access'2019 30/RTX1080Ti 0.826 0.700 - - 0.688 0.486 - - -
CMCFT Neurocomputing'2019 77.0 63.2 52.9 36.6 - - -
LGMG TCSVT'2019 7 82.9 65.5 71.1 46.8 0.688 0.486 - - -
LTODA Arxiv'2019 84.3 67.7 76.7 54.5 - - -
fusionnet NeuroComputing'2018 - 85.2 62.6 - - - - - - -
CMR ECCV'2018 - 82.7 64.3 69.4 46.3 - - - - -
SCCF PRCV'2018 - 85.0 68.1 - - - - -
LMDCN CISP'2018 - 61.7 38.7 - - -
TMGRMR Signal Processing: Image Communication'2018 7 84.2 62.2 - - -

VTUAV/VOT-RGBT2019/2020

VTUAV VOT-RGBT2019 VOT-RGBT2020
Methods Venue Speed PR NPR SR R A EAO R A EAO
TBSI-E TPAMI'2024 58.3/3080Ti 88.1 73.2
S2OTFormer ACMTOMM'2024 40/3090Ti 56.7 44.5
AINet arxiv'2024 37.5/4090 88.0 75.3
CAFormer Arxiv'2024 83.6/RTX3080Ti 88.6 76.2
CKD ACMMM'2024 96.4/4090 90.2 77.8
MMSTC TIP'2024 83.9 67.7
MELT ACMTOMM'2024 20 68.3 55.6 83.2 65.2 50.8
MIGTD PR'2024 26.9 88.3 - 75.5 - - - - - -
Proformer TCSVT'2024 22 84.6 - 71.0 - - - - - -
IPL Arxiv'2024 - 87.5 - 75.6 - - - - - -
MPT TCSVT'2024 - 66.1 - 54.1 57.2 79.1 35.6 - - -
MPLKD Arxiv'2024 81.8/RTX3090 80.2 - 67.4 - - - - - -
QueryTrack TIP'2024 27/RTX A6000 75.5 - 63.3 - - - - - -
UStrack IJCAI'2024 84.2 86.9 74.4
AFter Arxiv'2024 RTX 4090/23 84.9 - 72.5 - - - - - -
CAFF JSEE'2024 - - - - 56.9 52.9 30.0 - - -
DBDFAN Drones'2023 31.2 64.2 - 51.6 - - - - - -
DFAT INFFUS'2023 22/RTX2080Ti - - - 0.670 0.756 0.413 0.672 0.779 0.390
DTAT JIPS'2023 61.3 - 51.2. - - - - - -
MACFT Sensors'2023 31 80.1/td> - 66.8 - - - - - -
LSAR TCSVT'2023 8 42.7/td> - 36.3 81.1 61.6 40.0 - - -
CSMMA Sensors Journal'2023 20 - - - 66.1 60.5 38.7 - - -
STRT Applied Intelligence'2023 15/GTX1080Ti 69.2 - 57.2 - - - - - -
LCMIT SCIENCE CHINA Information Sciences'2023 -/td> - - 71.6 62.5 38.1 - - -
MMMPT IPT'2023 73.1/td> - 60.2 - - - - - -
PLASSO-ADSPF KBS'2023 63 - - - - - - 81.2 67.0 49.9
SRCDCF IPT'2023 53 - - - - - - 72.0 61.7 37.0
SiamCAF Remote Sensing'2023 67.0 54.1
QAT ACMMM'2023 22/ 80.1 - 66.7 - - - - - -
TEFNet PRCV'2023 25.5 75.8 64.9
CSRDCF_RGBT IEEE SENSORS JOURNAL'2023 65.3 56.1
TAAT arxiv'2022 39/RTX2080Ti 0.696 - 0.560 0.643 0.735 0.416 - - -
HMFT CVPR'2022 30/GTXTitan 0.758 - 0.627 0.553 0.760 0.347 - - -
TFNet TCSVT'2022 20 59.4 46.2 28.8
JMMAC TIP'2021 4/RTX2080Ti - - - 0.665 0.821 0.483 0.662 0.818 0.420
MANet++ TIP'2021 - - - - 0.509 0.538 0.272 - - -
FANet TIV'2021 19/RTX2080Ti - - - 0.472 0.508 0.247 - - -
ADRNet IJCV'2021 25/RTX2080Ti 0.622 - 0.466 0.622 0.766 0.396 - - -
SiamCDA TCSVT'2021 - - - - 0.682 0.757 0.424 - - -
MaCNet Sensors'2020 - - - - 0.545 0.591 0.305 - - -
mfDiMP ICCVW'2019 28/GTXTitan 0.673 - 0.554 0.602 0.804 0.388 0.638 0.793 0.380
MANet ICCVW'2019 - 0.455 0.383 0.326 0.582 0.701 0.346 - - -
DAPNet ACM MM'2019 - 0.431 0.383 0.314 - - - - - -
DAFNet ICCVW'2019 - 0.620 - 0.458 - - - - - -
FSRPN 2019 - 0.653 - 0.544 - - 0.355 - - -
SiamFT IEEE Access'2019 30/RTX1080Ti - - - 0.630 0.639 0.310 - - -

Methods


Competitions

  1. AntiUAV 1st The first AntiUAV challenge is a multi-modal challenge. It was held in 2020. The website for the 1st AntiUAV is covered by the newest one. The winner is team xiaobaibai. One of its member is Tianyang Xu
  2. VOT-RGBT2019 The first competition in the RGBT tracking community. It was held in 2019. The winner is SiamDW.
  3. [VOT-RGBT2020] A competition in the RGBT tracking community. It was held in 2020. The winner is DFAT. One of its member is Zhangyong Tang

Contributors

Questions

If you have any questions, please contact zhangyong_tang_jnu@163.com, and wechat: Tzy18861871359 is also welcomed.