/RGBT

Primary LanguagePython

RGB and Thermal Feature-level Sensor Fusion (RGBT)

Download the FLIR pre-processed dataset here

ToDo List

  • Finish literature review of both Object Detection and Fusion
  • Find the conversion Matrix btw. RGB and IR Frame
  • Prepare the FLIR Dataset for fusion
  • Implement Scaled_Yolov4
  • Test the RGB frames from FLIR Dataset on Scaled_Yolov4 (Baseline)
  • Ablation Study for the Baseline
  • Dive deeper into SPP, PAN, FPN, DCN, SAM, GIOU and other relevant papers for Fusion
  • Design the Fusion Network
  • Vanilla Fusion Training
  • CBAM Fusion Training
  • Entropy-based Channel Attention Module (EBAM)
  • Entropy-based Spatial Attention Module
  • Entropy-based Channel & Spatial Attention Module
  • EBAM Fusion Training
  • Day & Night Analysis
  • Synthetic Fog Analysis
  • TensorRT and Latency Results
  • Installation and User Guidlines
Model Test Size PersonAP@.5test BicycleAP@.5test CarAP@.5test OverallmAP@.5test Num. of Param.
RGB Baseline 320 39.6% 50.4% 79.4% 56.6% 52.5
IR Baseline 320 49.6% 54.9% 84.4% 63.0% 52.5
Vanila Fusion 320 56.9% 56.7% 82.0% 65.2% 81.8
Fusion + CBAM 320 57.6% 60.5% 83.6% 67.2% 82.7
Fusion + EBAM_C 320 62.6% 65.9% 86.0% 71.5% 82.7%
RGBT 320 63.7% 67.1% 86.4% 72.4% 82.7
CFR_3 640 74.4% 57.7% 84.9% 72.3% 276
RGBT 640 80.1% 76.7% 91.8% 82.9% 82.7%

Repo Structure:

Datasets: A summary of all available dataset with IR-RGB image pair
FLIR_PP: Preprocessing of the FLIR dataset - The cross-labeling algorithm can be used by using pp.py
Fusion: The implementation of the RGBT fusion network
Related_Works: Literature Review
To Train use the train_org.py
To Test use the test_org.py

Citation

@INPROCEEDINGS{9827087,
  author={Vadidar, Sam and Kariminezhad, Ali and Mayr, Christian and Kloeker, Laurent and Eckstein, Lutz},
  booktitle={2022 IEEE Intelligent Vehicles Symposium (IV)}, 
  title={Robust Environment Perception for Automated Driving: A Unified Learning Pipeline for Visual-Infrared Object Detection}, 
  year={2022},
  volume={},
  number={},
  pages={367-374},
  keywords={Visualization;Intelligent vehicles;Fuses;Roads;Pipelines;Object detection;Thermal sensors},
  doi={10.1109/IV51971.2022.9827087}}