Miaoyu Li, Ji Liu, Ying Fu, Yulun Zhang, and Dejing Dou, "Spectral Enhanced Rectangle Transformer for Hyperspectral Image Denoising" CVPR 2023
Abstract: Denoising is a crucial step for Hyperspectral image (HSI) applications. Though witnessing the great power of deep learning, existing HSI denoising methods suffer from limitations in capturing the non-local self-similarity. Transformers have shown potential in capturing long-range dependencies, but few attempts have been made with specifically designed Transformer to model the spatial and spectral correlation in HSIs. In this paper, we address these issues by proposing a spectral enhanced rectangle Transformer, driving it to explore the non-local spatial similarity and global spectral low-rank property of HSIs. For the former, we exploit the rectangle self-attention horizontally and vertically to capture the non-local similarity in the spatial domain. For the latter, we design a spectral enhancement module that is capable of extracting global underlying low-rank property of spatial-spectral cubes to suppress noise, while enabling the interactions among non-overlapping spatial rectangles. Extensive experiments have been conducted on both synthetic noisy HSIs and real noisy HSIs, showing the effectiveness of our proposed method in terms of both objective metric and subjective visual quality.
We also provide the checkpoints of our competing methods.
Method | Publication | Params (M) | GFLOPs (512×512×31) | Model Zoo (Gaussian noise) | Model Zoo (Complex noise) |
---|---|---|---|---|---|
GRNet | TGRS 2022 | 41.44 | 610.7 | Google Drive | Google Drive |
QRNN3D | TNNLS 2020 | 0.86 | 2513.7 | Google Drive | Google Drive |
T3SC | NeurIPS 2021 | 0.83 | - | Google Drive | Google Drive |
MACNet | TGRS 2022 | 0.43 | - | Google Drive | Google Drive |
SST | AAAI 2023 | 4.10 | 2082.4` | Google Drive | Google Drive |
SERT | CVPR 2023 | 1.91 | 1018.9 | Google Drive | Google Drive |
Method | Params (M) | GFLOPs(512×512×34) | PSNR | Model Zoo |
---|---|---|---|---|
GRNet | 44.40 | 611.9 | Google Drive | |
QRNN3D | 0.86 | 2756.9 | Google Drive | |
T3SC | 0.83 | - | Google Drive | |
MACNet | 0.43 | - | Google Drive | |
SST | 2.14 | Google Drive | ||
SERT | 1.91 | 1021.9 | Google Drive |
Method | Params (M) | GFLOPs | Model Zoo |
---|---|---|---|
GRNet | 44.53 | Google Drive | |
QRNN3D | 0.86 | Google Drive | |
T3SC | 0.83 | Google Drive | |
MACNet | 0.43 | Google Drive | |
SST | 22.76 | GoogleDriver | |
SERT | 8 | Google Drive |
- The entire ICVL dataset download link: https://icvl.cs.bgu.ac.il/hyperspectral/
- split the entire dataset into training samples, testing samples and validating samples. The files used in training are listed in utility/icvl_partition/icvl_train_list.txt.
- generate lmdb dataset for training
python utility/lmdb_data.py
- download the testing data from BaiduDisk or generate them by yourself through
python utility/mat_data.py
- Please refer to [github-link] for "Hyperspectral Image Denoising with Realistic Data in ICCV, 2021" to download the dataset
Testing HSIs are listed in utility/realistic_partition/test.txt. Other HSIs in the dataset are used for training.
- The training dataset are from link: https://apex-esa.org/. The origin Urban dataset are from link: https://rslab.ut.ac.ir/data.
-
Run the create_big_apex_dataset() funtion in utility/mat_data.py to generate training samples.
-
Run the createDCmall() function in utility/lmdb_data.py to generate training lmdb dataset.
#for gaussian noise
#----training----
python hside_simu.py -a sert_base -p sert_base_gaussian
#----testing---- The results are shown in Table 1 in the main paper.
python hside_simu_test.py -a sert_base -p sert_base_gaussian_test -r -rp checkpoints/icvl_gaussian.pth --test-dir /icvl_noise/512_50
#for comlpex noise
#----training----
python hside_simu_complex.py -a sert_base -p sert_base_complex
#----testing---- The results are shown in Table 2 in the main paper.
python hside_simu_test.py -a sert_base -p sert_base_complex_test -r -rp checkpoints/icvl_complex.pth --test-dir /icvl_noise/512_mix
#----training----
python hside_urban.py -a sert_urban -p sert_urban
#----testing---- The results are shown in Figure 4 in the main paper.
python hside_urban_test.py -a sert_urban -p sert_urban_test -r -rp ./checkpoints/real_urban.pth
#----training----
python hside_real.py -a sert_real -p sert_real
#----testing---- The results are shown in Table 3 in the main paper.
python hside_real_test.py -a sert_real -p sert_real_test -r -rp ./checkpoints/real_realistic.pth
If you find the code helpful in your resarch or work, please cite the following paper(s).
@inproceedings{li2023spatial,
title={Spatial-Spectral Transformer for Hyperspectral Image Denoising},
author={Li, Miaoyu and Fu, Ying and Zhang, Yulun},
booktitle={AAAI},
year={2023}
}
@inproceedings{li2023spectral,
title={Spectral Enhanced Rectangle Transformer for Hyperspectral Image Denoising},
author={Miaoyu Li and Ji Liu and Ying Fu and Yulun Zhang and Dejing Dou},
booktitle={CVPR},
year={2023}
}
The codes are based on QRNN3D.