/HVI-CIDNet

"You Only Need One Color Space: An Efficient Network for Low-light Image Enhancement"

Primary LanguagePythonMIT LicenseMIT

 

You Only Need One Color Space: An Efficient Network for Low-light Image Enhancement

Qingsen Yan∗ , Yixu Feng∗ , Cheng Zhang∗ , Pei Wang, Peng Wu, Wei Dong, Jinqiu Sun, Yanning Zhang

arXiv

PWC PWC

PWC PWC

PWC PWC

PWC PWC

PWC PWC

PWC PWC

PWC PWC

 

News 💡

  • 2024.06.19 Update newest version of paper in Arxiv.

  • 2024.05.12 Update peer results on LOLv1. 🤝

  • 2024.04.28 Synchronize all Baidu Pan datasets, outputs, and weights to One Drive. 🌎

  • 2024.04.18 Full version Code, models, visual comparison have been released. We promise that all the weights only trained on train sets, and each weights is reproducible. Hope it will help your future work. If you have trained a better result, please contact us. We look forward to subsequent work based on the HVI color "space"! 💎

  • 2024.04.14 Update test fine tuning result and weights on LOLv1 dataset. 🧾

  • 2024.03.04 Update five unpaired datasets (DICM, LIME, MEF, NPE, VV) visual results. ✨

  • 2024.03.03 Update pre-trained weights and output results on our HVI-CIDNet using Baidu Pan. 🧾

  • 2024.02.08 Update HVI-CIDNet in Arxiv. The new code, models and results will be uploaded. 🎈

⚙ Proposed HVI-CIDNet

HVI-CIDNet pipeline:

results3

Lighten Cross-Attention (LCA) Block structure:

results4

🖼 Visual Comparison

LOL-v1, LOL-v2-real, and LOL-v2-synthetic:

results1

DICM, LIME, MEF, NPE, and VV:

results2

LOL-Blur:

results2

Weights and Results 🧾

All the weights that we trained on different datasets is available at [Baidu Pan] (code: yixu) and [One Drive] (code: yixu). Results on DICM, LIME, MEF, NPE, and VV datasets can be downloaded from [Baidu Pan] (code: yixu) and [One Drive] (code: yixu). Bolded fonts represent impressive metrics.

  • The metrics of HVI-CIDNet on paired datasets are shown in the following table, all the link code is yixu:
Folder (test datasets) PSNR SSIM LPIPS GT Mean Results Weights Path
(LOLv1)
v1 w perc loss/ wo gt mean		 23.8091 0.8574 0.0856 Baidu Pan and One Drive LOLv1/w_perc.pth
(LOLv1)
v1 w perc loss/ w gt mean		 27.7146 0.8760 0.0791 √ ditto LOLv1/w_perc.pth
(LOLv1)
v1 wo perc loss/ wo gt mean		 23.5000 0.8703 0.1053 Baidu Pan and One Drive LOLv1/wo_perc.pth
(LOLv1)
v1 wo perc loss/ w gt mean		 28.1405 0.8887 0.0988 √ ditto LOLv1/wo_perc.pth
(LOLv2_real)
v2 wo perc loss/ wo gt mean		 23.4269 0.8622 0.1691 Baidu Pan and One Drive (lost)
(LOLv2_real)
v2 wo perc loss/ w gt mean		 27.7619 0.8812 0.1649 √ ditto (lost)
(LOLv2_real)
v2 best gt mean		 28.1387 0.8920 0.1008 √ Baidu Pan and One Drive LOLv2_real/w_prec.pth
(LOLv2_real)
v2 best Normal		 24.1106 0.8675 0.1162 Baidu Pan and One Drive (lost)
(LOLv2_real)
v2 best PSNR		 23.9040 0.8656 0.1219 Baidu Pan and One Drive LOLv2_real/best_PSNR.pth
(LOLv2_real)
v2 best SSIM		 23.8975 0.8705 0.1185 Baidu Pan and One Drive LOLv2_real/best_SSIM.pth
(LOLv2_real)
v2 best SSIM/ w gt mean		 28.3926 0.8873 0.1136 √ None LOLv2_real/best_SSIM.pth
(LOLv2_syn)
syn wo perc loss/ wo gt mean		 25.7048 0.9419 0.0471 Baidu Pan and One Drive LOLv2_syn/wo_perc.pth
(LOLv2_syn)
syn wo perc loss/ w gt mean		 29.5663 0.9497 0.0437 √ ditto LOLv2_syn/wo_perc.pth
(LOLv2_syn)
syn w perc loss/ wo gt mean		 25.1294 0.9388 0.0450 Baidu Pan and One Drive LOLv2_syn/w_perc.pth
(LOLv2_syn)
syn w perc loss/ w gt mean		 29.3666 0.9500 0.0403 √ ditto LOLv2_syn/w_perc.pth
Sony_Total_Dark 22.9039 0.6763 0.4109 Baidu Pan and One Drive SID.pth
LOL-Blur 26.5719 0.8903 0.1203 Baidu Pan and One Drive LOL-Blur.pth
SICE-Mix 13.4235 0.6360 0.3624 √ Baidu Pan and One Drive SICE.pth
SICE-Grad 13.4453 0.6477 0.3181 √ Baidu Pan and One Drive SICE.pth
  • Performance on five unpaired datasets are shown in the following table:
metrics DICM LIME MEF NPE VV
NIQE 3.79 4.13 3.56 3.74 3.21
BRISQUE 21.47 16.25 13.77 18.92 30.63
  • While we don't recommend that you perform finetuning on the test set, in order to demonstrate the effectiveness of our model, we also provide here the results of test finetuning training on the LOLv1 dataset. Using the fine tuning technique on the test set does make the PSNR metrics higher, but other metrics are not found to be significantly changed on CIDNet, which may result in a lower generalization of the model, so we do not recommend you do this.
Folder (test datasets) PSNR SSIM LPIPS GT Mean Results Weights Path
(LOLv1)
v1 test finetuning		 25.4036 0.8652 0.0897 Baidu Pan and One Drive LOLv1/test_finetuning.pth
(LOLv1)
v1 test finetuning		 27.5969 0.8696 0.0869 √ ditto ditto
  • Contributions from other peers: this section is where other peers have trained better versions of weights using our model, and we will show both their weights and results here. If you have trained better weights, please contact us by email or submit issue.
Datasets PSNR SSIM LPIPS GT Mean Results Weights Path Contributor Detail GPU
LOLv1 24.7401 0.8604 0.0896 Baidu Pan and One Drive LOLv1/other/PSNR_24.74.pth [Xi’an Polytechnic University]
Yingjian Li		 NVIDIA 4070

1. Get Started 🌑

Dependencies and Installation

  • Python 3.7.0
  • Pytorch 1.13.1

(1) Create Conda Environment

conda create --name CIDNet python=3.7.0
conda activate CIDNet

(2) Clone Repo

git clone git@github.com:Fediory/HVI-CIDNet.git

(3) Install Dependencies

cd HVI-CIDNet
pip install -r requirements.txt

Data Preparation

You can refer to the following links to download the datasets. Note that we only use low_blur and high_sharp_scaled subsets of LOL-Blur dataset.

Then, put them in the following folder:

datasets (click to expand) 
├── datasets
	├── DICM
	├── LIME
	├── LOLdataset
		├── our485
			├──low
			├──high
		├── eval15
			├──low
			├──high
	├── LOLv2
		├── Real_captured
			├── Train
				├── Low
				├── Normal
			├── Test
				├── Low
				├── Normal
		├── Synthetic
			├── Train
				├── Low
				├── Normal
			├── Test
				├── Low
				├── Normal
	├── LOL_blur
		├── eval
			├── high_sharp_scaled
			├── low_blur
		├── test
			├── high_sharp_scaled
				├── 0012
				├── 0017
				...
			├── low_blur
				├── 0012
				├── 0017
				...
		├── train
			├── high_sharp_scaled
				├── 0000
				├── 0001
				...
			├── low_blur
				├── 0000
				├── 0001
				...
	├── MEF
	├── NPE
	├── SICE
		├── Dataset
			├── eval
				├── target
				├── test
			├── label
			├── train
				├── 1
				├── 2
				...
		├── SICE_Grad
		├── SICE_Mix
		├── SICE_Reshape
	├── Sony_total_dark
		├── eval
			├── long
			├── short
		├── test
			├── long
				├── 10003
				├── 10006
				...
			├── short
				├── 10003
				├── 10006
				...
		├── train
			├── long
				├── 00001
				├── 00002
				...
			├── short
				├── 00001
				├── 00002
				...
	├── VV

2. Testing 🌒

Download our weights from [Baidu Pan] (code: yixu) and put them in folder weights:

├── weights
    ├── LOLv1
        ├── w_perc.pth
        ├── wo_perc.pth
        ├── test_finetuning.pth
    ├── LOLv2_real
        ├── best_PSNR.pth
        ├── best_SSIM.pth
        ├── w_perc.pth
    ├── LOLv2_syn
        ├── w_perc.pth
        ├── wo_perc.pth
    ├── LOL-Blur.pth
    ├── SICE.pth
    ├── SID.pth
  • You can test our method as followed, all the results will saved in ./output folder:
# LOLv1
python eval.py --lol --perc # weights that trained with perceptual loss
python eval.py --lol # weights that trained without perceptual loss

# LOLv2-real
python eval.py --lol_v2_real --best_GT_mean # you can choose best_GT_mean or best_PSNR or best_SSIM

# LOLv2-syn
python eval.py --lol_v2_syn --perc # weights that trained with perceptual loss
python eval.py --lol_v2_syn # weights that trained without perceptual loss

# SICE
python eval.py --SICE_grad # output SICE_grad
python eval.py --SICE_mix # output SICE_mix

# Sony-Total-Dark
python eval_SID_blur --SID

# LOL-Blur
python eval_SID_blur --Blur

# five unpaired datasets DICM, LIME, MEF, NPE, VV. 
# We note that: you can choose one weights in ./weights folder, and set the alpha float number (defualt=1.0) as illumination scale of the datasets.
# You can change "--DICM" to the other unpaired datasets "LIME, MEF, NPE, VV".
python eval.py --unpaired --DICM --unpaired_weights --alpha
# e.g.
python eval.py --unpaired --DICM --unpaired_weights ./weights/LOLv2_syn/w_perc.pth --alpha 0.9

# Custome Datasets
python eval.py --unpaired --custome --custome_path ./your/costome/dataset/path --unpaired_weights ./weights/LOLv2_syn/w_perc.pth --alpha 0.9
  • Also, you can test all the metrics mentioned in our paper as follows:
# LOLv1
python measure.py --lol

# LOLv2-real
python measure.py --lol_v2_real

# LOLv2-syn
python measure.py --lol_v2_syn

# Sony-Total-Dark
python measure_SID_blur.py --SID

# LOL-Blur
python measure_SID_blur.py --Blur

# SICE-Grad
python measure.py --SICE_grad

# SICE-Mix
python measure.py --SICE_mix


# five unpaired datasets DICM, LIME, MEF, NPE, VV. 
# You can change "--DICM" to the other unpaired datasets "LIME, MEF, NPE, VV".
python measure_niqe_bris.py --DICM


# Note: Following LLFlow, KinD, and Retinxformer, we have also adjusted the brightness of the output image produced by the network, based on the average value of GroundTruth (GT). This only works in paired datasets. If you want to measure it, please add "--use_GT_mean".
# 
# e.g.
python measure.py --lol --use_GT_mean
  • Evaluating the Parameters, FLOPs, and running time of HVI-CIDNet:
python net_test.py

3. Training 🌓

  • We put all the configurations that need to be adjusted in the ./data/options.py folder and explained them in the file. We apologize that some of the training parameters we are no longer able to provide and share with you, but we guarantee that all the weights are trainable by parameter tuning. You can train our HVI-CIDNet by:
python train.py
  • All weights are saved to the ./weights/train folder and are saved in steps of the checkpoint set in the options.py as epoch_*.pth where * represent the epoch number.
  • Also, for every weight saved, metrics are measured for the validation set and printed to the command line. Finally, the results of all weights' test metrics on the validation set and options in ./data/options.py will be saved to ./results/training/metrics.md.
  • In each epoch, we save an output (test) and GT image to the ./results/training folder to facilitate the visualization of the training results and progress of each epoch, as well as to detect the generation of gradient explosion in advance.
  • After each checkpoint, we save all the validation set outputs for this time in the ./results folder to the corresponding folder. Note that we use a replacement strategy for different checkpoints for each dataset. That is, we do not save the plots of all checkpoints, but only the weights of each checkpoint.

4. Contacts 🌔

If you have any questions, please contact us or submit an issue to the repository!

Yixu Feng (yixu-nwpu@mail.nwpu.edu.cn) Cheng Zhang (zhangcheng233@mail.nwpu.edu.cn)

5. Citation 🌕

If you find our work useful for your research, please cite our paper

@misc{feng2024hvi,
      title={You Only Need One Color Space: An Efficient Network for Low-light Image Enhancement}, 
      author={Yixu Feng and Cheng Zhang and Pei Wang and Peng Wu and Qingsen Yan and Yanning Zhang},
      year={2024},
      eprint={2402.05809},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}