/HVI-CIDNet

"You Only Need One Color Space: An Efficient Network for Low-light Image Enhancement"

Primary LanguagePythonMIT LicenseMIT

 

You Only Need One Color Space: An Efficient Network for Low-light Image Enhancement

Yixu Fengβˆ— , Cheng Zhangβˆ— , Pei Wang , Peng Wu , Qingsen Yan , Yanning Zhang

arXiv

PWC PWC

PWC PWC

PWC PWC

PWC PWC

PWC PWC

PWC PWC

PWC PWC

 

πŸ’‘News

  • 2024.04.18 Full version Code, models, visual comparison have been released. We promise that all the weights only trained on train sets, and each weights is reproducible. Hope it will help your future work. If you have trained a better result, please contact us. We look forward to subsequent work based on the HVI color "space"! πŸ’Ž

  • 2024.04.14 Update test fine tuning result and weights on LOLv1 dataset. 🧾

  • 2024.03.04 Update five unpaired datasets (DICM, LIME, MEF, NPE, VV) visual results. ✨

  • 2024.03.03 Update pre-trained weights and output results on our HVI-CIDNet using Baidu Pan. 🧾

  • 2024.02.08 Update HVI-CIDNet in Arxiv. The new code, models and results will be uploaded. 🎈

βš™ Proposed HVI-CIDNet

HVI-CIDNet pipeline:

results3

CID dual-branch UNet structure:

results4

πŸ–Ό Visual Comparison

LOL-v1, LOL-v2-real, and LOL-v2-synthetic:

results1

DICM, LIME, MEF, NPE, and VV:

results2

LOL-Blur:

results2

🧾Weights and Results

All the weights that we trained on different datasets is available at [Baidu Pan] (code: yixu). Results on DICM, LIME, MEF, NPE, and VV datasets can be downloaded from [Baidu Pan] (code: yixu). Bolded fonts represent impressive metrics.

  • The metrics of HVI-CIDNet on paired datasets are shown in the following table:
Folder (test datasets) PSNR SSIM LPIPS GT Mean Results Weights Path
(LOLv1)
v1 w perc loss/ wo gt mean
23.8091 0.8574 0.0856 Baidu Pan LOLv1/w_perc.pth
(LOLv1)
v1 w perc loss/ w gt mean
27.7146 0.8760 0.0791 √ ditto LOLv1/w_perc.pth
(LOLv1)
v1 wo perc loss/ wo gt mean
23.5000 0.8703 0.1053 Baidu Pan LOLv1/wo_perc.pth
(LOLv1)
v1 wo perc loss/ w gt mean
28.1405 0.8887 0.0988 √ ditto LOLv1/wo_perc.pth
(LOLv2_real)
v2 wo perc loss/ wo gt mean
23.4269 0.8622 0.1691 Baidu Pan (lost)
(LOLv2_real)
v2 wo perc loss/ w gt mean
27.7619 0.8812 0.1649 √ ditto (lost)
(LOLv2_real)
v2 best gt mean
28.1387 0.8920 0.1008 √ Baidu Pan LOLv2_real/w_prec.pth
(LOLv2_real)
v2 best Normal
24.1106 0.8675 0.1162 Baidu Pan (lost)
(LOLv2_real)
v2 best PSNR
23.9040 0.8656 0.1219 Baidu Pan LOLv2_real/best_PSNR.pth
(LOLv2_real)
v2 best SSIM
23.8975 0.8705 0.1185 Baidu Pan LOLv2_real/best_SSIM.pth
(LOLv2_real)
v2 best SSIM/ w gt mean
28.3926 0.8873 0.1136 √ None LOLv2_real/best_SSIM.pth
(LOLv2_syn)
syn wo perc loss/ wo gt mean
25.7048 0.9419 0.0471 Baidu Pan LOLv2_syn/wo_perc.pth
(LOLv2_syn)
syn wo perc loss/ w gt mean
29.5663 0.9497 0.0437 √ ditto LOLv2_syn/wo_perc.pth
(LOLv2_syn)
syn w perc loss/ wo gt mean
25.1294 0.9388 0.0450 Baidu Pan LOLv2_syn/w_perc.pth
(LOLv2_syn)
syn w perc loss/ w gt mean
29.3666 0.9500 0.0403 √ ditto LOLv2_syn/w_perc.pth
Sony_Total_Dark 22.9039 0.6763 0.4109 Baidu Pan SID.pth
LOL-Blur 26.5719 0.8903 0.1203 Baidu Pan LOL-Blur.pth
SICE-Mix 13.4235 0.6360 0.3624 √ Baidu Pan SICE.pth
SICE-Grad 13.4453 0.6477 0.3181 √ Baidu Pan SICE.pth
  • Performance on five unpaired datasets are shown in the following table:
metrics DICM LIME MEF NPE VV
NIQE 3.79 4.13 3.56 3.74 3.21
BRISQUE 21.47 16.25 13.77 18.92 30.63
  • While we don't recommend that you perform finetuning on the test set, in order to demonstrate the effectiveness of our model, we also provide here the results of test finetuning training on the LOLv1 dataset. Using the fine tuning technique on the test set does make the PSNR metrics higher, but other metrics are not found to be significantly changed on CIDNet, which may result in a lower generalization of the model, so we do not recommend you do this.
Folder (test datasets) PSNR SSIM LPIPS GT Mean Results Weights Path
(LOLv1)
v1 test finetuning
25.4036 0.8652 0.0897 Baidu Pan LOLv1/test_finetuning.pth
(LOLv1)
v1 test finetuning
27.5969 0.8696 0.0869 √ ditto ditto

1. Get Started πŸŒ‘

Dependencies and Installation

  • Python 3.7.0
  • Pytorch 1.13.1

(1) Create Conda Environment

conda create --name CIDNet python=3.7.0
conda activate CIDNet

(2) Clone Repo

git clone git@github.com:Fediory/HVI-CIDNet.git

(3) Install Dependencies

cd HVI-CIDNet
pip install -r requirements.txt

Data Preparation

You can refer to the following links to download the datasets. Note that we only use low_blur and high_sharp_scaled subsets of LOL-Blur dataset.

Then, put them in the following folder:

datasets (click to expand)
β”œβ”€β”€ datasets
    β”œβ”€β”€ DICM
    β”œβ”€β”€ LIME
    β”œβ”€β”€ LOLv1
        β”œβ”€β”€ our485
            β”œβ”€β”€low
            β”œβ”€β”€high
	    β”œβ”€β”€ eval15
            β”œβ”€β”€low
            β”œβ”€β”€high
    β”œβ”€β”€ LOLv2
        β”œβ”€β”€ Real_captured
            β”œβ”€β”€ Train
                β”œβ”€β”€ Low
                β”œβ”€β”€ Normal
	        β”œβ”€β”€ Test
                β”œβ”€β”€ Low
                β”œβ”€β”€ Normal
        β”œβ”€β”€ Synthetic
            β”œβ”€β”€ Train
                β”œβ”€β”€ Low
                β”œβ”€β”€ Normal
	        β”œβ”€β”€ Test
                β”œβ”€β”€ Low
                β”œβ”€β”€ Normal
    β”œβ”€β”€ LOL_blur
        β”œβ”€β”€ eval
            β”œβ”€β”€ high_sharp_scaled
            β”œβ”€β”€ low_blur
        β”œβ”€β”€ test
            β”œβ”€β”€ high_sharp_scaled
                β”œβ”€β”€ 0012
                β”œβ”€β”€ 0017
                ...
            β”œβ”€β”€ low_blur
                β”œβ”€β”€ 0012
                β”œβ”€β”€ 0017
                ...
        β”œβ”€β”€ train
            β”œβ”€β”€ high_sharp_scaled
                β”œβ”€β”€ 0000
                β”œβ”€β”€ 0001
                ...
            β”œβ”€β”€ low_blur
                β”œβ”€β”€ 0000
                β”œβ”€β”€ 0001
                ...
    β”œβ”€β”€ MEF
    β”œβ”€β”€ NPE
    β”œβ”€β”€ SICE
        β”œβ”€β”€ Dataset
            β”œβ”€β”€ eval
                β”œβ”€β”€ target
                β”œβ”€β”€ test
            β”œβ”€β”€ label
            β”œβ”€β”€ train
                β”œβ”€β”€ 1
                β”œβ”€β”€ 2
                ...
        β”œβ”€β”€ SICE_Grad
        β”œβ”€β”€ SICE_Mix
        β”œβ”€β”€ SICE_Reshape
    β”œβ”€β”€ Sony_total_dark
        β”œβ”€β”€ eval
            β”œβ”€β”€ long
            β”œβ”€β”€ short
        β”œβ”€β”€ test
            β”œβ”€β”€ long
                β”œβ”€β”€ 10003
                β”œβ”€β”€ 10006
                ...
            β”œβ”€β”€ short
                β”œβ”€β”€ 10003
                β”œβ”€β”€ 10006
                ...
        β”œβ”€β”€ train
            β”œβ”€β”€ long
                β”œβ”€β”€ 00001
                β”œβ”€β”€ 00002
                ...
            β”œβ”€β”€ short
                β”œβ”€β”€ 00001
                β”œβ”€β”€ 00002
                ...
    β”œβ”€β”€ VV

2. Testing πŸŒ’

Download our weights from [Baidu Pan] (code: yixu) and put them in folder weights:

β”œβ”€β”€ weights
    β”œβ”€β”€ LOLv1
        β”œβ”€β”€ w_perc.pth
        β”œβ”€β”€ wo_perc.pth
        β”œβ”€β”€ test_finetuning.pth
    β”œβ”€β”€ LOLv2_real
        β”œβ”€β”€ best_PSNR.pth
        β”œβ”€β”€ best_SSIM.pth
        β”œβ”€β”€ w_perc.pth
    β”œβ”€β”€ LOLv2_syn
        β”œβ”€β”€ w_perc.pth
        β”œβ”€β”€ wo_perc.pth
    β”œβ”€β”€ LOL-Blur.pth
    β”œβ”€β”€ SICE.pth
    β”œβ”€β”€ SID.pth
  • You can test our method as followed, all the results will saved in ./output folder:
# LOLv1
python eval.py --lol --perc # weights that trained with perceptual loss
python eval.py --lol # weights that trained without perceptual loss

# LOLv2-real
python eval.py --lol_v2_real --best_GT_mean # you can choose best_GT_mean or best_PSNR or best_SSIM

# LOLv2-syn
python eval.py --lol_v2_syn --perc # weights that trained with perceptual loss
python eval.py --lol_v2_syn # weights that trained without perceptual loss

# SICE
python eval.py --SICE_grad # output SICE_grad
python eval.py --SICE_mix # output SICE_mix

# Sony-Total-Dark
python eval_SID_blur --SID

# LOL-Blur
python eval_SID_blur --Blur

# five unpaired datasets DICM, LIME, MEF, NPE, VV. 
# We note that: you can choose one weights in ./weights folder, and set the alpha float number (defualt=1.0) as illumination scale of the datasets.
# You can change "--DICM" to the other unpaired datasets "LIME, MEF, NPE, VV".
python eval.py --unpaired --DICM --unpaired_weights --alpha
# e.g.
python eval.py --unpaired --DICM --unpaired_weights ./weights/LOLv2_syn/w_perc.pth --alpha 0.9
  • Also, you can test all the metrics mentioned in our paper as follows:
# LOLv1
python measure.py --lol

# LOLv2-real
python measure.py --lol_v2_real

# LOLv2-syn
python measure.py --lol_v2_syn

# Sony-Total-Dark
python measure_SID_blur.py --SID

# LOL-Blur
python measure_SID_blur.py --Blur

# SICE-Grad
python measure.py --SICE_grad

# SICE-Mix
python measure.py --SICE_mix


# five unpaired datasets DICM, LIME, MEF, NPE, VV. 
# You can change "--DICM" to the other unpaired datasets "LIME, MEF, NPE, VV".
python measure_niqe_bris.py --DICM


# Note: Following LLFlow, KinD, and Retinxformer, we have also adjusted the brightness of the output image produced by the network, based on the average value of GroundTruth (GT). This only works in paired datasets. If you want to measure it, please add "--use_GT_mean".
# 
# e.g.
python measure.py --lol --use_GT_mean
  
  • Evaluating the Parameters, FLOPs, and running time of HVI-CIDNet:
python net_test.py

3. Training πŸŒ“

  • We put all the configurations that need to be adjusted in the ./data/options.py folder and explained them in the file. We apologize that some of the training parameters we are no longer able to provide and share with you, but we guarantee that all the weights are trainable by parameter tuning. You can train our HVI-CIDNet by:
python train.py
  • All weights are saved to the ./weights/train folder and are saved in steps of the checkpoint set in the options.py as epoch_*.pth where * represent the epoch number.
  • Also, for every weight saved, metrics are measured for the validation set and printed to the command line. Finally, the results of all weights' test metrics on the validation set and options in ./data/options.py will be saved to ./results/training/metrics.md.
  • In each epoch, we save an output (test) and GT image to the ./results/training folder to facilitate the visualization of the training results and progress of each epoch, as well as to detect the generation of gradient explosion in advance.
  • After each checkpoint, we save all the validation set outputs for this time in the ./results folder to the corresponding folder. Note that we use a replacement strategy for different checkpoints for each dataset. That is, we do not save the plots of all checkpoints, but only the weights of each checkpoint.

4. Contacts πŸŒ”

If you have any questions, please contact us or submit an issue to the repository!

Yixu Feng (yixu-nwpu@mail.nwpu.edu.cn) Cheng Zhang (zhangcheng233@mail.nwpu.edu.cn)

5. Citation

If you find our work useful for your research, please cite our paper

@misc{feng2024need,
      title={You Only Need One Color Space: An Efficient Network for Low-light Image Enhancement}, 
      author={Yixu Feng and Cheng Zhang and Pei Wang and Peng Wu and Qingsen Yan and Yanning Zhang},
      year={2024},
      eprint={2402.05809},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}