Ruikang Li, Yujin Wang, Shiqi Chen, Fan Zhang, Jinwei Gu and Tianfan Xue
- Sept 29, 2024: Paper accepted at ECCV 2024. 😊
- Nov 4, 2024: Training and inferencing code released. 🌹
- Nov 10, 2024: Pre-trained models and visual results released. Feel free to download! 📝
- Instruction, Results
- Demo on Huggingface for processing real-captured images.
- We have captured a new test set with real_captured RAW images, and we will release a new benchmark for dual-denoising!
- Evaluate on SIDD benchmark.
Abstract: There are two typical ways to inject a denoiser into the Image Signal Processing (ISP) pipeline: applying a denoiser directly to captured raw frames (raw domain) or to the ISP's output sRGB images (sRGB domain). However, both approaches have their limitations. Residual noise from raw-domain denoising can be amplified by the subsequent ISP processing, and the sRGB domain struggles to handle spatially varying noise since it only sees noise distorted by the ISP. Consequently, most raw or sRGB domain denoising works only for specific noise distributions and ISP configurations. Unlike previous single-domain denoising, DualDn consists of two denoising networks: one in the raw domain and one in the sRGB domain. The raw domain denoising adapts to sensor-specific noise as well as spatially varying noise levels, while the sRGB domain denoising adapts to ISP variations and removes residual noise amplified by the ISP. Both denoising networks are connected with a differentiable ISP, which is trained end-to-end and discarded during the inference stage.
See INSTALL.md for the installation of environment and dependencies required to run DualDn.
We trained DualDn on a single GPU ONLY using: (1) clean raw images, (2) a reasonable noise model.
We chose the MIT-Adobe FiveK Dataset for training, as it's a robust dataset containing multiple raw images in DNG format with EXIF metadata. Although some images contain noise, MIT-Adobe FiveK is sufficient for training DualDn to generalize effectively to in-the-wild scenes.
And we believe that if more clean raws or a more accurate noise model are given, DualDn’s performance could improve even further.
| Training on | Evaluating on | Test Sets | Pre-trained Model | Instructions(TODO) | Visual Results |
|---|---|---|---|---|---|
| MIT-Adobe FiveK Download |
|||||
| Synthetic Images | Download | Download | Prerequisites \ Train \ Test \ Inference | Download | |
| Real_captured Images | Download | Download | Prerequisites \ Train \ Test \ Inference | Download | |
| DND Benchmark | Download | Prerequisites \ Train \ Test \ Inference | Download |
HINTS: We only provide models and results trained using Restormer backbone, as it demonstrated the best performance in DualDn. However, you can also train DualDn with other backbones, such as SwinIR or MIRNet-v2, by following our instructions. Reference evaluation metrics are also available in our paper.
@article{li2024dualdn,
title={DualDn: Dual-domain Denoising via Differentiable ISP},
author={Li, Ruikang and Wang, Yujin and Chen, Shiqi and Zhang, Fan and Gu, Jinwei and Xue, Tianfan},
journal={arXiv preprint arXiv:2409.18783},
year={2024}
}
This code is based on the BasicSR toolbox.