Implementation of Efficient Texture Matching with Attention and Gradient Enhancing for Large Scale Image Super Resolution
Our pretrained models are avaiable at Extracter-rec-pretrained and Extracter-All-Losses-pretrained
Recent Reference-Based image super-resolution (RefSR) has improved SOTA deep methods by introducing attention mechanisms to enhance low-resolution(LR) images by transferring high-resolution textures from a high-resolution reference image. The main idea is to search for matches between patches using LR and Reference image pairs in a feature space and merge them using deep architectures. However, existing methods lack the accurate and efficient search of textures. They divide images into as many patches as possible, resulting in inefficient memory usage, and cannot manage inference using large images. Herein, we propose a deep search that can dynamically change the window and stride size in the inference mode resulting in a more efficient memory usage that reduces significantly the number of image patches and finds the k most relevant texture match for each low-resolution patch over the high-resolution reference patches, resulting in an accurate texture match. Our main contribution is that, using larger kernels, we reduce the multiplication cost between LR and Reference features maintaining the SoTA performance and allowing us to generate large-dimension images (1024 px) in a single GPU. We enhance the Super Resolution result by adding gradient density information using a simple residual architecture showing competitive metrics results: PSNR and SSIM.
- python 3.7
- pytorch >= 1.1.0
- torchvision >= 0.4.0
- Clone this github repo
git clone https://github.com/esteban-rs/EXTRACTER.git
cd EXTRACTER
-
Download pre-trained models
-
Run notebooks
-
Results are in "args.save_dir"
- Download CUFED train set and CUFED test set
- Make dataset structure be:
- CUFED
- train
- input
- ref
- test
- CUFED5
- train
- Prepare CUFED dataset
- Run last cell in 01-Extracter-rec-v2.
@misc{reyessaldana2023extracter,
title={EXTRACTER: Efficient Texture Matching with Attention and Gradient Enhancing for Large Scale Image Super Resolution},
author={Esteban Reyes-Saldana and Mariano Rivera},
year={2023},
eprint={2310.01379},
archivePrefix={arXiv},
primaryClass={cs.CV}
}