Develop a super-resolution method that can actively sample from the space of plausible super-resolutions.
We are part of the CVPR 2022 NTIRE New Trends in Image Restoration and Enhancement workshop.
To participate in this challenge, please sign up using the following link and clone this repo to benchmark your results.
Challenge participants can submit their paper to this CVPR 2022 Workshop.
Usually, super-resolution (SR) is trained using pairs of high- and low-resolution images. Infinitely many high-resolution images can be downsampled to the same low-resolution image. That means that the problem is ill-posed and cannot be inverted with a deterministic mapping. Instead, one can frame the SR problem as learning a stochastic mapping, capable of sampling from the space of plausible high-resolution images given a low-resolution image. This problem has been addressed in recent works [1, 2, 3]. The one-to-many stochastic formulation of the SR problem allows for a few potential advantages:
- The development of more robust learning formulations that better accounts for the ill-posed nature of the SR problem.
- Multiple predictions can be sampled and compared.
- It opens the potential for controllable exploration and editing in the space of SR predictions.
 |
 |
 |
|---|---|---|
| [Paper] [Project] | [Paper] [Project] | [Paper] [Project] |
| [1] SRFlow: Learning the Super-Resolution Space with Normalizing Flow. Lugmayr et al., ECCV 2020. | [2] Explorable Super-Resolution. Bahat & Michaeli, CVPR 2020. | [3] DeepSEE: Deep Disentangled Semantic Explorative Extreme Super-Resolution. Bühler et al., ACCV 2020. |
We organize this challenge to stimulate research in the emerging area of learning one-to-many SR mappings that are capable of sampling from the space of plausible solutions. Therefore the task is to develop a super-resolution method that:
- Each individual SR prediction should achieve highest possible photo-realism, as perceived by humans.
- Is capable of sampling an arbitrary number of SR images capturing meaningful diversity, corresponding to the uncertainty induced by the ill-posed nature of the SR problem together with image priors.
- Each individual SR prediction should be consistent with the input low-resolution image.
The challenge contains two tracks, targeting 4X and 8X super-resolution respectively. You can download the training and validation data in the table below. At a later stage, the low-resolution of the test set will be released.
| Training | Validation | |||
|---|---|---|---|---|
| Low-Resolution | High-Resolution | Low-Resolution | High-Resolution | |
| Track 4X | 4X LR Train | 4X HR Train | 4X LR Valid | 4X HR Valid |
| Track 8X | 8X LR Train | 8X HR Train | 8X LR Valid | 8X HR Valid |
[Paper] Full Report
[Paper] [Code] Noise Conditional Flow Model for Learning the Super-Resolution Space
- Combine SRFlow with SoftFlow
- Models Noise
[Paper] [Code] SRFlow-DA: Super-Resolution Using Normalizing Flow with Deep
- Based on SRFlow
- New LR encoder
- Based on SRFlow
- Improved LR encoding
- Improved Affine couplings
[Paper] [Code] Generating Unobserved Alternatives
- Based on Implicit Maximum Likelihood Estimation
- Uses progressive upscaling
[Paper] Variational AutoEncoder for Reference based Image Super-Resolution
- Reference based
- Components: VGG Encoder, Conditional Variational AutoEncoder, Image Decoder
Flexible SR using Conditional Objective
- Based on RRDB and SFT
- Components: SR-, Conditioning Branch
To guide the research towards useful and generalizable techniques, submissions need to adhere to the following rules. All participants must submit code of their solution along with the final results.
- The method must be able to generate an arbitrary number of diverse samples. That is, your method cannot be limited to a maximum number of different SR samples (corresponding to e.g. a certain number of different output network heads).
- All SR samples must be generated by a single model. That is, no ensembles are allowed.
- No self-ensembles during inference (e.g. flipping and rotation).
- All SR samples must be generated using the same hyper-parameters. That is, the generated SR samples shall not be the result of different choices of hyper-parameters during inference.
- We accept submissions of deterministic methods. However, they will naturally score zero in the diversity measure and therefore not be able to win the challenge.
- Other than the validation and test split of the DIV2k dataset, any training data or pre-training is allowed. You are not allowed to use DIV2K validation or test sets (low- and high-resolution images) for training.
A method is evaluated by first predicting a set of 10 randomly sampled SR images for each low-resolution image in the dataset. From this set of images, evaluation metrics corresponding to the three criteria above will be considered. The participating methods will be ranked according to each metric. These ranks will then be combined into a final score. The three evaluation metrics are described next.
git clone --recursive https://github.com/andreas128/NTIRE22_Learning_SR_Space.git
python3 measure.py OutName path/to/Ground-Truch path/to/Super-Resolution n_samples scale_factor
# n_samples = 10
# scale_factor = 4 for 4X and 8 for 8XTo assess the photo-realism, a human study will be performed on the test set for the final submission.
Automatically assessing the photo-realism and image quality is an extremely difficult task. All existing methods have severe shortcomings. As a very rough guide, you can use the LPIPS distance. Note: LPIPS will not be used to score photo-realism of you final submission. So beware of overfitting to LPIPS, as that can lead to worse results. LPIPS is integrated in our provided toolkit in measure.py.
The samples of the developed method should provide a meaningful diversity. To measure that, we define the following score. We sample 10 images, densely calculate a metric between the samples and the ground truth. To obtain the local best we pixel-wise select the best score out of the 10 samples and take the full image's average. The global best is obtained by averaging the whole image's score and selecting the best. Finally, we calculate the score using the following formula:
score = (global best - local best)/(global best) * 100
| ESRGAN | SRFlow | |
|---|---|---|
| Track 4X | 0 | 25.36 |
| Track 8X | 0 | 10.62 |
To measure how much information is preserved in the super-resloved image from the low-resolution image, we measure the LR-PSNR. The goal in this challenge is to obtain a LR-PSNR of 45dB. All approaches that have an average PSNR above this value will be ranked equally in terms of this criteria.
| ESRGAN | SRFlow | |
|---|---|---|
| Track 4X | 39.01 | 49.91 |
| Track 8X | 31.28 | 50.0 |
| Date | Event |
|---|---|
| 2022.03.23 | Final test data release (inputs only) |
| 2022.03.30 | test result submission deadline |
| 2022.03.30 | fact sheet / code / model submission deadline |
| 2022.04.11 | challenge paper submission deadline |
| 2022.04.19 | camera-ready deadline |
| 2022.06.19 | workshop day |
After the final testing phase, participants will be asked to submit:
- SR predictions on the test set
- A fact sheet describing their method
- Code
Sign up to get notified about further details.
In case of any questions about the challenge or the toolkit, feel free to open an issue on Github.
- Andreas Lugmayr (andreas.lugmayr@vision.ee.ethz.ch)
- Martin Danelljan (martin.danelljan@vision.ee.ethz.ch)
- Radu Timofte (radu.timofte@vision.ee.ethz.ch)
The terms and conditions for participating in the challenge are provided here
To participate in this challenge, please sign up using the following link and clone this repo to benchmark your results.
Challenge participants can submit their paper to this CVPR 2022 Workshop.