SpA GAN for Cloud Removal

new

2020.9.29 The draft is released now at https://arxiv.org/abs/2009.13015.

1. INTRODUCTION

This is the source code of Cloud Removal for Remote Sensing Imagery via Spatial Attention Generative Adversarial Network. In this work, I proposes a novel cloud removal model called spatial attention generative adversarial networks or SpA GAN, which use spatial attention networks (SPANet) as generator. The architecture of SpA GAN is shown as fellow:

Generator

SpA GAN uses spatial attention networks an generator. See ./models/gen/SPANet.py for more details.

Discriminator

Discriminator is a fully CNN that C is convolution layer, B is batch normalization and R is Leaky ReLU. See ./models/dis/dis.py for more details.

Loss

The total loss of SpA GAN is formulated as fellow:

the first part is the loss of GAN

the second part is standard $L_1$ loss where $\lambda_c$ is a hyper parameter to control the weight of each channel to the loss.

the third part is attention loss where $A$ is the attention map and $M$ is the mask of cloud that computed from $M=|I_{in}-I_{gt}|_1$.

2. DATASET

2.1. RICE_DATASET

Click official address or Google Drive to download the open source RICE dataset. Build the file structure as the folder data shown. Here cloudy_image is the folder where the cloudy image is stored and the folder ground_truth stores the corresponding cloudless images.

./
+-- data
    +--	RICE_DATASET
        +-- RICE1
        |   +-- cloudy_image
        |   |   +-- 0.png
        |   |   +-- ...
        |   +-- ground_truth
        |       +-- 0.png
        |       +-- ...
        +-- RICE2
            +-- cloudy_image
            |   +-- 0.png
            |   +-- ...
            +-- ground_truth
                +-- 0.png
                +-- ...

2.2. Perlin Dataset

Construct the dataset by adding Perlin noise as cloud into the image.

3. TRAIN

Modify the config.yml to set your parameters and run:

python train.py

4. TEST

python predict.py --config <path_to_config.yml_in_the_out_dir> --test_dir <path_to_a_directory_stored_test_data> --out_dir <path_to_an_output_directory> --pretrained <path_to_a_pretrained_model> --cuda

There're my pre-trained models on RICE1(./pretrained_models/RICE1/gen_model_epoch_200.pth) and RICE2(./pretrained_models/RICE2/gen_model_epoch_200.pth).

Some results are shown as bellow and the images from left to right are: cloudy image, attention map, SpA GAN's output, ground truth.

5. EXPERIMENTS

In this section, I compares SpA GAN with conditional GAN and cycle GAN using peak signal to noise ratio (PSNR) and structural similarity index (SSIM) as metrics on datasets RICE1 and RICE2.

5.1 RICE1

qualitative analysis

The result are shown as bellow and the images from left to right are: cloudy image, conditional GAN's output, cycle GAN's output , SpA GAN's output, ground truth.

quantitative analysis

	PSNR	SSIM
cGAN	26.547	0.903
cycle GAN	25.880	0.893
SpA GAN	30.232	0.954

5.1 RICE2