github-pengge/inpainting

Image inpainting using GAN and/or partial convolution with progressive growing training scheme.

Image inpainting

Image inpainting using GAN and/or partial convolution with progressive growing training scheme.

Warning: progressive growing scheme might be problematic and use more gpu memory. In my experiments, no improvement was observed.

Dataset

In my experiments, I use CelebA-HQ datasets. To create it, see tkarras/progressive_growing_of_gans.

You can use generate_holes.py to create rectangle holes with multiple resolutions. For irregular holes, generate_irregular_holes.py is a way to create it(probably not a good way).

Proposed config

• progressive_growing = False
• use_gan = True if g_input = masked_X; use_gan = False if g_input = masked_X+mask
• decoder_partial_conv = False
• pix2pix_style = False
• block = basic for lower gpu memory consumption.

Related papers

• Partial conv
• GAN
• pix2pix

Problems

• For large rectangle holes(ratio > 0.5), the results might be unnatural.
• When using partial conv in decoder, the model failed, this is because the masked area is not filled and learned. But the paper uses partial conv in decoder. I have no idea how to get it right.
• Progressive growing training scheme uses more gpu memory than I think. May be it can be optimized. May be even the implementation is problematic.
• When pix2pix_style=True, gradient of discriminator vanished. But why?