/inpainting

Image inpainting using GAN and/or partial convolution with progressive growing training scheme.

Primary LanguagePython

Image inpainting

Image inpainting using GAN and/or partial convolution with progressive growing training scheme.

Warning: progressive growing scheme might be problematic and use more gpu memory. In my experiments, no improvement was observed.

Dataset

In my experiments, I use CelebA-HQ datasets. To create it, see tkarras/progressive_growing_of_gans.

You can use generate_holes.py to create rectangle holes with multiple resolutions. For irregular holes, generate_irregular_holes.py is a way to create it(probably not a good way).

Proposed config

  • progressive_growing = False
  • use_gan = True if g_input = masked_X; use_gan = False if g_input = masked_X+mask
  • decoder_partial_conv = False
  • pix2pix_style = False
  • block = basic for lower gpu memory consumption.

Related papers

Problems

  • For large rectangle holes(ratio > 0.5), the results might be unnatural.
  • When using partial conv in decoder, the model failed, this is because the masked area is not filled and learned. But the paper uses partial conv in decoder. I have no idea how to get it right.
  • Progressive growing training scheme uses more gpu memory than I think. May be it can be optimized. May be even the implementation is problematic.
  • When pix2pix_style=True, gradient of discriminator vanished. But why?