Watch your Up-Convolution: CNN Based Generative Deep Neural Networks are Failing to Reproduce Spectral Distributions

This repository provides the official Python implementation of Watch your Up-Convolution: CNN Based Generative Deep Neural Networks are Failing to Reproduce Spectral Distributions (Paper: https://arxiv.org/abs/2003.01826).

Common up-convolution methods are inducing heavy spectral distortions into generated images. (Left) Statistics (mean and variance) after azimuthal integration over the power-spectrum of real and GAN generated images. (Right) Results of the same experiments as above, adding our proposed spectral loss during GAN training.

Spectral Regularization

Since common generative network architectures are mostly exclusively using image-space based loss functions, it is not possible to capture and correct spectral distortions directly. Hence, we extend existing GAN architectures in two ways:

Spectral loss term

Replace last last up-convolution unit.

Dependencies

Tested on Python 3.6.x.

Pytorch (1.1.0)
NumPy (1.16.2)
Opencv (4.0.0)
Matplotlib (3.1.1)

Downloading Dataset

Link to download CelebA dataset.

Training Netwroks

Training vanilla models

We train different GAN models using this repo. Then, we employ our Visualization script to analyse the frequency behaviour.

Training Spectral Regularization models

Make the following changes to incorporate the regularizer. Then, train the model.

Substitue train.py for train_spectrum.py
Substitue module.py for module_spectrum.py

Click here to go the Regularization implementation.

Deepfake detection

Experiments and code for the deepfake detection parts of the paper can be found in the repository of our prior Arxiv pre-print.

Paper: https://arxiv.org/abs/1911.00686
Code: https://github.com/cc-hpc-itwm/DeepFakeDetection

Citation

If this work is useful for your research, please cite our paper:

@misc{durall2020upconv,
    title={Watch your Up-Convolution: CNN Based Generative Deep Neural Networks are Failing to Reproduce Spectral Distributions},
    author={Ricard Durall and Margret Keuper and Janis Keuper},
    year={2020},
    eprint={2003.01826},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

keshik6/UpConv