On the Compatibility of GAN
Closed this issue · 2 comments
Hello,
Sorry to bother you again. In fact, I noticed that your compressai seems to have integrated quite comprehensive compression models, from FactorizedPrior to ELIC. I have greatly benefited from using the models you have built in some of my previous attempts and explorations :).
Now, I would like to try using the conditional GAN mentioned in the High-Fidelity Generative Image Compression paper to further improve the quality of image reconstruction. However, according to the principles of GAN training, it seems to be not very compatible with the components in your code, such as the lack of alternating training of G/D in the main function, and the inclusion of optimizers and loss functions.
I would like to know if you have explored this area, or if you have any relevant suggestions? For example, incorporating the GAN discriminator and loss as components into the current JointAutoregressiveHierarchicalPriors or other model architectures. (Although I tried to directly find the HFGIC source code, I noticed that they haven't been updated for a long time, and many components, including CUDA, are not compatible with the current GPU architecture.)
Looking forward to your reply!
I haven't run GAN-based methods personally, but you're right that it would need a different training loop with alternating G/D training.
Here's a list of implementations I found after a quick search:
- https://github.com/tensorflow/compression/tree/master/models/hific (TensorFlow)
- https://github.com/Justin-Tan/high-fidelity-generative-compression (PyTorch)
- https://github.com/search?q=repo%3Afacebookresearch%2FNeuralCompression%20hific&type=code (PyTorch, training instructions)
- https://github.com/iwa-shi/fidelity_controllable_compression (PyTorch, not HiFiC but GAN-based)
Until we have a good human-like perceptual metric (i.e., better than MSE or MS-SSIM), GAN-based methods theoretically should have higher human-assessed perceptual quality at very low bitrates. Perhaps at some point in the future, we might add an implementation of HiFiC to CompressAI for convenience, since GANs are certainly an interesting paradigm.
Thank you for your response. I came up with this question after reviewing Justin Tan's code. :) I might try to modify some parts of the code, and I also look forward to the day when you add the HiFiC architecture to CompressAI!