PhotoWCT2: Compact Autoencoder for Photorealistic Style Transfer Resulting from Blockwise Training and Skip Connections of High-Frequency Residuals
Code for our WACV 2022 paper PhotoWCT2: Compact Autoencoder for Photorealistic Style Transfer Resulting from Blockwise Training and Skip Connections of High-Frequency Residuals
- Our PhotoWCT2 transfers stronger effects than WCT2 and comparable effects to those transferred by PhotoWCT.
- Our PhotoWCT2 preserves better high-frequency details than PhotoWCT (three examples are shown below).
We apply the proposed blockwise training to train two models:
-
A pre-trained VGG-19 encoder (from input layer to conv41 layer; fixed during training) and a blockwisely trained decoder which can reproduce the conv31, conv21, conv11 features and the input image. The ZCA transformations are embedded at the bottleneck and the reproduced convN1 layers in the decoder.
- The model is in
utils/model_conv.py
and the associated checkpoint is inckpts/ckpts-conv
. - A demo that uses this model to stylize example images in
figures/
is shown inconv_demo.ipynb
. The resulting stylized images are inresults/conv
.
- The model is in
-
A pre-trained VGG-19 encoder (from input layer to relu41 layer; fixed during training) and a blockwisely trained decoder which can reproduce the relu31, relu21, relu11 features and the input image. The ZCA transformations are embedded at the bottleneck and the reproduced reluN1 layers in the decoder.
- The model is in
utils/model_relu.py
and the associated checkpoint is inckpts/ckpts-relu
. - A demo that uses this model to stylize example images in
figures/
is shown inrelu_demo.ipynb
. The resulting stylized images are inresults/relu
.
- The model is in
Stylization with both models requires guided filtering in utils/photo_gif.py
as the post-processing. The file is adapted from the one used in the PhotoWCT code.
The performance of these two models is similar. Please choose whichever you would like to use.
train.py
is the training code for our PhotoWCT2 model. The usage is provided in the file.
- tensorflow v2.0.0 or above (we developed the models with tf-v2.4.1 and we also tested them in tf-v2.0.0)
If you find this repo useful, please cite our paper PhotoWCT2: Compact Autoencoder for Photorealistic Style Transfer Resulting from Blockwise Training and Skip Connections of High-Frequency Residuals published in WACV 2022.