Orthogonal weight init
Kaixhin opened this issue · 7 comments
Shouldn't the repo be called pytorch-SRGAN instead of pytorch-SRResNet?
Anyway, the same group mention using orthogonal weight initialisation in the ESPCN paper released roughly at the same time period - even if they haven't specified it for SRGAN, it's definitely worth trying. The text is as follows:
Biases are initialised to 0 and weights use orthogonal initialisation with gain √2 following recommendations in [30].
So for all convolutional layers you'll want:
nn.init.orthogonal(layer.weight, math.sqrt(2))
nn.init.constant(layer.bias, 0)Also there is a v5 of the paper with I think more training details, so worth checking carefully to see if there's anything you missed.
You may also want to check out the pytorch superresolution example for info on weight init, if need be.
They use orthogonal init there (but it was written before nn.init) so indeed you should do that. Make note that the √2 is for layers preceding ReLUs, and therefore the final (output) layer does not have a gain (or uses the default of 1).
Hi, I have a question, why not implement the VGG loss in the original paper.
@CasdDesnDR I updated the code for content loss support.