Why concatenate after norm ?

[Medabid1]

Hello,
could you plz explain, why double the input after the batchNorm and concat [input, -input]?

thanks

[chrischute]

This only happens for checkerboard coupling, because half of the inputs get zero-ed out by the checkerboard mask. Doubling re-scales the variance so the batchnorm operation is the same as it would be if you only considered the non-zeroed-out inputs.

The corresponding code released by the authors of the paper can be found here.

[KeAWang]

@chrischute Do you remember why you concatenate x and -x? Also do you know why the authors also concatenate x (res in their code) and the mask?

[chrischute]

I was following the original paper's code as closely as possible, just porting from TensorFlow to PyTorch. I'm not sure why they concatenate x and -x... It might be a trick to make the mean 0. I never understood why they concatenate res and mask. If you ask the original authors I bet they'd have an answer.

[KeAWang]

Got it. Thanks!