Is there any special consideration of initialization ?

[CoinCheung]

Hi,

Since we use kaiming_normal to initialize conv layers followed by relu, and xavier to initiailize linear conv layers. Is there any specification to normalize the conv layers when using FReLU. Besides, there is a conv layer within the FReLU module, how could I initialize this embedded conv layer?

[nmaac]

@CoinCheung For the dw-conv in the FReLU, we suggest to initialize with mean=0, std=0.01; for other conv layers and linear layers we suggest the normal initializations.

[CoinCheung]

Thanks, I will have a try of this.