Custom Weight Initialization

[glenn-jocher]

I noticed you use code for custom weight initialization:

ghostnet.pytorch/ghost_net.py

Lines 162 to 169 in 2c90e67

	def _initialize_weights(self):
	for m in self.modules():
	if isinstance(m, nn.Conv2d):
	nn.init.kaiming_normal_(m.weight, mode='fan_out', nonlinearity='relu')
	elif isinstance(m, nn.BatchNorm2d):
	m.weight.data.fill_(1)
	m.bias.data.zero_()

I've not seen this before. Is there a reason behind this specific strategy? Do you know the effect this has on the training, and have you compared this with the pytorch default weight initialization? Thank you!

[iamhankai]

kaiming_normal_ is a commonly used initialization strategy.

[glenn-jocher]

@iamhankai thank you! Do you know what the default pytorch weights init strategy is?

I suppose this makes for easier comparisons with the TF version of ghostnet to use the same strategy on both?

[iamhankai]

@glenn-jocher TF version of ghostnet also used Kaiming normal initialization.