About batch normalization

[utkarshojha]

The batch_normalization_layer() function doesn't compute the statistics of population i.e. population mean and variance. The part implemented is only taking care of the training procedure (batch statistics), but while testing one will need the population statistics

[dongzhuoyao]

yes, the bn implementation is wrong! please notice

[jingjing-gong]

agree, batch_normalization is wrong.

[wenxinxu]

Thank you very much for pointing it out. I will fix it.

[WoNiuHu]

@wenxinxu, @utkarsh2254 @dongzhuoyao hi , so could you please tell me how to fix this bug?

[WoNiuHu]

So, what's the difference ? pop_mean and pop_var will not changed during training and pop_var does not used. Best regards. At 2017-11-09 14:32:02, "Utkarsh Ojha" <notifications@github.com> wrote: In the resnet.py file, batch_normalization_layer function can be modified like this `def batch_normalization_layer(input_layer, dimension): ''' Helper function to do batch normalziation :param input_layer: 4D tensor :param dimension: input_layer.get_shape().as_list()[-1]. The depth of the 4D tensor :return: the 4D tensor after being normalized ''' mean, variance = tf.nn.moments(input_layer, axes=[0, 1, 2]) beta = tf.get_variable('beta', dimension, tf.float32, initializer=tf.constant_initializer(0.0, tf.float32),trainable=False) gamma = tf.get_variable('gamma', dimension, tf.float32, initializer=tf.constant_initializer(1.0, tf.float32),trainable=False) pop_mean = tf.get_variable('pop_mean', dimension, trainable=False) pop_var = tf.get_variable('pop_var', dimension, trainable=False) bn_layer = tf.nn.batch_normalization(input_layer, pop_mean, variance, beta, gamma, BN_EPSILON) return bn_layer` — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[dongzhuoyao]

hi, @WoNiuHu , you should calculate the bn' mean, variance by EMA.

here is a practical demo you can follow: https://github.com/ppwwyyxx/tensorpack/blob/master/tensorpack/models/batch_norm.py#L161

[TyrionChou]

you can change tf.nn.batch_normalization to tf.layers.batch_normalization