Kalman Normalization

By Guangrun Wang, Jiefeng Peng, Ping Luo, Xinjiang Wang and Liang Lin.

Sun Yat-sen University (SYSU), the Chinese University of Hong Kong (CUHK), SenseTime Group Ltd.

Results
Introduction
Citation
Dependencies
Usage

Results

Under the context of micro-batches(batch size = 2), the validataion curves on CIFAR10:

top line: Batch Normalization(BN); mid line: Group Normalization; bottom line: Kalman Normalization

Under the context of large-batches(batch size = 128), the validataion curves on CIFAR10:

top line: Batch Normalization(BN); bottom line: Kalman Normalization)

Introduction

This repository contains the original models described in the paper "Batch Kalman Normalization: Towards Training Deep Neural Networks with Micro-Batches" (https://arxiv.org/abs/1802.03133). These models are those used in ILSVRC and CIFAR

Citation

If you use these models in your research, please cite:

@article{wang2018batch,
	title={Batch Kalman Normalization: Towards Training Deep Neural Networks with Micro-Batches},
	author={Wang, Guangrun and Peng, Jiefeng and Luo, Ping and Wang, Xinjiang and Lin, Liang},
	journal={arXiv preprint arXiv:1802.03133},
	year={2018}
}

Dependencies

Python 2.7 or 3
TensorFlow >= 1.3.0
Tensorpack The code depends on Yuxin Wu's Tensorpack. For convenience, we provide a stable version 'tensorpack-installed' in this repository.
```
# install tensorpack locally:
cd tensorpack-installed
python setup.py install --user
```

Usage

To run Group Normalization, use:

cd KalmanNorm
python cifar-gn.py --gpu 0 -n 5 --log_dir gn

To run Batch Normalization under the context of micro-batches, use:

cd KalmanNorm
python cifar-bn-microbatch.py --gpu 0 -n 5 --log_dir  bn-microbatch

To run Kalman Normalization under the context of micro-batches, use:

cd KalmanNorm
python cifar-kn-microbatch.py --gpu 0 -n 5 --log_dir  kn-microbatch

To run Batch Normalization under the context of large-batches, use:

cd KalmanNorm
python cifar-bn-largebatch.py   --gpu 0 -n 18 --log_dir  bn-largebatch

To run Kalman Normalization under the context of large-batches, use:

cd KalmanNorm
python cifar-kn-largebatch.py   --gpu 0 -n 18 --log_dir  kn-largebatch