/awesome-normalization-techniques

Papers for normalization techniques, released codes collections.

Primary LanguagePython

Awesome-Normalization-Techniques Awesome

Papers for normalization techniques, released codes collections.

Any addition or bug feel free to open an issue or pull requests.

2020 - 2019 - 2018 - 2017 - 2016

2020

  • A New Look at Ghost Normalization
    • Neofytos Dimitriou, Ognjen Arandjelovic
    • [Paper]
  • TaskNorm: Rethinking Batch Normalization for Meta-Learning (ICML 2020)
    • John Bronskill, Jonathan Gordon, James Requeima, Sebastian Nowozin, Richard E Turner
    • [Paper]
    • [Python Reference]
  • Instance Enhancement Batch Normalization: an Adaptive Regulator of Batch Noise (AAAI 2020)
  • Towards Stabilizing Batch Statistics in BackWard Propagation of Batch Normalization (ICLR 2020)
    • Junjie Yan, Ruosi Wan, Xiangyu Zhang, Wei Zhang, Yichen Wei, Jian Sun
    • [Paper]
  • Rethinking Spatially-Adaptive Normalization
    • Zhentao Tan, Dongdong Chen, Qi Chu, Menglei Chai, Jing Liao, Mingming He, Lu Yuan, Nenghai Yu
    • [Paper]
  • Filter Response Normalization Layer: Eliminating Batch Dependence in the Training of Deep Neural Networks (CVPR 2020)
  • Extended Batch Normalization
  • Knowledge Distillation via Adaptive Instance Normalization
    • Jing Yang, Brais Martinez, Adrian Bulat, and Georgios Tzimiropoulos
    • [Paper]
  • Four Things Everyone Should Know to Improve Batch Normalization
    • Cecilia Summers, Michael J. Dinneen
    • [Paper]
  • Region Normalization for Image Inpainting (AAAI 2020)
    • Tao Yu, Zongyu Guo, Xin Jin, Shilin Wu, Zhibo Chen, Weiping Li, Zhizheng Zhang, Sen Liu
    • [Paper]
    • [Python Reference]
  • Local Context Normalization: Revisiting Local Normalization (CVPR 2020)
    • Anthony Ortiz , Caleb Robinson, Mahmudulla Hassan, Dan Morris2, Olac Fuentes1, Christopher Kiekintveld, and Nebojsa Jojic
    • [Paper]
  • Attentive Normalization for Conditional Image Generation (CVPR 2020)
  • Cross-Iteration Batch Normalization
  • SEAN: Image Synthesis with Semantic Region-Adaptive Normalization (CVPR 2020)
  • Evolving Normalization-Activation Layers
    • Hanxiao Liu, Andrew Brock, Karen Simonyan, Quoc V. Le
    • [Paper]

2019

  • Differentiable Dynamic Normalization for Learning Deep Representation (PMLR 2019/ICML 2019)
    • Ping Luo, Peng Zhanglin, Shao Wenqi, Zhang Ruimao, Ren Jiamin, Wu Lingyun
    • [Paper]
  • Mean Spectral Normalization of Deep Neural Networks for Embedded Automation
  • An Empirical Study of Batch Normalization and Group Normalization in Conditional Computation
    • Vincent Michalski, Vikram Voleti, Samira Ebrahimi Kahou, Anthony Ortiz, Pascal Vincent, Chris Pal, Doina Precup
    • [Paper]
  • Generalized Batch Normalization: Towards Accelerating Deep Neural Networks (AAAI 2019)
    • Xiaoyong Yuan, Zheng Feng, Matthew Norton, Xiaolin Li
    • [Paper]
  • Split Batch Normalization: Improving Semi-Supervised Learning under Domain Shift
    • Michał Zając, Konrad Zolna, Stanisław Jastrzębski
    • [Paper]
  • Channel Normalization in Convolutional Neural Network avoids Vanishing Gradients (ICML 2019)
  • Unpaired Image Translation via Adaptive Convolution-based Normalization
    • Wonwoong Cho, Kangyeol Kim, Eungyeup Kim, Hyunwoo J. Kim, Jaegul Choo
    • [Paper]
  • EvalNorm: Estimating Batch Normalization Statistics for Evaluation (ICCV 2019)
    • Saurabh Singh, Abhinav Shrivastava
    • [Paper]
  • Online Normalization for Training Neural Networks (NIPS 2019)
  • Transferable Normalization: Towards Improving Transferability of Deep Neural Networks (NIPS 2019)
  • Iterative Normalization: Beyond Standardization Towards Efficient Whitening (CVPR 2019)
  • Domain-Specific Batch Normalization for Unsupervised Domain Adaptation (CVPR 2019)
  • Attentive Normalization
  • Rethinking Normalization and Elimination Singularity in Neural Networks
    • Siyuan Qiao, Huiyu Wang, Chenxi Liu, Wei Shen, Alan Yuille
    • [Paper]
  • Dynamic Instance Normalization for Arbitrary
    • Yongcheng Jing, Xiao Liu, Yukang Ding, Xinchao Wang, Errui Ding, Mingli Song, Shilei Wen
    • [Paper]
  • Semantic Image Synthesis with Spatially-Adaptive Normalization (CVPR 2019)
  • Differentiable Learning-to-Normalize via Switchable Normalization (ICLR 2019)
  • SSN: Learning Sparse Switchable Normalization via Sparsestmax (CVPR 2019)
    • Wenqi Shao, Tianjian Meng, Jingyu Li, Ruimao Zhang, Yudian Li, Xiaogang Wang, Ping Luo
    • [Paper]
  • A Mean Field Theory of Batch Normalization
    • Greg Yang, Jeffrey Pennington, Vinay Rao, Jascha Sohl-Dickstein, Samuel S. Schoenholz
    • [Paper]
  • Restructuring Batch Normalization to Accelerate CNN Training
    • Wonkyung Jung, Daejin Jung, Byeongho Kim, Sunjung Lee, Wonjong Rhee, Jung Ho Ahn
    • [Paper]
  • A Novel Convolutional Neural Network for Image Steganalysis with Shared Normalization (TMM 2019)
    • Songtao Wu, Sheng-hua Zhong, and Yan Liu
    • [Paper]
  • Training Faster by Separating Modes of Variation in Batch-normalized Models (TPAMI 2019)
    • Mahdi M. Kalayeh, Mubarak Shah
    • [Paper]
  • Uncertainty Estimation via Stochastic Batch Normalization (ISNN 2019)
    • Andrei Atanov, Arsenii Ashukha, Dmitry Molchanov, Kirill Neklyudov, Dmitry Vetrov
    • [Paper]
    • [Python Reference]

2018

  • Decorrelated Batch Normalization (CVPR 2018)
  • MegDet: A Large Mini-Batch Object Detector (CVPR 2018)
    • Chao Peng, Tete Xiao, Zeming Li, Yuning Jiang, Xiangyu Zhang, Kai Jia, Gang Yu, Jian Sun
    • [Paper]
  • Batch-Instance Normalization for Adaptively Style-Invariant Neural Networks (NIPS 2018)
    • Hyeonseob Nam, Hyo-Eun Kim
    • [Paper]
  • Kalman Normalization: Normalizing Internal Representations Across Network Layers (NIPS 2018)
    • Guangrun Wang, Jiefeng Peng, Ping Luo, Xinjiang Wang, Liang Lin
    • [Paper]
  • Do Normalization Layers in a Deep ConvNet Really Need to Be Distinct?
    • Ping Luo, Zhanglin Peng, Jiamin Ren, Ruimao Zhang
    • [Paper]
  • L1-Norm Batch Normalization for Efficient Training of Deep Neural Networks (TNNLS)
    • Shuang Wu , Guoqi Li, Lei Deng, Liu Liu, Dong Wu, Yuan Xie, Luping Shi
    • [Paper]
  • In-Place Activated BatchNorm for Memory-Optimized Training of DNNs (CVPR 2018)
  • Group Normalization (ECCV 2018)
  • Spectral Normalization for Generative Adversarial

2017

  • Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models (NIPS 2017)
  • Modulating Early Visual Processing by Language (NIPS 2017)
    • Harm de Vries, Florian Strub, Jérémie Mary, Hugo Larochelle, Olivier Pietquin, Aaron Courville
    • [Paper]
    • [Python Reference]
  • Instance Normalization: the Missing Ingredient for Fast Stylization
  • Cosine Normalization: Using Cosine Similarity Instead of Dot Product in Neural Networks
    • Luo Chunjie, Zhan Jianfeng, Wang Lei, Yang Qiang
    • [Paper]
  • Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization (ICCV 2017)
  • Normalizing the Normalizers: Comparing and Extending Network Normalization Schemes
    • Mengye Ren, Renjie Liao, Raquel Urtasun, Fabian H. Sinz, Richard S. Zemel􏲗
    • [Paper]
  • Recurrent Batch Normalization
    • Tim Cooijmans, Nicolas Ballas, César Laurent, Çag ̆lar Gülçehre, Aaron Courville
    • [Paper]

2016

  • Layer Normalization

  • Normalization Propagation: A Parametric Technique for Removing Internal Covariate Shift in Deep Networks

    • Devansh Arpit, Yingbo Zhou, Bhargava U. Kota, Venu Govindaraju, SUNY Buffalo
    • [Paper]

  • Weight Normalization: a Simple Reparameterization to Accelerate Training of Deep Neural Networks (NIPS 2016)

    • Tim Salimans, Diederik P. Kingma
    • [Paper]

2015

  • Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift