Awesome Normalizing Flows

A list of awesome resources for understanding and applying normalizing flows (NF). It's a relatively simple yet powerful new tool in statistics for constructing expressive probability distributions from simple base distribution using a chain (flow) of trainable smooth bijective transformations (diffeomorphisms).

^{Figure inspired by Lilian Weng. Created in TikZ. Source code found here.}

📝 Publications

April 1, 2011 - Iterative Gaussianization: from ICA to Random Rotations by Laparra et. al.

Normalizing flows in the form of Gaussianization in an iterative format. Also shows connections to information theory.
Oct 30, 2014 - Non-linear Independent Components Estimation by Laurent Dinh, David Krueger, Yoshua Bengio.

Introduces the additive coupling layer (NICE) and shows how to use it for image generation and inpainting.
Feb 12, 2015 - Masked Autoencoder for Distribution Estimation by Mathieu Germain, Karol Gregor, Iain Murray, Hugo Larochelle.

Introduces MADE, a feed-forward network that uses carefully constructed binary masks on its weights to control the precise flow of information through the network. The masks ensure that each output unit receives signals only from input units that come before it in some arbitrary order. Yet all outputs can be computed in a single pass.
A popular and efficient method to bestow flows with autoregressivity is to construct them from MADE nets.
May 21, 2015 - Variational Inference with Normalizing Flows by Danilo Rezende, Shakir Mohamed.

They show how to go beyond mean-field variational inference by using flows to increase the flexibility of the variational family.
May 27, 2016 - Density estimation using Real NVP by Laurent Dinh, Jascha Sohl-Dickstein, Samy Bengio.

They introduce the affine coupling layer (RNVP), a major improvement in terms of flexibility over the additive coupling layer (NICE) with unit Jacobian while keeping a single-pass forward and inverse transformation for fast sampling and density estimation, respectively.
Jun 15, 2016 - Improving Variational Inference with Inverse Autoregressive Flow by Diederik Kingma et al.
Mar 6, 2017 - Multiplicative Normalizing Flows for Variational Bayesian Neural Networks by Christos Louizos, Max Welling.

They introduce a new type of variational Bayesian neural network that uses flows to generate auxiliary random variables which boost the flexibility of the variational family by multiplying the means of a fully-factorized Gaussian posterior over network parameters. This turns the usual diagonal covariance Gaussian into something that allows for multimodality and non-linear dependencies between network parameters.
May 19, 2017 - Masked Autoregressive Flow for Density Estimation by George Papamakarios, Theo Pavlakou, Iain Murray.

To improve predictive accuracy and uncertainty in Bayesian neural networks, they interpret multiplicative noise in neural network parameters as auxiliary random variables and show how to model these using flows. As in variational inference, the idea is again to use the power of flows to augment the approximate posterior while maintaining tractability.

^{Figure created in TikZ. Source code found here.}
Mar 15, 2018 - Sylvester Normalizing Flow for Variational Inference by Rianne van den Berg, Leonard Hasenclever, Jakub M. Tomczak, Max Welling.

Introduces Sylvester normalizing flows which remove the single-unit bottleneck from planar flows for increased flexibility in the variational posterior.
3 July, 2018 - Deep Density Destructors by Inouye & Ravikumar

Normalizing flows but from an iterative perspective. Features a Tree-based density estimator.
9 Jul, 2018 - Glow: Generative Flow with Invertible 1x1 Convolutions by Kingma, Dhariwal.

They show that flows using invertible 1x1 convolution achieve high likelihood on standard generative benchmarks and can efficiently synthesize realistic-looking, large images.
2 Oct, 2018 - FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models by Grathwohl & Chen et. al.

Uses Neural ODEs as a solver to produce continuous-time normalizing flows (CNF)
9 Apr, 2019 - Block Neural Autoregressive Flow - De Cao et. al.
30 May, 2019 - Graph Normalizing Flows by Jenny Liu et al. A new, reversible graph network for prediction and generation.

They perform similarly to message passing neural networks on supervised tasks, but at significantly reduced memory use, allowing them to scale to larger graphs. Combined with a novel graph auto-encoder for unsupervised learning, graph normalizing flows are a generative model for graph structures.
25 Aug, 2019 (Revised Aug 6, 2020) - Normalizing Flows: An Introduction and Review of Current Methods by Kobyzev et al.

Another very thorough and very readable review article going through the basics of NFs as well as some of the state-of-the-art. Also highly recommended.
10 Jun, 2019 - Neural Spline Flows by Conor Durkan et. al.

Uses monotonic ration splines as a coupling layer. This is currently one of the state of the art.
Dec 5, 2019 - Normalizing Flows for Probabilistic Modeling and Inference by Papamakarios et al.

A thorough and very readable review article by some of the guys at DeepMind involved in the development of flows. Highly recommended.
15 Jan, 2020 (Revised Apr 13, 2020) - Invertible Generative Modeling using Linear Rational Splines by Dolatabadi et. al.

A successor to the Neural spline flows which features an easy-to-compute inverse.
Mar 4, 2020 - Gaussianization Flows by Meng et. al.

Uses a repeated composition of trainable kernel layers and orthogonal transformations. Very competitive versus some of the SOTA like Real-NVP, Glow and FFJORD.

📺 Videos

Oct 4, 2018 - Sylvester Normalizing Flow for Variational Inference by Rianne van den Berg.

Introduces Sylvester normalizing flows which remove the single-unit bottleneck from planar flows for increased flexibility in the variational posterior.
Sep 24, 2019 - Graph Normalizing Flows by Jenny Liu (University of Toronto, Vector Institute).

Introduces a new graph generating model for use e.g. in drug discovery, where training on molecules that are known to bind/dissolve/etc. may help to generate novel, similarly effective molecules.
Oct 9, 2019 - A primer on normalizing flows by Laurent Dinh (Google Brain).

The first author on both the NICE and RNVP papers and one of the first in this field gives an introductory talk at "Machine Learning for Physics and the Physics of Learning 2019".
Dec 6, 2019 - What are normalizing flows? by Ari Seff (Princeton).

A great 3blue1brown-style video explaining the basics of normalizing flows.
Flow Models by CS294-158-SP20 Deep, Unsupervised Learning, Spring 2020 (Berkeley)

A really thorough explanation of normalizing flows. Also includes some sample code.

🌐 Blog Posts

Jan 17, 2018 - Normalizing Flows Tutorial by Eric Jang.

Part 1: Distributions and Determinants. Part 2: Modern Normalizing Flows. Lots of great graphics.
Apr 3, 2018 - Normalizing Flows by Adam Kosiorek.
Oct 13, 2018 - Flow-based Deep Generative Models by Lilian Weng.

🧑‍💻 Code

Jul 11, 2017 - normalizing_flows_overview.ipynb by PyMC3.

A very helpful notebook showcasing how to work with flows in practice and comparing it to PyMC3's NUTS-based HMC kernel.
Sep 1, 2018 - pytorch-flows by Ilya Kostrikov.

PyTorch implementations of density estimation algorithms: MAF, RNVP, Glow.
Dec 30, 2018 - normalizing_flows by Kamen Bliznashki.

Pytorch implementations of density estimation algorithms: BNAF, Glow, MAF, RealNVP, planar flows.
Dec 9, 2019 - pytorch-normalizing-flows by Andrej Karpathy.

A Jupyter notebook with PyTorch implementations of the most commonly used flows: NICE, RNVP, MAF, Glow, NSF.
pytorch_flows by acids-ircam

A great repo with some basic implementations of normalizing flows from scratch.
nflows by bayesiains

A suite of most of the SOTA methods using PyTorch. From an ML group in Edinburgh. They created the current SOTA spline flows. Almost as complete as you'll find from a single repo.
Normalizing Flows - Introduction (Part 1) by pyro.ai

A tutorial about how to use the pyro-ppl library (based on PyTorch) to use Normalizing flows. They have some of the SOTA methods including NSF and MAF.
destructive-deep-learning by David Inouye

An entire suite of iterative methods to normalizing flows. Includes the Tree-based method as well as Gaussianization methods.
Neural Transport by numpyro

Features an example of how Normalizing flows can be used to get more robust posteriors from Monte Carlo methods. Uses the numpyro library which is a PPL with JAX as the backend. The NF implementations include the basic ones like IAF and BNAF.
NuX by Information Fusion Labs (UMass)

A library that has normalizing flows using JAX as the backend. Has some of the SOTA methods. They also feature a surjective flow via quantization.
jax-flows by Chris Waites

Another library that has normalizing flows using JAX as the backend. Has some of the SOTA methods.
Density Estimation with Neural ODEs and Density Estimation with FFJORDs by torchdyn

Example of how to use FFJORD as a continuous normalizing flow (CNF). Based on the PyTorch suite torchdyn which offers continuous neural architectures.

🎉 Open to Suggestions!