/awesome-neural-ode

A collection of resources regarding the interplay between differential equations, dynamical systems, deep learning, control and optimization.

MIT LicenseMIT

awesome-neural-ode

A collection of resources regarding the interplay between differential equations, dynamical systems, deep learning, control and scientific machine learning.

NOTE: Feel free to suggest additional resources via Issues or Pull Requests.

Table of Contents

Differential Equations in Deep Learning

General Architectures

  • Recurrent Neural Networks for Multivariate Time Series with Missing Values: Scientific Reports18

  • Learning unknown ODE models with Gaussian processes: arXiv18

  • Deep Equilibrium Models: NeurIPS19

  • Fast and Deep Graph Neural Networks: AAAI20

  • Hamiltonian Neural Networks: NeurIPS19

  • Deep Lagrangian Networks: Using Physics as Model Prior for Deep Learning: ICLR19

  • Lagrangian Neural Networks: ICLR20 DeepDiffEq

Neural ODEs

  • Neural Ordinary Differential Equations: NeurIPS18

  • Graph Neural Ordinary Differential Equations: arXiv19

  • Augmented Neural ODEs: NeurIPS19

  • Latent ODEs for Irregularly-Sampled Time Series: arXiv19

  • ODE2VAE: Deep generative second order ODEs with Bayesian neural networks: NeurIPS19

  • Symplectic ODE-Net: Learning Hamiltonian Dynamics with Control: arXiv19

  • How to Train you Neural ODE: arXiv20

  • Dissecting Neural ODEs: arXiv20

  • Stable Neural Flows: arXiv20

  • On Second Order Behaviour in Augmented Neural ODEs arXiv20

  • Neural Controlled Differential Equations for Irregular Time Series: arXiv20

Training of Neural ODEs

  • Accelerating Neural ODEs with Spectral Elements: arXiv19

  • Adaptive Checkpoint Adjoint Method for Gradient Estimation in Neural ODE: ICML20

Neural SDEs

  • Neural SDE: Stabilizing Neural ODE Networks with Stochastic Noise: arXiv19

  • Neural Jump Stochastic Differential Equations: arXiv19

  • Towards Robust and Stable Deep Learning Algorithms for Forward Backward Stochastic Differential Equations: arXiv19

  • Scalable Gradients and Variational Inference for Stochastic Differential Equations: AISTATS20

Normalizing Flows

  • Monge-Ampère Flow for Generative Modeling: arXiv18

  • FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models: ICLR19

  • Equivariant Flows: sampling configurations for multi-body systems with symmetric energies: arXiv18

Applications

  • Learning Dynamics of Attention: Human Prior for Interpretable Machine Reasoning: NeurIPS19

Deep Learning Methods for Differential Equations

Solving Differential Equations

Learning PDEs

  • PDE-Net: Learning PDEs From Data: ICML18

Model Discovery

  • Universal Differential Equations for Scientific Machine Learning: arXiv20

Deep Control

Model-Predictive-Control

  • Differentiable MPC for End-to-end Planning and Control: NeurIPS18

Dynamical System View of Deep Learning

Recurrent Neural Networks

  • A Comprehensive Review of Stability Analysis of Continuous-Time Recurrent Neural Networks: IEEE Transactions on Neural Networks 2006

  • AntysimmetricRNN: A Dynamical System View on Recurrent Neural Networks: ICLR19

  • Recurrent Neural Networks in the Eye of Differential Equations: arXiv19

  • Visualizing memorization in RNNs: distill19

  • One step back, two steps forward: interference and learning in recurrent neural networks: arXiv18

  • Reverse engineering recurrent networks for sentiment classification reveals line attractor dynamics: arXiv19

  • System Identification with Time-Aware Neural Sequence Models: AAAI20

  • Universality and Individuality in recurrent networks: NeurIPS19

Theory and Perspectives

  • Deep Learning Theory Review: An Optimal Control and Dynamical Systems Perspective: arXiv19

  • Stable Architectures for Deep Neural Networks: IP17

  • Beyond Finite Layer Neural Network: Bridging Deep Architects and Numerical Differential Equations: ICML18

  • Review: Ordinary Differential Equations For Deep Learning: arXiv19

Optimization

  • Gradient and Hamiltonian Dynamics Applied to Learning in Neural Networks: NIPS96

  • Maximum Principle Based Algorithms for Deep Learning: JMLR17

  • Hamiltonian Descent Methods: arXiv18

  • Port-Hamiltonian Approach to Neural Network Training: CDC19, code

  • An Optimal Control Approach to Deep Learning and Applications to Discrete-Weight Neural Networks: arXiv19

  • Optimizing Millions of Hyperparameters by Implicit Differentiation: arXiv19

  • Shadowing Properties of Optimization Algorithms: NeurIPS19

Software and Libraries

Python

  • torchdyn: PyTorch library for all things neural differential equations. repo, docs

  • torchdiffeq: Differentiable ODE solvers with full GPU support and O(1)-memory backpropagation: repo

  • torchsde: Stochastic differential equation (SDE) solvers with GPU support and efficient sensitivity analysis: repo

  • torchSODE: PyTorch Block-Diagonal ODE solver: repo

Julia

  • DiffEqFlux: Neural differential equation solvers with O(1) backprop, GPUs, and stiff+non-stiff DE solvers. Supports stiff and non-stiff neural ordinary differential equations (neural ODEs), neural stochastic differential equations (neural SDEs), neural delay differential equations (neural DDEs), neural partial differential equations (neural PDEs), and neural jump stochastic differential equations (neural jump diffusions). All of these can be solved with high order methods with adaptive time-stepping and automatic stiffness detection to switch between methods. repo

  • NeuralNetDiffEq: Implementations of ODE, SDE, and PDE solvers via deep neural networks: repo

Websites and Blogs

  • Scientific ML Blog (Chris Rackauckas and SciML): link