/awesome-information-geometry

About A collection of AWESOME things about information geometry Topics

MIT LicenseMIT

awesome-information-geometry

MIT License

This repo is a collection of AWESOME things about information geometry, including papers, code, etc. Feel free to star and fork.

Contents

Books

Papers

Survey

Information Geometry for Neural Networks

  • A Rate-Distortion View of Uncertainty Quantification (ICML, 2024)
  • Riemannian SAM: Sharpness-Aware Minimization on Riemannian Manifolds (NeurIPS, 2023)
  • The Geometry of Neural Nets' Parameter Spaces Under Reparametrization (NeurIPS, 2023)
  • Group Equivariant Sparse Coding (GSI, 2023)
  • Can Generalised Divergences Help for Invariant Neural Networks? ([GSI, 2023](Can Generalised Divergences Help for Invariant Neural Networks?))
  • Continuous Kendall Shape Variational Autoencoders (GSI, 2023)
  • Functional Properties of PDE-Based Group Equivariant Convolutional Neural Networks (GSI, 2023)
  • A Neurogeometric Stereo Model for Individuation of 3D Perceptual Units (GSI, 2023)
  • Fisher-Legendre (FishLeg) optimization of deep neural networks (ICLR, 2023)
  • The Fisher–Rao loss for learning under label noise (Information Geometry, 2022)
  • A Reparametrization-Invariant Sharpness Measure Based on Information Geometry (NeurIPS, 2022)
  • IGAGCN: Information geometry and attention-based spatiotemporal graph convolutional networks for traffic flow prediction (Neural Networks, 2021)
  • Catastrophic Fisher Explosion: Early Phase Fisher Matrix Impacts Generalization (ICML, 2021)
  • An Information-Geometric Distance on the Space of Tasks (ICML, 2021)
  • Information Geometry of Orthogonal Initializations and Training (ICLR, 2020)
  • Fisher-rao metric, geometry, and complexity of neural networks (AISTATS, 2019)
  • Principles of Riemannian Geometry in Neural Networks (NeurIPS, 2017)
  • f-GANs in an Information Geometric Nutshell (NeurIPS, 2017)
  • Principal whitened gradient for information geometry (Neural Networks, 2008)
  • Geometrical Singularities in the Neuromanifold of Multilayer Perceptrons (NeurIPS, 2001)
  • Algebraic Information Geometry for Learning Machines with Singularities (NeurIPS, 2000)
  • Gradient systems in view of information geometry (Physica D: Nonlinear Phenomena, 1995)
  • Information geometry of the EM and em algorithms for neural networks (Neural Networks, 1995)
  • Information geometry of Boltzmann machines (IEEE Trans. Neural Networks, 1992)

Information Geometry for Clustering

  • A novel clustering algorithm based on information geometry for cooperative spectrum sensing (IEEE Systems Journal, 2020)
  • On Clustering Histograms with k-Means by Using Mixed α-Divergences (Entropy, 2014)
  • Barycentric distribution estimation for texture clustering based on information-geometry tools (ISETC, 2012)

Information Geometry for NMF

  • Non-negative low-rank approximations for multi-dimensional arrays on statistical manifold (Information Geometry, 2023)
  • Geometrical formulation of the nonnegative matrix factorization (ICONIP, 2018)
  • Generalized alpha-beta divergences and their application to robust nonnegative matrix factorization (Entropy, 2011)

Information Geometry for MCMC

Information Geometry for HMM

Information Geometry for Dimension Reduction

  • Generalized t-SNE Through the Lens of Information Geometry (IEEE Access, 2021)
  • Dimension Reduction for Mixtures of Exponential Families (ICANN, 2008)
  • The e-PCA and m-PCA: Dimension reduction of parameters by information geometry (IJCNN, 2004)

Information Geometry and Optimization

  • On a Cornerstone of Bare-Simulation Distance/Divergence Optimization (GSI, 2023)

Fisher Information Matrix

  • FIT: A Metric for Model Sensitivity (ICLR, 2023)
  • A Statistical Manifold Framework for Point Cloud Data (ICML, 2022)
  • On the Variance of the Fisher Information for Deep Learning (NeurIPS, 2021)
  • On the Fisher-Rao Information Metric in the Space of Normal Distributions (GSI, 2019)
  • The Spectrum of the Fisher Information Matrix of a Single-Hidden-Layer Neural Network (NeurIPS, 2018)
  • General Fisher information matrices of a random vector (Adv. Appl. Math., 2017)
  • Evaluating neuronal codes for inference using Fisher information (NeurIPS, 2010)
  • Invariant Fisher information (Differ Geom Appl., 1994)

Fisher Kernels

  • Learning Representation from Neural Fisher Kernel with Low-rank Approximation (ICLR, 2022)
  • Deep active learning for biased datasets via fisher kernel self-supervision (CVPR, 2020)
  • Persistence fisher kernel: A riemannian manifold kernel for persistence diagrams (NeurIPS, 2018)
  • The fisher kernel: a brief review (RN, 2011)
  • Improving the fisher kernel for large-scale image classification (ECCV, 2010)
  • Indefinite kernel fisher discriminant (ICPR, 2008)
  • A Kullback-Leibler divergence based kernel for SVM classification in multimedia applications (NeurIPS, 2003)
  • A novel graph-based fisher kernel method for semi-supervised learning (ICPR, 2014)
  • Learning the Similarity of Documents: An Information-Geometric Approach to Document Retrieval and Categorization (NeurIPS, 1999)

Information Diffusion Kernels

  • Learning parameterized histogram kernels on the simplex manifold for image and action classification (ICCV, 2011)
  • Sentiment classification with interpolated information diffusion kernels (ADKDD, 2007)
  • Diffusion kernels on statistical manifolds (JMLR, 2005)

Natural Gradients

  • Linear Convergence of Natural Policy Gradient Methods with Log-Linear Policies (ICLR, 2023)
  • Invariance properties of the natural gradient in overparametrised systems (Information Geometry, 2022)
  • Tractable structured natural-gradient descent using local parameterizations (ICML, 2021)
  • Marginalized Stochastic Natural Gradients for Black-Box Variational Inference (ICML, 2021)
  • Sinkhorn Natural Gradient for Generative Models (NeurIPS, 2020)
  • An Improved Analysis of (Variance-Reduced) Policy Gradient and Natural Policy Gradient Methods (NeurIPS, 2020)
  • Natural Policy Gradient Primal-Dual Method for Constrained Markov Decision Processes (NeurIPS, 2020)
  • Understanding Approximate Fisher Information for Fast Convergence of Natural Gradient Descent in Wide Neural Networks (NeurIPS, 2020)
  • Ngboost: Natural gradient boosting for probabilistic prediction (ICML, 2020)
  • A Formalization of the Natural Gradient Method for General Similarity Measures (GSI, 2019)
  • Fast Convergence of Natural Gradient Descent for Over-Parameterized Neural Networks (NeurIPS, 2019)
  • Limitations of the empirical Fisher approximation for natural gradient descent (NeurIPS, 2019)
  • Adaptive Stochastic Natural Gradient Method for One-Shot Neural Architecture Search (ICML, 2019)
  • Online natural gradient as a Kalman filter (Electron. J. Stat., 2018)
  • Fast yet simple natural-gradient descent for variational inference in complex models (ISITA, 2018)
  • Natural gradient via optimal transport (Information Geometry, 2018)
  • Fast Approximate Natural Gradient Descent in a Kronecker Factored Eigenbasis (NeurIPS, 2018)
  • Exact natural gradient in deep linear networks and its application to the nonlinear case (NeurIPS, 2018)
  • SLANG: Fast Structured Covariance Approximations for Bayesian Deep Learning with Natural Gradient (NeurIPS, 2018)
  • Comparison-based natural gradient optimization in high dimension (GECCO, 2014)
  • Projected Natural Actor-Critic (NeurIPS, 2013)
  • Stochastic Gradient Riemannian Langevin Dynamics on the Probability Simplex (NeurIPS, 2013)
  • New Sparse Adaptive Algorithms Based on the Natural Gradient and the $L_0$-Norm (IEEE J. Ocean. Eng., 2012)
  • Natural Policy Gradient Methods with Parameter-based Exploration for Control Tasks (NeurIPS, 2010)
  • A Generalized Natural Actor-Critic Algorithm (NeurIPS, 2009)
  • Stochastic search using the natural gradient (ICML, 2009)
  • Incremental Natural Actor-Critic Algorithms (NeurIPS, 2007)
  • Topmoumoute Online Natural Gradient Algorithm (NeurIPS, 2007)
  • Natural Actor-Critic for Road Traffic Optimisation (NeurIPS, 2006)
  • Rprop using the natural gradient (Trends and Applications in Constructive Approximation, 2005)
  • A Natural Policy Gradient (NeurIPS, 2001)
  • Natural gradient descent for on-line learning (PRL, 1998)
  • Natural gradient works efficiently in learning (Neural Computation, 1998)

Statistical Manifolds and Hessian Information Geometry

  • On the Tangent Bundles of Statistical Manifolds (GSI, 2023)
  • Geometric Properties of Beta Distributions (GSI, 2023)
  • KV Cohomology Group of Some KV Structures on ℝ^2 (GSI, 2023)
  • Alpha-parallel Priors on a One-Sided Truncated Exponential Family (GSI, 2023)
  • Conformal Submersion with Horizontal Distribution and Geodesics (GSI, 2023)

alpha-Divergence

Bregman Divergence

Jensen-Shannon Divergence

  • Quasi-arithmetic Centers, Quasi-arithmetic Mixtures, and the Jensen-Shannon ∇-Divergences (GSI, 2023)
  • On a variational definition for the Jensen-Shannon symmetrization of distances based on the information radius (Entropy, 2021)
  • $\alpha$-Geodesical Skew Divergence (Entropy, 2021)
  • On a generalization of the Jensen–Shannon divergence and the Jensen–Shannon centroid (Entropy, 2020)
  • On the Jensen–Shannon Symmetrization of Distances Relying on Abstract Means (Entropy, 2019)

Exponential and Mixture Families

  • On partial likelihood and the construction of factorisable transformations (Information Geometry, 2022)
  • On a convergence property of a geometrical algorithm for statistical manifolds (ICONIP, 2019)
  • Testing the Number and the Nature of the Components in a Mixture Distribution (GSI, 2019)
  • Sobolev Statistical Manifolds and Exponential Models (GSI, 2019)
  • Minimization of the Kullback-Leibler Divergence over a Log-Normal Exponential Arc (GSI, 2019)
  • Riemannian Distance and Diameter of the Space of Probability Measures and the Parametrix (GSI, 2019)
  • Information geometry of positive measures and positive-definite matrices: decomposable dually flat structure (Entropy, 2014)
  • Geometry of deformed exponential families: Invariant, dually-flat and conformal geometries (Physica A, 2012)
  • Geometry of q-Exponential Family of Probability Distributions (Entropy, 2011)

Inequalities

  • Curvature Inequalities and Simons’ Type Formulas in Statistical Geometry (GSI, 2021)
  • Inequalities for Statistical Submanifolds in Hessian Manifolds of Constant Hessian Curvature (GSI, 2019)
  • B. Y. Chen Inequalities for Statistical Submanifolds in Sasakian Statistical Manifolds (GSI, 2019)
  • Generalized Wintgen Inquality for Legendrian Submanifolds in Sasakian Statistical Manifolds (GSI, 2019)
  • Cramér-Rao lower bound and information geometry (Connected at Infinity II., 2013)
  • Inequalities for Tsallis relative entropy and generalized skew information (Linear Multilinear Algebra, 2011)

Transport Information Geometry

  • Riemannian Metric Learning via Optimal Transport (ICLR, 2023)
  • Wasserstein information matrix (Information Geometry, 2023)
  • Wasserstein Statistics in One-Dimensional Location-Scale Models (GSI, 2021)
  • Traditional and Accelerated Gradient Descent for Neural Architecture Search (GSI, 2021)
  • Recent Developments on the MTW Tensor (GSI, 2021)
  • Wasserstein Proximal of GANs (GSI, 2021)
  • Hessian Curvature and Optimal Transport (GSI, 2019)
  • Information geometry connecting Wasserstein distance and Kullback–Leibler divergence via the entropy-relaxed transportation problem (Information Geometry, 2018)

Computational Information Geometry

  • λ-Deformed Evidence Lower Bound (λ-ELBO) Using Rényi and Tsallis Divergence (GSI, 2023)
  • On the f-Divergences Between Hyperboloid and Poincaré Distributions (GSI, 2023)
  • Geometry of Parametric Binary Choice Models (GSI, 2023)
  • A q-Analogue of the Family of Poincaré Distributions on the Upper Half Plane (GSI, 2023)
  • Computing Statistical Divergences with Sigma Points (GSI, 2021)
  • Remarks on Laplacian of Graphical Models in Various Graphs (GSI, 2021)
  • Classification in the Siegel Space for Vectorial Autoregressive Data (GSI, 2021)
  • Information Metrics for Phylogenetic Trees via Distributions of Discrete and Continuous Characters (GSI, 2021)
  • Wald Space for Phylogenetic Trees (GSI, 2021)
  • A Necessary Condition for Semiparametric Efficiency of Experimental Designs (GSI, 2021)
  • Parametrisation Independence of the Natural Gradient in Overparametrised Systems (GSI, 2021)
  • Properties of Nonlinear Diffusion Equations on Networks and Their Geometric Aspects (GSI, 2021)
  • Rényi Relative Entropy from Homogeneous Kullback-Leibler Divergence Lagrangian (GSI, 2021)
  • Statistical Bundle of the Transport Model (GSI, 2021)
  • Topological Methods for Unsupervised Learning (GSI, 2019)
  • Geometry and Fixed-Rate Quantization in Riemannian Metric Spaces Induced by Separable Bregman Divergences (GSI, 2019)
  • The Statistical Minkowski Distances: Closed-Form Formula for Gaussian Mixture Models (GSI, 2019)
  • Parameter Estimation with Generalized Empirical Localization (GSI, 2019)
  • Properties of the Cross Entropy Between ARMA Processes (GSI, 2019)
  • Computational information geometry for binary classification of high-dimensional random tensors (Entropy, 2018)
  • Computational information geometry in statistics: theory and practice (Entropy, 2014)
  • Computational Information Geometry and its Applications (ICNNB, 2005)

Library

Lectures and Tutorials

PDF

  • An elementary introduction to information geometry (Entropy, 2022)
  • Divergence function, information monotonicity and information geometry (WITMSE, 2009)

YouTube

  • Information Geometry and Its Applications: Survey by Shun-Ichi Amari (YouTube)
  • Information Geometry by Microsoft Research (YouTube)
  • Nihat Ay : Information Geometric structures in Cognitive Systems Research (YouTube)
  • Computational Information Geometry with Frank Nielsen (YouTube)
  • Information Geometry and its Application by Melvin Leok (YouTube)

Journals

Conferences

  • International Conference on Information Geometry for Data Science (IG4DS)
  • International Conference on Geometric Science of Information (GSI)

Workshops