cilvrRG/DeepLearn-Theory

Topics on theoretical, mathematical aspects of DL

DeepLearn-Theory

Reading Group on Topics on theoretical, mathematical aspects of DL. NYU, Fall 2016. Moderator: J. Bruna.

The purpose of this reading group is to define good open problems that relate Deep Learning models with aspects of statistics, applied maths and physics. We are particularly interested in connections with statistical physics, optimization and harmonic analysis. Everyone is welcome.

##Information Thursdays at 5:30pm, Center for Data Science, NYU. 60 5th ave, 6th floor, Room 606, 7th floor open-space.
##Logistics The goal is that each week a designated person(s) will present a selected paper, and possibly a bit of the mathematical context that is required to address it.

##Tentative List of Topics:

• Statistical Physics, Maximum Entropy
• Unsupervised Learning for Images and Time Series.
• Stochastic Optimization and Stability.
• Gradient Descent, bassins of Attraction and Tensor Analysis.
• Graph Theory, Invariance Groups and Convolutions.
• Bandits.

##Tentative Agenda:

• [9/22]: Optimization: Convex Review

  • Theory of Convex Optimization, by S. Bubeck.
  • Large Scale Machine Learning and convex optimization by Francis Bach.

• [9/29]: Stochastic Optimization, by Leon Bottou.

• [10/6]: Optimization: Non-Convex

  • Learn Faster, Generalize Better by Hardt, Recht and Singer.
  • Gradient Descent Conveges to Minimizers Lee et al.

• [10/13]: Optimization: Non-Convex

  • When are Nonconvex Problems Not Scary?, by Sun et al.
  • Beyond Convexity: Stochastic Quasi-Convex Optimization, Hazan, Levy, Shalev.

• [10/20]: Optimization: Neural Networks

  • On the Quality of the Initial Bassin in overspecified Neural Networks, Safran and Shamir.
  • Distribution Specific Hardness of Learning Neural Networks, Shamir.

• [10/27]: Optimization: Neural Networks

• [11/10] Maithra Raghu (Google Brain)

• [11/17] Optimization: Neural Networks [Levent]

  • Gradient Descent learns linear dynamical systems, Hardt, Ma, Recht.

• [11/24] SDP [Afonso?]

• [12/1] Semi-definite Programming II [Agustin]

• [11/3]: Statistical Physics Basics

  • The Spin Glass Model.

• [11/10]: Statistical Physics II

  • Large Deviation Principles, Micro-canonical ensembles, Entropy.

• [11/17]: The Renormalization Group

  • Renormalization Group: an introduction
  • K. Wilson Nobel Lecture
  • A mapping between RG and Deep Learning

• [11/24]: Microcanonical Mixtures and CNNs.

  • Max-Entropy Gaussaniazation by Multiscale Scattering (J.B, S.M).

• [12/1]: Random Graphs I.

  • The Stochastic Block Model
  • Community Dectection/Clustering.
  • Other Community models Modularity and Community Structure in Networks

• [12/8]: Algorithms for Random Graphs

  • Spectral Methods
  • Graph Neural Networks
  • Semidefinite Programming.

• [12/15]: Deep Learning on Graphs

  • Geometric Deep Learning, by Bronstein, Bruna, Szlam, Lecun, Vandergyst.

##Pool of Papers/Books [please fill]

• Les Houches Ellis Statistical Physics.
• Gibbs Models and Sampling.
• Renormalization Group (RG)
• Learn faster generalize better
• Draft Microcanonical Mixtures (JB)
• Bassins of Attraction Shamir
• Randomized PCA (Tygert et Al)