Lecture Slides for Deeplearning book
This repo contains lecture slides for Deeplearning book. This project is maintained by InfoLab @ DGIST (Large-scale Deep Learning Team), and have been made for InfoSeminar. It is freely available only if the source is marked.
The slides contain additional materials which have not detailed in the book.
Also, some materials in the book have been omitted.
Maintained by
InfoLab @ DGIST(Daegu Gyeongbuk Institute of Science & Technology)
Coverage
This repo covers Chapter 5 to 20 in the book.
Credits
Name | Chapters |
---|---|
Jinwook Kim | 5-1, 8, 16, 17 |
Heechul Lim | 9-2, 11, 12, 15, 19 |
Hyun-Lim Yang | 5-2, 7-2, 10-2 |
Keonwoo Noh | 7-1, 9-1, 14, 20 |
Eunjeong Yi | 6, 10-1, 13, 18 |
The contents of lecture slides
Part 1. Applied Math and Machine Learning Basics
5-1. Machine Learning Basics
- Learning algorithms
- Capacity, overfitting and underfitting
- Hyperparameters and validation sets
- Estimators, bias and variance
- Maximum likelihood estimation
5-2. Machine Learning Basics
- Bayesian statistics
- Supervised learning algorithms
- Unsupervised learning algorithms
- Stochastic gradient descent
Part 2. Deep Networks: Modern Practices
6. Deep Feedforward Networks
- Example: Learning XOR
- Gradient-Based Learning
- Hidden Units
- Architecture Design
- Back-Propagation and Other Differentiation
7-1 Regularization for Deep Learning
- Parameter Norm Penalties
- Norm Penalties as Constrained Optimization
- Regularization and Under-Constrained Problems
- Dataset Augmentation
- Noise Robustness
- Semi-Supervised Learning
- Multitask Learning
- Early stopping
7-2 Regularization for Deep Learning
- Parameter Tying and Parameter Sharing
- Bagging and Other Ensemble Methods
- Dropout
- Adversarial Training
8 Optimization for Training Deep Models
- How Learning Differs from Pure Optimization
- Challenges in Neural Networks
- Basic Algorithms
- Algorithms with Adaptive Learning Rates
- Parameter Initialization Strategies
- Approximate Second-order Methods
- Optimization Strategies and Meta-algorithms
9-1 Convolutional Networks
- The Convolution Operation
- Motivation
- Pooling
- Convolution and Pooling as an Infinitely Strong Prior
- Variants of the Basic Convolution Function
9-2 Convolutional Networks
- Structured Outputs
- Data Types
- Efficient Convolution Algorithms
- Random or Unsupervised Features
- The Neuroscientific Basis for Convolutional Networks
10-1 Sequence modeling: Recurrent and Recursive Nets
- Unfolding Computational Graphs
- Recurrent Neural Networks
- Bidirectional RNNs
- Encoder-Decoder Sequence-to-Sequence Architectures
- Deep Recurrent Networks
- Recursive Neural Networks
10-2 Sequence modeling: Recurrent and Recursive Nets
- The challenge of Long-term
- Echo State Networks
- Leaky Units and Other strategies for Multiple Time Scales
- The Long Short-Term Memory and Other Gated RNNs
- Optimization for Long-Term Dependencies
- Explicit Memory
11, 12 Practical Methodology and Applications
- Performance Metrics
- Default Baseline Models
- Determining Whether to Gather More Data
- Selecting Hyperparameters
- Debugging Strategies
- Computer Vision
Part 3. Deep Learning Research
13 Linear Factor Models
- Probabilistic PCA and Factor Analysis
- Independent Component Analysis
- Sparse Coding
14 Autoencoders
- Introduction
- Stochastic Encoders and Decoders
- Regularized autoencoders
- Representational Power, Layer Size and Depth
15 Representation Learning
- Unsupervised pre-training
- Introduction of supervised(SL) and unsupervised learning(UL)
- Representation
- Clustering
- K-means
- Gaussian Mixture Model
- EM algorithm
- Practical example
16, 17 Structured Probabilistic Models for Deep Learning and Monte Carlo Methods
- The Challenge of Unstructured Modeling
- Using Graphs to Describe Model Structure
- Sampling from Graphical Models
- The Deep Learning Approach to Structured Probabilistic Models
- Sampling and Monte Carlo Methods
- Markov Chain Monte Carlo Methods
- Gibbs Sampling
18 Confronting the Partition Function
- The Log-Likelihood Gradient
- Stochastic Maximum Likelihood and Contrastive Divergence
- Estimating the Partition Function
19 Approximate Inference
- Approximation
- Maximum Likelihood(MLE) and Maximum A Posteriori(MAP)
- Inference
- Taxonomy of deep generative models
- KL-Divergence
- Variational Inference
20 Deep Generative Models
- Generative models
- Boltzmann Machines
- Restricted Boltzmann Machines
- Deep Belief Networks
License: CC-BY