ffiirree/dl_notes

Deep Learning Notes

TeX

Papers

Basic

Activation Function & Initialization

[2010] Xavier - Understanding the difficulty of training deep feedforward neural networks [AISTATS]
[2015] PReLU,Kaiming - Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification [ICCV]

PCA & Whitening & Smoothness of the Optimization Landscape

[UFLDL Tutorial] PCA Whitening
[1997] Edges are the 'independent components' of natural scenes
[2015] Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift [arXiv]
[2018] Smoothness of the Optimization Landscape - How Does Batch Normalization Help Optimization? [NeurIPS]
[2019] Rethinking the Usage of Batch Normalization and Dropout in the Training of Deep Neural Networks [arXiv]

Deep supervision

[2014] Deeply-Supervised Nets
[2015] Training Deeper Convolutional Networks with Deep Supervision

Data Augmentation

Optimization Algorithms

An overview of gradient descent optimization algorithms

Models

[1998] LeNet - GradientBased Learning Applied to Document Recognition
[2012] AlexNet - ImageNet Classification with Deep Convolutional Neural Networks
[2013] NIN, Global Average Pooling, 1 x 1 convolution - Network In Network [arXiv]
[2014] VGGNet - Very Deep Convolutional Networks for Large-Scale Image Recognition [arXiv]
[2021] RepVGG: Making VGG-style ConvNets Great Again [CVPR]

Inception

[2014] Inception V1, GooLeNet - Going deeper with convolutions [CVPR]
[2015] Inception V2, Inception V3 - Rethinking the Inception Architecture for Computer Vision [CVPR]
[2016] Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning [arXiv]

Skip connections

[2015] ResNet- Deep Residual Learning for Image Recognition [CVPR]
[2016] Identity Mappings in Deep Residual Networks. [CVPR]
[2017] The Shattered Gradients Problem: If resnets are the answer, then what is the question? [arXiv]
[2017] DenseNet - Densely Connected Convolutional Networks [CVPR]

Interpretability

Visualization

[2013] Visualizing and Understanding Convolutional Networks
[2013] Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
[2015] Understanding Neural Networks Through Deep Visualization
[2015] DeepDream - Inceptionism: Going Deeper into Neural Networks

Attention Mechanism

[2017] Transformer - Attention is all you need [NeurIPS]
[2018] Non-local Neural Networks [CVPR]
[2018] SENet - Squeeze-and-Excitation Networks [CVPR]
[2018] CBAM - Convolutional Block Attention Module [CVPR]
[2019] DANet - Dual Attention Network for Scene Segmentation [CVPR]

Object Detection & Semantic Segmentation

UNet family

Neural Style Transfer

Super Resolution

Pose Estimation

Deep Generative Models

Auto Regressive Models

[2016] PixelRNN - Pixel Recurrent Neural Networks [arXiv]
[2016] PixelCNN - Conditional Image Generation with PixelCNN Decoders [NeurIPS]

VAE

[2013] VAE - Auto-Encoding Variational Bayes
[2014] Stochastic Backpropagation and Approximate Inference in Deep Generative Models
[2015] CVAE - Learning Structured Output Representation using Deep Conditional Generative Models [NeurIPS]
[2016] Deep Unsupervised Clustering with Gaussian Mixture Variational Autoencoders [arXiv]
[2016] Tutorial on Variational Autoencoders
[2018] VI - Variational Inference: A Review for Statisticians

GAN