This is the Deep Learning part of the Unsupervised Learning Course of the Master in High-Performance Computing (SISSA/ICTP)
First Part
- Artificial neural networks
- Train, validate and test a deep learning model
- Convolutional neural networks
- Elementary aspects of unsupervised deep learning models
Second Part
- Introduction to the basics of representation learning
- Autoencoders
- Variational autoencoders
- Alessio Ansuini (First Part)
- Alberto Cazzaniga (Second Part)
Institute: Research and Technology Institute, AREA Science Park)
Notebooks of the First Part are gently made available by Marco Zullich (PhD student in Industrial and Information Engineering at University of Trieste)
Day 1
- The artificial neuron
- Possiblities and limitations of a single neuron
- Linear layer
- Non-linearities
- Fully connected architectures
- Softmax layer
- Cross-entropy loss and the MLE principle
Sources (see below): Michael Nielsen's online book, PyTorch Tutorials
Day 2
- Stochastic gradient descent
- Optimization
- Regularization
- Data augmentation
Sources: Michael Nielsen's online book, PyTorch Tutorials
Day 3
- Convolutional networks basics
- Transfer learning
Michael Nielsen's online book, image kernels, PyTorch Tutorials
Day 4
- What is a "good representation"?
- From Principal Component Analysis to Autoencoders
- Autoencoders: construction, properties and drawbacks
Day 5
- The objective of VAEs
- Construction and loss function
- Reparameterization trick and learning algorithm
Two different and somewhat complementary references are [Doersch] and [Kingma, Welling].
You might want to start to take a look at [Ding, et al.].
There are excellent free resources to deepen your knowledge on topics such as Deep Learning, Reinforcement Learning and more in general Artificial Intelligence.
Here is a selection of very good ones.
Books for free
-
Michael Nielsen
-
The Deep Learning Book
https://www.deeplearningbook.org/
in a single pdf version
-
Information Theory, Pattern Recognition, and Neural Networks (Dave McKay)
-
Probabilistic Machine Learning (Kevin Murphy)
-
Dive into Deep Learning (Amazon group)
Courses for free
-
Fast AI (Jeremy Howards)
Invaluable resource for quickly getting your hands dirt into practical Deep Learning
-
Deep learning specialization (auditing is for free) on Coursera (Andrew Ng).
One of the best resources to learn basic and intermediate concepts.
(Check the Coursera website for other resources: auditing is sometimes for free, certificates are generally not.)
-
Deep unsupervised learning (Pieter Abbeel)
A glimpse into state-of-the-art research problems.
-
Deep reinforcement learning (Dave Silver)
The legendary course of Dave Silver on YouTube
https://www.youtube.com/watch?v=2pWv7GOvuf0&list=PLzuuYNsE1EZAXYR4FJ75jcJseBmo4KQ9-
-
Neural network class (Hugo Larochelle)
After almost 10 years still a very useful resource: crystal clear explanations of an impressive amount of topics, starting from the very basics (I used this a lot!)
https://www.youtube.com/watch?v=SGZ6BttHMPw&list=PL6Xpj9I5qXYEcOhn7TqghAJ6NAPrNmUBH
-
Information Theory, Pattern Recognition, and Neural Networks (Dave McKay) (See also the accompanying book)
The Lectures of Dave MacKay will accompany you to the study of its beautiful book: on of the most precious resources you will find on this topic.
Information theory is very relevant in many fields, and particularly in Unsupervised Deep Learning.
https://www.youtube.com/watch?v=BCiZc0n6COY&list=PLruBu5BI5n4aFpG32iMbdWoRVAA-Vcso6
Websites and Blogs
-
Deepmind
-
OpenAI
YouTube channels
-
Yannick Kilcher's channel
-
Two minutes papers