efficient-deep-learning: A Jupyter Notebook repository from ilyassmoummad

Repository for the course "Efficient Deep Learning" at IMT Atlantique

Course organisation / Syllabus

Here is a detailed schedule, session by session:

Introduction / Refresher on Deep Learning
1. General Intro - Why do we need to optimize deep learning ? Introduction of the MicroNet Challenge.
2. Course - Deep Learning and Transfer Learning.
3. Practical session - introduction to PyTorch, transfer learning.
4. Short project - exploring hyper parameters on a fixed architecture
5. Presentation of the Challenge Objectives and Rules (Long Project)
Time for the project
1. Short evaluation on Deep Learning Essentials
2. Course - Layers of a CNN, their computational cost and their memory footprint
3. Short project - exploring hyper parameters on a fixed architecture
Quantification
1. Short evaluation on CNN layers
2. Students presentations of short project - exploring hyper parameters on a fixed architecture
3. Course - Quantifying Deep neural networks
4. Practical session - quantification on a small convolutional network
5. Long project 1 - Challenge
Pruning
1. Short evaluation on Quantification
2. Course - Pruning Deep neural networks
3. Practical session - pruning on a small convolutional network.
4. Long project 2 - Challenge
Data Augmentation, Regularization and Self-Supervised Learning
1. Short evaluation on Pruning
2. Students presentations on current work on Challenge
3. Course - Data Augmentation, Regularization and Self-supervised Learning
4. Practical Session - Data Augmentation
5. Long Project 3 - Challenge
Factorization
1. Short evaluation on Data Augmentation
2. Course - Factorizing Deep neural networks
3. Practical session - factorizing a small convolutional network
4. Long Project 4 - Challenge
Distillation & Embedded Software and Hardware for Deep Learning
1. Long evaluation on Factorization and previous courses
2. Course - Distillation of knowledge and features between neural networks
3. Course - Embedded Software and Hardware for Deep Learning
4. Long Project 5 - Challenge
Final Session
1. Short evaluation on distillation and embedded software and hardware for Deep Learning
2. Long Project 6 - Challenge
3. Students presentations - Challenge final results

What is expected for the Long Project

Short version : Exploration of the accuracy / complexity tradeoff

Long version : this course is mostly based on the long project, and you have a lot of freedom, which we expect you to use. The overarching goal is to explore the tradeoff between the performances of neural networks (= Accuracy on the test set) and complexity. Complexity can be either computational complexity (number of arithmetic operations), or memory complexity (memory size necessary to use the network).

We encourage students to get creative and test combinations of the various ideas that we present. Starting from the end of Session 1, you already have enough knowledge to explore the tradeoff between architecture, number of parameters, and accuracy. Then, we study new notions that open new avenues to explore this tradeoff : quantization, pruning, factorization, distillation. In session 7, you'll have a deeper insight on how to thing about specific software or hardware architecture in order to fully exploit all the optimizations that can be done.

Evaluation in this course

There are short written evaluations during the first 10 minutes of each session starting from session 2. Don't be late!

For the final session, we ask you to prepare a 15 minutes presentation, that will be followed by 5 Minutes of question.

What we expect for the presentations :

Explain your strategy to explore the complexity / accuracy tradeoff. We will judge whether you understood the goal, and whether the proposed strategy follows a rigourous approach.
The clarity of your exposition and quality of your support (slides)

Importantly : We will NOT judge you based on accuracy scores, as long as you have baselines (non optimized) that are not too far from the usual accuracy for the considered dataset.