madhadron/machine-learning-curriculum

:computer: Make machines learn so that you don't have to program them; The ultimate list

Machine Learning Curriculum

Machine Learning is a branch of Artificial Intelligence dedicated at making machines learn from observational data without being explicitly programmed.

Machine learning and AI are not the same. Machine learning is an instrument in the AI symphony — a component of AI. So what is Machine Learning — or ML — exactly? It’s the ability for an algorithm to learn from prior data in order to produce a behavior. ML is teaching machines to make decisions in situations they have never seen.

Machine Learning in General

Study this section to understand fundamental concepts and develop intuitions before going any deeper.

A computer program is said to learn from experience E with respect to some class of tasks T and performance measure P if its performance at tasks in T, as measured by P, improves with experience E.

• Artificial Intelligence, Revealed a quick introduction by Yann LeCun, mostly about Machine Learning ideas
• How do I learn machine learning? - Quora
• Intro to Machine Learning - Udacity hands on scikit-learn (python) programming learning on core ML concepts
• Machine Learning: Supervised, Unsupervised & Reinforcement - Udacity the 2 instructors are hilarious
• Machine Learning Mastery very carefully laid out step-by-step guide to some particular algorithms, though contains inaccuracies about general principles such as the bias-variance tradeoff.
• Andrew Ng's Course on Coursera recommended for people who want to know the details of ML algorithms under the hood, understand enough maths to be dangerous and do coding assignments in Octave programming language (Note: Andrew said that you do not need to know Calculus beforehand but he talked about it quite often so you will regret if you don't know Calculus 😆)
• ML Recipes - YouTube Playlist a really nicely designed concrete actionable content for ML introduction
• Machine Learning is Fun Part 1 simple approach to machine learning for non-maths people
• Machine Learning with Python - YouTube Playlist
• Machine Learning Yearning by Andrew Ng
• Machine Learning Crash Course: Part 1
• https://www.kadenze.com/courses/machine-learning-for-musicians-and-artists-iv
• Rules of Machine Learning: Best Practices for ML Engineering

Reinforcement Learning

Building a machine that senses the environment and then chooses the best policy (action) to do at any given state to maximize its expected long-term scalar reward is the goal of reinforcement learning.

• Advanced Topics: RL 2015 (COMPM050/COMPGI13) by David Silver (the guy behind AlphaGo)
• An Introduction Book by Richard S. Sutton and Andrew G. Barto
• Deep Reinforcement Learning: Pong from Pixels
• Lecture 10: Reinforcement Learning - YouTube
• A Survey Paper
• Deep Reinforcement Learning: A Tutorial - OpenAI
• CS 294: Deep Reinforcement Learning, Spring 2017

Deep Learning

Deep learning is a branch of machine learning where deep artificial neural networks (DNN) — algorithms inspired by the way neurons work in the brain — find patterns in raw data by combining multiple layers of artificial neurons. As the layers increase, so does the neural network’s ability to learn increasingly abstract concepts.

The simplest kind of DNN is a Multilayer Perceptron (MLP).

• A friendly introduction to Deep Learning and Neural Networks
• A Neural Network Playground Tinker with a simple neural network designed to help you visualize the learning process
• Deep Learning Demystified - Youtube explain inspiration of deep learning from real neurons to artificial neural networks
• Learn TensorFlow and deep learning, without a Ph.D. This 3-hour course (video + slides) offers developers a quick introduction to deep-learning fundamentals, with some TensorFlow thrown into the bargain.
• A Guide to Deep Learning by YN^2 a curated maths guide to Deep Learning
• Practical Deep Learning For Coders Course at Fast.ai taught by Jeremy Howard (Kaggle's #1 competitor 2 years running, and founder of Enlitic)
• Deep learning - Udacity recommended for visual learner who knows some ML, this course provides high level ideas of deep learning, dense intuitive details put in a short amount of time, you will use TensorFlow inside the course
• Deep Learning Resources (Papers, Online Courses, Books) - deeplearning4j.org
• Introduction to Deep Neural Networks - deeplearning4j.org
• NVIDIA Deep Learning Institute because GPU are efficient at training Neural Networks, NVIDIA notices this market !
• Deep Learning Book recommended for math nerds who want to understand the theoretical side, the book is crafted by our deep learning wizards (Goodfellow, Bengio and Courville)
• Unsupervised Feature Learning and Deep Learning
• DeepMind Publications
• DeepLearning.TV - YouTube broad overview of deep learning, no implementation, just pure ideas
• CS224d: Deep Learning for Natural Language Processing
• Deep Learning Summer School, Montreal 2015
• UFLDL Deep Learning Tutorial
• Neural networks class - YouTube Playlist
• http://deeplearning.net/
• https://developer.nvidia.com/deep-learning
• http://neuralnetworksanddeeplearning.com/index.html a hands-on online book for deep learning maths intuition, I can say that after you finish this, you will be able to explain deep learning in a fine detail.
• https://github.com/lisa-lab/DeepLearningTutorials
• https://www.kadenze.com/courses/creative-applications-of-deep-learning-with-tensorflow-i You will implement a lot of things inside TensorFlow such as Autoencoders, Convolutional neural net, Feedforward neural nets, Generative models (Generative Adversarial Networks, Recurrent networks), visualizing the network, etc. You will have lots of assignments to finish. The course director (Parag) is also approachable and active.
• Deep Learning Lectures by Yann LeCun Why not learn Deep Learning from the guy who invented Convolutional nets?
• 6.S094: Deep Learning for Self-Driving Cars a course at MIT
• The Neural Network Zoo a bunch of neural network models that you should know about (I know about half of them so don't worry that you don't know many because most of them are not popular or useful in the present)
• 6.S191: Introduction to Deep Learning a course for 2017
• The GAN Zoo a list of GAN papers which have their own name

Convolutional Neural Networks

DNNs that work with grid data like sound waveforms, images and videos better than ordinary DNNs. They are based on the assumptions that nearby input units are more related than the distant units. They also utilize translation invariance. For example, given an image, it might be useful to detect the same kind of edges everywhere on the image. They are sometimes called convnets or CNNs.

• CS231n: Convolutional Neural Networks for Visual Recognition a course taught at Stanford university
• How Convolutional Neural Networks work - Youtube technical explanation including pooling operations, ReLU, fully connected layer, optimization using gradient descent
• Neural Network that Changes Everything - Computerphile
• A Beginner's Guide To Understanding Convolutional Neural Networks
• Deep Learning for Computer Vision (Andrej Karparthy, OpenAI) this is my most favorite video of convolutional net. Andrej explains convnet in detail answering all the curious questions that one might have. For example, most articles only talk about convolution in grayscale image, but he describe convolution in images with color channels as well. He also talks about the concerns and the assumptions that convnets make. This is a great lecture!
• Understanding Neural Networks Through Deep Visualization explains how to visualize a convnet using various techniques

Recurrent Neural Networks

DNNs that have states. They also understand sequences that vary in length. They are sometimes called RNNs.

• http://karpathy.github.io/2015/05/21/rnn-effectiveness/
• http://colah.github.io/posts/2015-08-Understanding-LSTMs/
• http://www.wildml.com/2015/09/recurrent-neural-networks-tutorial-part-1-introduction-to-rnns/

Glossary

• An overview of gradient descent optimization algorithms
• Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs) an activation function that is trying to be better than its predecessor (ReLU) in a lot of way

Open Source Trained Models

• Self-Driving Car by Udacity
• deepdream inceptionism - a deep model that takes an image and hallucinates animals/buildings from it
• Magenta: Music and Art Generation with Machine Intelligence
• SyntaxNet (Parsey McParseface)
• Neural Storyteller convert image caption into a romantic one
• https://github.com/facebookresearch/deepmask sharp object segmentation on image at pixel-level
• https://github.com/facebookresearch/multipathnet convnet for classifying DeepMask+SharpMask model above
• https://github.com/facebookresearch/fastText Library for fast text representation and classification; Tutorial
• https://github.com/tensorflow/models
• https://github.com/google/seq2seq A general-purpose encoder-decoder framework for Tensorflow (you need to provide it a sequence of vectors as input and also provide sequence of vectors as output to train this deep RNN)
• https://github.com/phillipi/pix2pix Image-to-image translation using conditional adversarial nets; TensorFlow port of pix2pix; Watch the presentation of this work: Learning to see without a teacher

Interesting Techniques & Applications

• http://deeplearninggallery.com/ Deep Learning Gallery - a curated list of awesome deep learning projects
• How do GANs intuitively work? this is my article explaining GANs, I try to be as intuitive as possible, GANs are so awesome that I can't just ignore and not talk about it. You can also watch the official tutorial by Ian Goodfellow.
• https://deepart.io/ transfer image style to other image
• http://www.somatic.io/
• WaveNet: A Generative Model for Raw Audio by DeepMind
• Jukedeck Musical AI
• StackGAN StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networks
• Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling 3D-GAN, generating 3D models using GAN
• BEGAN: Boundary Equilibrium Generative Adversarial Networks this type of GAN generates more realistic images than ordinary GAN

Nice Blogs & Vlogs to Follow

• http://colah.github.io/ this guy knows how to explain
• https://karpathy.github.io/ this guy has taught some courses on Deep Nets
• http://ruder.io Sebastian Ruder's Deep Learning and NLP blog
• http://www.wildml.com/ this is like a repository on ML
• https://adeshpande3.github.io/adeshpande3.github.io/
• http://culurciello.github.io/
• Sirajology's YouTube Playlists lots of dense short hilarious introduction to ML
• Two Minute Papers on Deep Learning Playlist
• http://www.leviathan.ai/
• Welch Labs
• Distill.pub A modern medium for presenting research in Machine Learning

Impactful People

• Geoffrey Hinton, he has been called the godfather of deep learning by introducing 2 revolutionizing techniques (ReLU and Dropout) with his students. These techniques solve the Vanishing Gradient and Generalization problem of deep neural networks. He also taught a Neural Networks course at Coursera.
• Yann LeCun, he invented CNNs (Convolutional neural networks), the kind of network that is really popular among computer vision developers today
• Yoshua Bengio another serious professor at Deep Learning, you can watch his TEDx talk here (2017)
• Juergen Schmidhuber invented LSTM (a particular type of RNN)
• Jeff Dean, a Google Brain engineer, watch his TEDx Talk
• Ian Goodfellow, he invented GANs (Generative Adversarial Networks), is an OpenAI engineer
• David Silver this is the guy behind AlphaGo and Artari reinforcement learning game agents at DeepMind
• Demis Hassabis CEO of DeepMind, has given a lot of talks about AlphaGo and Reinforcement Learning achievements they have
• Andrej Karparthy he teaches convnet classes, wrote ConvNetJS, and produces a lot of content for DL community, he also writes a blog (see Nice Blogs & Vlogs to Follow section)
• Pedro Domingos he wrote the book The Master Algorithm: How the Quest for the Ultimate Learning Machine Will Remake Our World, watch his TEDx talk here

Libraries, Frameworks and Services

Glancing at their GitHub statistics can give you an estimate for how active/popular each of them is.

• Python Deep Learning Frameworks Reviewed 2017 read this to decide which framework is appropriate for you
• scikit-learn (Python) general machine learning library, high level abstraction, geared towards beginners
• TensorFlow (Python); Learning TensorFlow; Installing on Windows; Fresh Install on Ubuntu 16.04; Serving; Awesome; computation graph framework built by Google, has nice visualization board, probably the most popular framework nowadays for doing Deep Learning
• Theano (Python) another popular deep learning framework
• Lasagne (Python) Lightweight library to build and train neural networks in Theano
• Keras: Deep Learning library for Theano and TensorFlow (Python)
• Caffe (Python) originally created to tackle computer vision problems
• Microsoft Cognitive Toolkit (CNTK) Microsoft's framework (previously known as Computational Network Toolkit)
• Torch (LuaJIT) the most popular scientific computing framework for LuaJIT
• PyTorch (Python) PyTorch is a deep learning framework that puts Python first.
• MXNet: A Scalable Deep Learning Framework supports multiple language interfaces
• MinPy (Python) NumPy interface with mixed backend execution (MXNet, autograd)
• Chainer (Python) A flexible framework of neural networks for deep learning
• Kur (YAML, Python) Descriptive Deep Learning, get started in minutes because you don't need to code!
• DeepLearning4j (Java) not so popular, preferable for you if you like Java
• Bonsai (Inkling) a simplification layer for machine learning
• bitfusion - Software to Manage Deep Learning & GPUs contains Amazon Machine Images for many Deep Learning libraries including TensorFlow
• FloydHub a Heroku for Deep Learning (You focus on the model, they'll deploy)

Cutting-Edge Research

Steal the most recent techniques introduced by smart computer scientists (could be you).

• https://research.facebook.com/ai/
• http://research.google.com/pubs/MachineIntelligence.html
• https://www.openai.com/
• https://www.openai.com/requests-for-research/
• State of the art performance on each ML task
• http://www.gitxiv.com/

Practitioner Community

• https://www.kaggle.com
• https://gym.openai.com
• https://universe.openai.com/
• /r/MachineLearning
• https://www.facebook.com/groups/DeepNetGroup/
• https://www.facebook.com/groups/1892696574296664/ a Facebook group talking about Self-Driving Cars but it's usually not specific to that

Thoughtful Insights for Future Research

• What Can't Deep Learning Do? a list of problems that deep learning faces
• Pedro Domingos: "The Master Algorithm" - Talks at Google
• The AI Revolution: The Road to Superintelligence
• https://ai100.stanford.edu/2016-report
• Why does Deep Learning work so well? - The Extraordinary Link Between Deep Neural Networks and the Nature of the Universe
• These are three of the biggest problems facing today's AI
• Four Questions For: Geoff Hinton Geoff Hinton is referred to as "godfather of neural networks"
• What product breakthroughs will recent advances in deep learning enable? - Quora

Uncategorized

• Artificial Intelligence: A Modern Approach (Online Book)
• The Principles of Modern Game AI
• Scipy Lecture Notes
• https://www.youtube.com/user/aicourses
• The Fundamentals of Neuroscience learn how our brain works so that you can discover new deep learning breakthrough

Other Big Lists

• http://aireads.top AI/ML Reads - a directory of artificial intelligence/machine learning resources
• https://github.com/ZuzooVn/machine-learning-for-software-engineers
• https://github.com/josephmisiti/awesome-machine-learning
• https://github.com/ujjwalkarn/Machine-Learning-Tutorials
• https://github.com/terryum/awesome-deep-learning-papers
• https://github.com/ChristosChristofidis/awesome-deep-learning
• https://github.com/Developer-Y/cs-video-courses#machine-learning
• Open Source Deep Learning Curriculum an open source deep learning curriculum which contains a lot of course recommendations for you to consider enrolling. (Many recommendations are the same as in this list)
• Deep Learning Resources by Jeremy D. Jackson
• https://github.com/songrotek/Deep-Learning-Papers-Reading-Roadmap
• http://aimedicines.com/2017/03/17/all-ai-resources-at-one-place/
• A Large set of Machine Learning Resources for Beginners to Mavens

I am confused, too many links, where do I start?

If you are a beginner and want to get started with my suggestions, please read this issue: off99555#4

Disclaimer

This is a really big list because I also point to other people's list to ensure that most of the resources are accessible from this page without you looking anywhere else.

Most of these resources are the ones I enjoy reading/watching. I wouldn't put something that I am not interested in here.

NOTE: There is no particular rank for each link. The order in which they appear does not convey any meaning and should not be treated differently.

How to contribute to this list

1. Fork this repository, then apply your change.
2. Make a pull request and tag me if you want.
3. That's it. If your edition is useful, I'll merge it.

Or you can just submit a new issue containing the resource you want me to include if you don't have time to send a pull request.

The resource you want to include should be free to study.

---