Materials for the NextAI Technical Stream Course on Machine Learning.
You are encouraged to review the following topics in advance of the Machine Learning course. It's common to be rusty on these subjects, especially if you have been out of school for a while.
I've provided suggested readings from the new textbook Mathematics for Machine Learning (MML) by Marc Peter Deisenroth, A. Aldo Faisal, and Cheng Soon Ong. The pdf version of this textbook is freely available online.
I've also provided supplementary Jupyter Notebooks (via Google Colaboratory) that contain Python-based examples and exercises related to the review topics. More information on running the notebooks is provided below.
| Topic | Readings from MML | Jupyter Notebooks |
|---|---|---|
| Introduction | - Foreword - 1 Introduction & Motivation - 1.1 Finding Words for Intuitions - 1.2 Two Ways to Read This Book - 1.3 Exercises and Feedback |
1. Scientific Python |
| Linear Algebra | - 2 Linear Algebra - 2.1 Systems of Linear Equations - 2.2 Matrices - 2.3 Solving Systems of Linear Equations - 2.4 Vector Spaces - 2.5 Linear Independence - 2.6 Basis and Rank - 2.7 Linear Mappings - 2.8 Affine Spaces |
- 2a. Vectors and Matrices - 2b. Solving Systems of Linear Equations - 2c. Linear Independence |
| Analytic Geometry | - 3 Analytic Geometry - 3.1 Norms - 3.2 Inner Products - 3.3 Lengths and Distances - 3.4 Angles and Orthogonality - 3.5 Orthonormal Basis - 3.6 Orthogonal Complement (Optional) - 3.7 Inner Product of Functions (Optional) - 3.8 Orthogonal Projections - 3.9 Rotations |
- 3a. Analytic Geometry - 3b. Orthogonality |
| Matrix Decompositions | - 4 Matrix Decompositions - 4.1 Determinant and Trace - 4.2 Eigenvalues and Eigenvectors - 4.3 Cholesky Decomposition - 4.4 Eigendecomposition and Diagonalization - 4.5 Singular Value Decomposition - 4.6 Matrix Approximation - 4.7 Matrix Phylogeny |
- 4a. Determinant and Trace - 4b. Eigenvalues and Eigenvectors - 4c. Cholesky Decomposition - 4d. Eigendecomposition and Diagonalization - 4e. Singular Value Decomposition - 4f. Matrix Approximation |
| Vector Calculus | - 5 Vector Calculus - 5.1 Differentiation of Univariate Functions - 5.2 Partial Differentiation and Gradients - 5.3 Gradients of Vector-Valued Functions - 5.4 Gradients of Matrices - 5.5 Useful Identities for Computing Gradients - 5.6 Backpropagation and Automatic Differentiation - 5.7 Higher-Order Derivatives (Optional) - 5.8 Linearization and Multivariate Taylor Series (Optional) |
- 5a. Derivatives, Partial Derivatives, and Gradients - 5b. Automatic Differentiation |
| Probability and Distributions | - 6 Probability and Distributions - 6.1 Construction of a Probability Space - 6.2 Discrete and Continuous Probabilities - 6.3 Sum Rule, Product Rule, and Bayes’ Theorem - 6.4 Summary Statistics and Independence - 6.5 Gaussian Distribution - 6.6 Conjugacy and the Exponential Family (Optional) - 6.7 Change of Variables/Inverse Transform (Optional) |
- 6a. Discrete and Continuous Probabilities - 6b. Summary Statistics and Independence - 6c. The Gaussian and its Longer-tailed Cousins |
| Continuous Optimization | - 7 Continuous Optimization - 7.1 Optimization using Gradient Descent - 7.2 Constrained Optimization and Lagrange Multipliers - 7.3 Convex Optimization |
- 7a. Unconstrained Optimization - 7b. Constrained Optimization - 7c. Black-box Function Optimization |
Once the course begins, this site will be updated with supporting material.
Python is established as a leading choice in scientific computing, and in particular, in machine learning. Compared to its competitors (e.g. Matlab), it’s open-source and free! Thanks to a stable numerical base called Numpy which can take advantage of low-level accelerated libraries like the Intel Math Kernel Library (Intel MKL), one can get C-like speeds but with far less programming overhead than compiled languages.
We’ll be using Python for examples in this course.
The so-called "scientific Python" suite consists of a number of tools, including but not limited to:
- NumPy: fundamental package for scientific computing, array data structures and operations.
- SciPy: various indispensable tools for math, science, and engineering (depends on NumPy).
- Matplotlib: easy plotting and visualization, MatLab-style.
- IPython: MatLab-like interactive shell.
- Jupyter: an open-source web application that allows you to create and share documents that contain live code, equations, visualizations, and narrative text.
Pre-requisite material and in-course examples will be delivered as Jupyter notebooks. I have make these available through Colaboratory: a free Jupyter notebook environment that requires no setup and runs entirely in the cloud.
I recommend that you visit the Welcome to Colaboratory! notebook to get a feel for how this works.
To run a notebook in Collaboratory, all you need to do is click on one of the links below. By default, the notebooks will open with read-only privileges, so you will need to make a copy in order to save your changes.
Though it is not necessary for this course to install Python on your personal computer, if you wish to do so, I recommend using an open-source distribution that comes with all the packages described above. Miniforge is a community-driven minimal installer for conda environments. It provides the conda environment and package manager without including the large number of packages that come with alternatives like Anaconda. This makes it a lighter and more flexible option for many users. Miniforge is cross-platform and completely free for basic use.
If you choose to download the notebooks and run them on your computer instead of running them in Collaboratory, you can use the environment file provided to simplify setup. The recommended setup procedure is as follows:
- Download and install Miniforge. Ensure you select the correct installer for your operating system.
- Familiarize yourself with Managing environments using conda.
- Examine the YAML environment file
environment.ymlwhich contains dependencies for the course (examine it in a text editor so you understand what's included) - Create a new environment based on the YAML file:
conda env create -f environment.yml - Activate your new Python environment:
conda activate nextai - Start JupyterLab from this directory:
jupyter lab