Instructor: Miodrag Bolic, University of Ottawa, Ottawa, Canada
www.site.uottawa.ca/~mbolic
Time and place: Winter 2022, Monday 8:30 - 11:30
Course code: ELG 5218 (EACJ 5600)
Calendar Style Description: Uncertainty, Uncertainty propagation, Bayesian Inference, Bayesian Filtering, Data fusion, Metrology, Measurement Science, Error Analysis, Measures of Agreement, Data Quality, Data quality index. Case studies will be drawn from various fields including biomedical instrumentation, sensors and signal processing.
Prerequisites: We expect participating students to bring basic knowledge and experience in
- Elementary Probability
- Elementary Statistics
- Signal processing
- Machine learning
Grading: For collecting the credits the student are expected to
- Assignments or projects (30% of the grade)
- Midterm (15% of the grade)
- Scribing (20% of the grade)
- Final exam (35% of the grade)
All lectures are given online using Zoom this year.
About the course Over the last several years, deep neural networks advanced many applications including vision, language understanding, speech understanding, robotics and so on. But a major challenge still remains and that is: how to model uncertainty. Good models of uncertainty are crucial whenever decisions need to be made or an algorithm needs to decide how and when to acquire new information. Uncertainty quantification is related to combining computational models, physical observations, and possibly expert judgment to make inferences about a physical system. Types of uncertainties include:
- Experimental uncertainty (measurement errors)
- Model uncertainty/discrepancy
- Input/parameter uncertainty
- Prediction uncertainty.
Why uncertainty:
- Uncertainty quantification is a fundamental component of model validation
- The objective is to replace the subjective notion of confidence with a mathematical rigorous measure
- Uncertainties relate to the physics of the problem of interest and not to the errors in the mathematical description/solution.
PART 1: IID models and Bayesian inference
- Lec 1: Introduction to modeling, MLE and MAP, Beta-binomial model
- Lec 2: Linear Gaussian model, Bayesian linear regression, Logistic regression
- Lec 3a: Inference: Sampling: Rejection sampling, Importance sampling , Markov chain, MCMC: Metropolis Hastings, Gibbs sampling, Langevin dynamics, Stochastic gradient descent
- Lec 4: Variational inference, Black box variational inference
- Lec 5a: Uncertainty propagation and sensitivity analysis
- Lec 5b: Bayesian models: Mixture models, Hierarchical models
- Lec 6a: Model checking, model selection, Bayesian averaging, Information criteria
- Lec 6b: Expending traditional models: Introducing errors in both x and y variables, Bayesian neural networks
PART 2: Generative models, time series, heterogeneous data and decision making
- Lec 8: Gaussian Processes Regression and Classification
- Lec 9: Generative models: probabilistic PCA, Variational autoencoders
- Lec 10a: Sequential latent models: HMM, Kalman and particle filters
- Lec 10b: Deep sequential models: RNNs, RNN+stte space models
- Lec 11a: Time series: ARIMA models
- Lec 11b: Change points detection models
- Lec 12: Scientific machine learning
- Lec 13: Sequential decision making
Lecture
Reading: Z. Ghahramani, “Probabilistic machine learning and artificial intelligence,” Nature, 2015.
Reading: Appendix D Probability by Kevin Murphy
A Comprehensive Tutorial to Learn Data Science with Julia from Scratch
Lecture
Prof. Rai's slides Lec1 9 t0 26
Lec 1 Notebook
Additional Material
Prof. Rai's slides Lec2
Lecture
Lec 2 Notebook
Prof. Rai's slides Lec5
Prof. Rai's slides Lec6
Mandatory Exercise problems
Rehearsal Exercises->Probability Theory Review, Bayesian Machine Learning, Continuous Data and the Gaussian Distribution, Regression from Exercise: Bayesian Machine Learning and Information Processing and Solutions
Additional Material
Reading:
Chapters 3.6 and 11 by Kevin Murphy
Prof. Rai's slides Lec3
Prof. Rai's slides Lec4
Code:
Continuous Data and the Gaussian Distribution
Bayesian linear regression
Bayesian logistic regression
Lecture
Prof. Rai's slides Lec15, Lec16, Lec17
Lec 3 Notebook \
Additional Material
Reading:
C. Andrieu, "Monte Carlo Methods for Absolute Beginners," book chapter in Advanced Lectures on Machine Learning. pp. 113-145, 2003.
Michael Betancourt, "A Conceptual Introduction to Hamiltonian Monte Carlo," arXiv:1701.02434v2 [stat.ME] 16 Jul 2018.
S. Callh, Bayesian inference with Stochastic Gradient Langevin Dynamics \
Videos:
Hamiltonian Monte Carlo, Applied Bayesian Statistics course, 2020.
Exercise problems
Additional Material
Prof. Rai's slides Lec13, Lec14
Variational Inference: Foundations and Modern Methods by David Blei, Rajesh Ranganath, Shakir Mohamed, slides 55-81
Tutorials by Chantriolnt-Andreas Kapourani
Code: Turing Variational Inference
Additional Material
Reading: Blei et al JASA | Tran's VI Notes
Other material: Natural gradient notes | autograd in python | ForwardDiff in Julia
Code: SVI in Pyro
Slides: Probabilistic Machine Learning with PyMC3: Statistical Modeling for Engineers by Dr. Thomas Wiecki
Reading: Probabilistic programming by Dan MacKinlay
Christopher Krapu, Mark Borsuk, "Probabilistic programming: A review for environmental modellers," Environmental Modelling & Software, Volume 114, 2019, Pages 40-48.
https://doi.org/10.1016/j.envsoft.2019.01.014.
Notebooks:
Lecture
Julia Code:
- Uncertainty Programming, Generalized Uncertainty Quantification by Chris Rackauckas
- Global Sensitivity Analysis by Chris Rackauckas
Python code:
- A practical introduction to sensitivity analysis by Leif Rune Hellevik\
- SALib - Sensitivity Analysis Library in Python
Additional reading:
- V. G. Eck, W. P. Donders, J. Sturdy, J. Feinberg, T. Delhaas, L. R. Hellevik, and W. Huberts. A guide to uncertainty quantification and sensitivity analysis for cardiovascular applications. Int J Numer Method Biomed Eng, 32(8):e02755, 2016.
Lecture
Hierarchical models slides by Taylor R. Brown
Mixture Models slides by Russ Salakhutdinov
Code: Turing Mixture Models
PyMC3: A Primer on Bayesian Methods for Multilevel Modeling\
Video: Introduction to Bayesian Multilevel models
Model checking by Taylor
[Evaluating, comparing and expanding models by Taylor]https://github.com/tbrown122387/stat_6440_slides/blob/master/7/7.pdf
Reading (ordered by priority): Bayesian Data Analysis - Chapters 6 and 7 | Bayesian Model Selection, Model Comparison, and Model Averaging article
Lecture
Slides: Probabilistic Modeling meets Deep Learning by Prof. Rai slides 1-9
Stochastic variational inference and Bayesian neural networks by Nadezhda Chirkova
Reading: An introduction to Bayesian neural networks
Hands-on Bayesian Neural Networks - a Tutorial for Deep Learning Users and video
Code: Turing Julia Bayesian Neural Networks
Lecture
Slides by Andreas Lindholm 1 and 2
Reading: Gaussian Processes for Machine Learning - Chapters 1, 2.1-2.5, 3.1-3.4, 3.7, 4.1-4.3.
Code: GP Stheno Python and Julia | GPy for Python | GP summer school - Labs in Python | GP in Turing Regression | GP in Turing Classification
Other material: Visualize GP kernels
More slides
Linear latent variable models:
Reading: Chapter 2 from A.M. O’Sullivan's thesis
Slides: Continuous Latent Variable Models by Russ Salakhutdinov
Code: MultivariateStats, Turing example of pPCA
Variational autoencoders:
Reading: An Introduction to Variational Autoencoders by D. P. Kingma please read chapters 1 and 2 , | From Autoencoder to Beta-VAE by Lilian Weng
Probabilistic Models with Deep Neural Networks - includes probabilistic PCA, VAE and their connection.
Slides: Variational autoencoders by Prof.Mitesh Khapra | Video: Variational autoencoders by Prof.Mitesh Khapra
Slides and videos: Latent Variable Models, L4 by Pieter Abbeel
Code: Autoencoders by Hugh Murrell
Slides: HMM by John Paisley
ELEC-E8105 - Non-linear Filtering and Parameter Estimation by Simo Sarkaa, Lec 1, 2, 3, and 6
Particle filters
Reading: Tutorial paper on latent-variable models for time-series data, Barber and Cemgil, IEEE Signal Processing Magazine, 2010.
Kinematic Models for Target Tracking | Markov Models From The Bottom Up, with Python
Elements of Sequential Monte Carlo
Code: ForneyLab, ForneyLab Documentation
LowLevelParticleFilters
Kalman and Bayesian Filters in Python\
Slides:
DEEP SEQUENTIAL LATENT VARIABLE MODELS by J. Marino slides 31-49
Lecture 6: Recurrent Neural Networks by Ming Li
Generative modelling with deep latent variable models by M. Fraccaro slides 51-91.
Resources: Deep Learning book, Chapter 10: Sequence Modeling: Recurrent and Recursive Nets, Video lecture
Deep Learning Time Series Forecasting
Deep Latent Variable Models for Sequential Data by M. Fraccaro
Time series models
Slides and video: Time Series ARIMA Models by A. Katchova
Course: Applied Time Series Analysis
Code: Orbit: A Python Package for Bayesian Forecasting | GluonTS - Probabilistic Time Series Modeling
ARIMA for Turing
Change-point detection
Slides: Introduction to changepoint analysis with R by R. Killick | Bayesian Time Series Forecasting with Change Point and Anomaly Detection
Video: Introduction to changepoint analysis by R. Killick
Code: Changepoints.jl | Turing code by H. Kjellerstrand Python: Bayesian Online Changepoint Detection \
Time series classification
Reading: The Great Time Series Classification Bake Off | Bayesian Anomaly Detection and Classi cation by E Roberts at al
Time series clustering
Streaming data: Learning under Concept Drift
Slides: Integrating Scientific Theory with Machine Learning
Code: SciMLTutorials.jl: Tutorials for Scientific Machine Learning and Differential Equations
Introduction to Scientific Machine Learning through Physics-Informed Neural Networks
Chris Rackauckas
Bayesian Estimation of Differential Equations
ADCME
Video: COVID-19 Epidemic Mitigation via Scientific Machine Learning SciML
Slides: Reinforcement Learning and Multi-arm Bandits by Dr. Ravindran | Lecture and slides from deepmind
Videos: Reinforcement learning course
Reading: Algorithms for Decision Making Chapters 15, 16, 17 | Paper on playing atari games
Code: Deep Q-learning demo | Julia notebook
Slides: Review
Reading: Learning with non-IID data +other ML assumptions and how to break them - Tegan Maharaj
- Bayesian Learning by deep.TEACHING project
- Bayesian Machine Learning and Information Processing (5SSD0) by Prof.dr.ir. Bert de Vries
- Advanced Bayesian Learning by Mattias Villani
- Probabilistic Machine Learning (Summer 2020)
- Murphy, Kevin P. 2021. Probabilistic Machine Learning: An Introduction, MIT press.
- Murphy, Kevin P. 2022, Probabilistic Machine Learning: Advanced Topics.
- Bayesian Reasoning and Machine Learning by David Barber.
- Bayesian Methods for Hackers by Cameron Davidson-Pilon.
- N. D. Goodman, J. B. Tenenbaum, and The ProbMods Contributors (2016). Probabilistic Models of Cognition (2nd ed.)
- Model-Based Machine Learning by John Winn
- Variational Methods for Machine Learning with Applications to Deep Networks by L.P. Cinelli et al., 2021.
- Physics-based Deep Learning by N. Thuerey et al., Dec 2021.
- Statistics with Julia: Fundamentals for Data Science, Machine Learning and Artificial Intelligence, by Y. Nazarathy et al., 2021.
- Bayesian Modeling and Computation in Python by Martin, Osvaldo et al.,2021.
- An Introduction to Neural Network Methods for Differential Equations by Neha Yadav, Anupam Yadav, 2015.
- Probabilistic Machine Learning for Civil Engineers by James-A. Goulet, 2020.