The course gives a solid introduction to Bayesian statistical inference, with special emphasis on models and methods in computational statistics and machine learning.
- We will get off to a shocking start by introducing a very different probability concept than the one you are probably used to: subjective probability.
- We will then move on to the mathematics of the prior-to-posterior updating in basic statistical models, such as the Bernoulli, normal and multinomial models.
- Bayesian prediction and decision making under uncertainty is carefully explained, and you will hopefully see why Bayesian methods are so useful in modern application where so much focuses on prediction and decision making.
- The really interesting stuff starts to happen when we study regression and classification. You will learn how prior distributions can be seen as regularization that allows us to use flexible models with many parameters without overfitting the data and improving predictive performance.
- A new world will open up when we learn how complex models can be analyzed with simulation methods like Markov Chain Monte Carlo (MCMC), Hamiltonian Monte Carlo (HMC) and approximate optimization methods like Variational Inference (VI).
- You will also get a taste for probabilistic programming languages for Bayesian learning, in particular the popular Stan language in R.
- Finally, we'll round off with an introduction to Bayesian model selection and Bayesian variable selection.
The course will use the following book as the main course literature:
- Gelman, Carlin, Stern, Dunson, Vehtari, Rubin (2014). Bayesian Data Analysis (BDA). Chapman & Hall/CRC: Boca Raton, Florida. 3rd edition. Here is the book webpage and PDF.
- Villani, M. (2020). Bayesian Learning (BL). First chapters of a book that I am writing is here. This is a very rough version with lots of typos, but may be useful as a complement to Gelman et al.
- Additional course material linked from this page, such as articles and tutorial.
The course schedule on TimeEdit is here: Schedule.
-
The three computer labs are central to the course. Expect to allocate substantial time for each lab. Many of the exam questions will be computer based, so working on the labs will also help you with the exam.
-
You are strongly encouraged to do the labs in R, but any programming language is ok to use.
-
The labs should be done in pairs of students.
-
Each lab report should be submitted as a PDF along with the .R file with code. Submission is done through Athena.
-
There is only two hours of supervised time allocated to each lab. The idea is that you:
- should start working on the lab before the computer session
- so that you are in a position to ask questions at the session
- and then finish up the report after the lab session.
-
The datasets used in the labs are included in the BayesLearnSU R package which is available on github. To install it, you first need to install the devtools package.
# To install and load the package with the data
install.packages("devtools") # only one time
library(devtools)
install_github("ooelrich/BayesLearnSU") # only one time
library(BayesLearnSU)
# For general information about the package
help("BayesLearnSU")
# For a list of all available data sets
help(, "BayesLearnSU")
# For information about specific data sets
help(name_of_dataset)
Mattias Villani, Lecturer
Professor, Natural Born Bayesian 😜
Department of Statistics, Stockholm University
Division of Statistics and Machine Learning, Linköping University
Oscar Oelrich, Math exercises and Computer labs
PhD Candidate, specializing in Bayesian Statistics
Department of Statistics, Stockholm University
Lecturer: Mattias Villani
Lecture 1 - Subjective probability. Bayesian updating. Bernoulli model. Gaussian model.
Reading: BDA Ch. 1, 2.1-2.4 | BL Ch. 1-2, 4 | Slides
Notebooks: Analyzing email spam data with a Bernoulli model |
Analyzing internet download speeds with a Gaussian model
Lecture 2 - Poisson model. Summarizing posteriors. Conjugate priors. Prior elicitation.
Reading: BDA Ch. 2.-2.9 | BL Ch. 2 and 4 | Slides
Notebooks: Analyzing number of eBay bidders with an iid Poisson model
Lecture 3 - Multi-parameter models. Marginalization. Monte Carlo simulation. Multinomial model.
Reading: BDA Ch. 3 | BL Ch. 3 | Slides
Notebooks: Analyzing mobile phone survey data with a multinomial model
Lecture 4 - Linear Regression. Prediction. Making Decisions
Reading: BDA Ch. 14, Ch. 9.1-9.2 | BL Ch. 5 and 6 | Slides
Code: Prediction with two-parameter Gaussian model
Lecture 5 - Classification. Large-sample properties and Posterior approximation.
Reading: BDA Ch. 16.1-16.3, 4.1-4.2 | Slides
Notebook: Normal approximation of posterior for logistic regression
Lecture 6 - Nonlinear regression and Classification. Regularization.
Reading: BDA Ch. 20.1-20.2 | Slides
Lecture 7 - Gibbs sampling and Data augmentation
Reading: BDA Ch. 10-11 | Slides
Code: Gibbs sampling for a bivariate normal | Gibbs sampling for a mixture of normals
Extras: Illustration of Gibbs sampling when parameters are strongly correlated.
Lecture 8 - MCMC and Metropolis-Hastings
Reading: BDA Ch. 11 | Slides
Code: Simulating Markov Chains
Lecture 9 - HMC and Variational inference
Reading: BDA Ch. 12.4 and Ch. 13.7 | Slides
Extras: HMC animations | HarlMCMC shake
Lecture 10 - Probabilistic programming for Bayesian learning
Reading: RStan vignette | Linear regression in RStan | Prediction in RStan | Slides
RStan Code: Bernoulli | Normal model | Logistic regression | Logistic regression with random effects | Poisson model | Three plants data
Turing code (Julia): Bernoulli model | Normal model
Lecture 11 - Bayesian model comparison
Reading: BDA Ch. 7 | BL Ch. 11 | Slides
Code: Comparing models for count data
Lecture 12 - Bayesian variable selection and model averaging
Reading: Article on Bayesian variable selection | Slides
Assistant: Oscar Oelrich
Exercise session 1: Problem set
Exercise session 2: Problem set
Assistant: Oscar Oelrich
Computer Lab 1 - Exploring posterior distributions in one-parameter models by simulation and direct numerical evaluation.
Lab: Lab 1
Submission: Athena.
Computer Lab 2 - Polynomial regression and classification with logistic regression.
Lab: Lab 2 | Temp in Linköping dataset | WomenWork dataset
Submission: Athena.
Computer Lab 3 - Gibbs sampling, Metropolis-Hastings and Stan.
Lab: Lab 3 | Rainfall dataset | eBay dataset | How to code up Random Walk Metropolis | campylobacter dataset
Submission: Athena.
The course examination consists of:
- Written lab reports (deadlines given in Athena)
- Take home exam (first exam on Jan 15)
Some old exams with solutions:
- Old exams at SU
- Old exams at LiU are some past exams with solutions from a similar course that I have given at Linköping University. Note however that although the course contents are very similar that course had only a 4 hour exam in a computer room without internet access. You have more time and the internet available (from which you cannot copy or ask questions though) so your exam will be somewhat different.
- The main page with links to downloads for the programming language R
- RStudio - a very nice developing environment for R.
- An introduction to R | John Cook's intro to R for programmers | R cheat sheet .
As a bonus, I will occasionally post notebooks from a new notebook system called Pluto in the Julia programming language. The advantage of Pluto is that it is reactive: changing one variable in the notebook immediately updates the parts of the notebook that depend on the variable. Here are some instructions for how you can get started with Pluto Notebooks: Pluto instructions
