Introduction to the POMDPs.jl
framework and its ecosystem.
The course covers how to build and solve decision making problems in uncertain environments using the POMDPs.jl ecosystem of Julia packages. Topics covered include sequential decision making frameworks—namely, Markov decision processes (MDPs) and partially observable Markov decision processes (POMDPs)—running simulations, online and offline solution methods (value iteration, Q-learning, SARSA, and Monte Carlo tree search), reinforcement learning, deep reinforcement learning (including proximal policy optimization (PPO), deep Q-networks (DQN), and actor-critic methods), imitation learning through behavior cloning of expert demonstrations, state estimation through particle filtering, belief updating, alpha vectors, approximate methods (including grid interpolation for local approximation value iteration), and black-box stress testing to validate autonomous systems. The course is intended for a wide audience—no prior MDP/POMDP knowledge is expected.
The lectures can be found on Julia Academy and YouTube. They are broken down as follows.
Brief introduction to the content of this course.
Introduction to MDPs using the Grid World problem.
Introduction to POMDPs using the Crying Baby problem.
Using beliefs to estimate the state of an agent through particle filtering.
Approximating a continuous space using grid interpolation and value function approximation.
Introduction to deep reinforcement learning applied to the pendulum swing-up MDP.
Introduction to imitation learning using behavior cloning of expert demonstrations.
Stress testing a black-box system using adaptive stress testing.
Created and taught by Robert Moss.