Things I've found useful for learning probabilistic programming.
Probabilistic Models of Cognition: a really nice book that teaches you how to think using a PPL called Church. Complete with lots of interactive code examples and exercises. Church sandbox and language reference.
Probabilistic Programming and Bayesian Methods for Hackers.
Stan: I found it pretty easy to get up and running using their Python interface. Their reference is quite extensive. Lots of examples. Write models in Stan which compile to C++.
Anglican: a subset of Clojure, from Oxford.
WebPPL: PPL embedded in JavaScript.
PyMC3: a Python module for Bayesian statistical modeling.
Probabilistic graphical models: a course by Daphne Koller on PGMs, with tons of videos, quizzes, etc.
Pystan: Bayesian Inference for Fun and Profit: a nice demo of Stan.
Comparison of distributions: an interactive comparison of probability distributions on Desmos.