kylejcaron/python-ab-testing-handbook

AB Testing in Industry

A lightweight introduction to A/B testing in industry using Python. View the in-progress handbook here

[DRAFT] Table of Contents

Part 1: Introduction to A/B Testing

1. Introduction to A/B Testing
2. Randomised Controlled Experiments
3. Experiment Duration

Part II: Frequentist A/B Testing

1. Introduction to the T-Test
2. Frequentist Experiment Design
3. Power Analyses
4. Interpreting Experiment Results

Part III: Common Pitfalls

1. Guardrail Metrics
2. Multiple Comparisons Problem
3. P-Hacking 101
4. Overlapping tests
5. Spillover Effects

Part IV: Bayesian A/B Testing

1. Introduction to the beta distribution
2. Bayesian Analysis
3. Power Analyses for Bayesians
4. The Bayesian T-Test?

Part V: Advanced Topics

1. T-Test for Everything
2. Potential Outcomes Model
3. Multi-armed Bandits
4. Variance Reduction
5. Heterogenous Treatment Effects
6. Delta Method
7. Switchback Experiments
8. Surrogate Indices
9. Adaptive Capping
10. Off Policy Learning

Other Cool Topics

• Peaking in Bayesian A/B testing
• Quantile Testing (w/ Poisson Bootstrap?)
• Experiment Duration as a function of Discount Rate

More Resources

• Causal Inference for the Brave and True
• Causal Inference the Mixtape
• Trustworthy Online Controlled Experiments