A/B Testing Notes
Interview Questions
Spreadsheets of Tracking & Calculating Statistical Significance
Statistical Significance Calculator
Google's Course
Notes for the A/B Testing course by Google
Overview
- When can you use A/B testing?
Metric Choice
- Customer Funnel
- Click-Through-Rate (CTR) vs Click-Through-Probability
- Invariant Checking
- Evaluation
- Gathering Additional Data
- Choosing a Technique
- Defining a Metric
- Filtering and Segmenting
- Summary Metrics
- Sensitivity vs Robustness
Statistics
- Binomial Distribution
- Confidence Interval
- Margin of Error
- Hypothesis Testing/Inference
- Pooled Standard Error
- Statistical Power
- Sensitivity
- Absolute vs Relative Changes
- Variability
- Non-parametric Methods
Design
- Sample Size Calculator
- Example
Analyzing Results
- Sanity Checks
- Single Metric Evaluation
- Simpson’s paradox
- Multiple Metrics
- Disappearing Launch Effect