A/B Testing

Notes and resources on A/B testing and online experimentation.

General reading

The Surprising Power of Online Experiments, Kohavi and Thomke
https://hbr.org/2017/09/the-surprising-power-of-online-experiments
Good background info about the overall goals and advantages of successful A/B testing.

Controlled experiments on the web, Kohavi et al
This paper outlines the key concepts in online experimentation. Includes examples, key definitions and formulas, as well as pros and cons of different approaches.

A/B Testing at Stack Overflow, Julia Silge, Stack Overlflow:
https://stackoverflow.blog/2017/10/17/power-calculations-p-values-ab-testing-stack-overflow/
Shiny app: https://juliasilge.shinyapps.io/power-app/
This blog post offers a clear high-level overview of Stack Overflow's A/B testing procedure; includes a Shiny app that illustrates power calculations.

A more customizable sample size calculator, created by Evan Miller, is available here:
http://www.evanmiller.org/ab-testing/sample-size.html

Udacity A/B testing notes:
https://towardsdatascience.com/a-summary-of-udacity-a-b-testing-course-9ecc32dedbb1

http://thestatsgeek.com/2013/07/22/ab-testing/
Basic introduction about the chi-squared test and how it's used for A/B testing.

Effective A/B testing, Ben Tilly
http://elem.com/~btilly/effective-ab-testing/
This slide has a complete walkthrough of the A/B testing process using the G-test (a more modern version of the chi square test).

http://www.evanmiller.org/how-not-to-run-an-ab-test.html http://www.evanmiller.org/ab-testing/sample-size.html

Bayesian A/B testing

The Power of Bayesian A/B Testing, Michael Frasco, Convoy
https://medium.com/convoy-tech/the-power-of-bayesian-a-b-testing-f859d2219d5
Offers a good overview of the advantages of adopting a Bayesian methodology for experimentation.

Bayesian A/B Testing at Visual Website Optimizer, Chris Stucchio
https://cdn2.hubspot.net/hubfs/310840/VWO_SmartStats_technical_whitepaper.pdf
This whitepaper fully breaks down VWO's methodology for Bayesian A/B testing. Breaks down the same methodology in a presentation, slides available here:
https://www.chrisstucchio.com/pubs/slides/gilt_bayesian_ab_2015/slides.html#1

cgeng18/ab-testing

A/B Testing

General reading

Bayesian A/B testing

Further reading