/ABTesting

Udacity's AB testing course final project

Primary LanguageJupyter Notebook

Overview of A/B testing

Steps to set up an A/B test

  1. Decide on change
  2. Chose a Metric
  3. review statistics (observational)
  4. Design the test
  5. Analyze results

Ethics of setting up an A/B test

Things to keep in mind:

  1. Risk to participants
  2. Benefit - what will the experiment yield
  3. Choice - for instance: can provide the users the choice to select privacy level
  4. Privacy - how is the user privcy protected in the data (make data not personally identifiable)

Steps to selecting metrics for an experiment

  1. Define problem and suitable metrics
  2. Build intuition - measure the change between control and expt & the variability of the metric
  3. Characterize

Steps to design an experiment

  1. Choose the subject
  2. Choose the "population"
  3. Calculate Size of both classes needed
  4. Calculate the time the experiment needs to run

Steps to Analyze results of an A/B test

  1. Sanity Check
  2. Choose single metric or multiple metrics
  3. Calculate change
  4. Check for statistical significance
  5. Sign test - Positive/Negative change
  6. Statistical significance of sign test