In this lab, you'll go through the process of designing an experiment.
You will be able to:
- Design, structure, and run an A/B test
You've been tasked with designing an experiment to test whether a new email template will be more effective for your company's marketing team. The current template has a 5% response rate (with standard deviation .0475), which has outperformed numerous other templates in the past. The company is excited to test the new design that was developed internally but nervous about losing sales if it is not to work out. As a result, they are looking to determine how many individuals they will need to serve the new email template in order to detect a 1% performance increase (or decrease).
State your null hypothesis here (be sure to make it quantitative as before)
# H_0 = Your null hypothesis
State your alternative hypothesis here (be sure to make it quantitative as before)
# H_1 = Your alternative hypothesis
Now define what
Note: Be sure to calculate a normalized effect size using Cohen's d from the raw response rate difference.
# Calculate the required sample size
While you now know how many observations you need in order to run a t-test for the given formulation above, it is worth exploring what sample sizes would be required for alternative test formulations. For example, how much does the required sample size increase if you put the more stringent criteria of
#Your code; plot power curves for the various alpha and effect size combinations
Finally, now that you've explored some of the various sample sizes required for statistical tests of varying power, effect size and type I errors, propose an experimental design to pitch to your boss and some of the accompanying advantages or disadvantages with it.
In this lab, you practiced designing an initial experiment and then refined the parameters of the experiment based on an initial sample to determine feasibility.