In this lab, you'll go through a sample process of designing an experiment.
You will be able to:
- Design, structure, and run an A/B test
You've been tasked with designing an experiment to test whether a new email template will be more effective for your company's marketing team. The current template has a 5% response rate, which has outperformed numerous other templates in the past. As a result of some very poor performance from some of these alternative templates, the company is both excited to test this new design that was developed internally, and nervous about losing sales, if it is not to work out. As a result, they are looking to determine how many individuals they will need to serve the new email template to in order to detect a 1% performance increase (or drop).
State your null hypothesis here (be sure to make it quantitative as before)
h_0 :
State your alternative hypothesis here (be sure to make it quantitative as before)
h_1 :
Now define what alpha and beta you believe might be appropriate for this scenario. To start, we may arbitrarily set alpha and beta to .01, indicating that we wish to minimally open ourselves up to type I and type II errors. Later, we will be able to adapt these, if sample sizes turn out to be impractically large.
alpha = .01
beta = .01
Calculating n requires us to know the variance. In this case, we will have a binomial variable (they either respond to the email or don't) and thus the variance, can be calculated with a standard formula:
...So, after an initial trial of 35 individuals, you have a total of 2 responses.
Is this sufficient evidence to refute the null hypothesis stated above?
#Your code for testing the null hypothesis here
If we relax alpha and beta to .05, each (opening ourselves up to a higher probability of making type I and type II errors), how much would our required sample size drop?
#Your code here
In this lab, you practiced designing an intial experiment and then refined the parameters of the experiment based on an initial sample to determine feasability.