In this section, you'll get to develop your skills regarding AB testing. Before diving in, take a minute to note some key points which you should keep in mind when completing the various labs and conducting your own hypothesis tests in practice.
You've seen that a lot goes into the proper design of statistical tests. You've learned about Goodheart's law as well as the multiple comparisons problem. Additionally, you've also seen that a p-value by itself is prone to misinterpretation if not presented with other relevant design parameters such as effect size, sample size, and
A well-formulated question is essential to a good statistical experiment. This includes careful thought of unintended consequences, as you saw in the discussion of Goodheart's law. Additionally, the question should also be specific and measurable.
It cannot be stressed enough how important the relationship between
On the other end of problem formulation is formatting the data to actually answer said question. You'll encounter this most explicitly in the final lab of this section. There, you'll have to transform your data into an appropriate format before conducting the statistical test. Furthermore, it's important to note how idiosyncrasies in your data can impact results. For example, monumental outliers can drastically impact the outcome of statistical tests. Whether or not to remove these data points can be a source of contention and will vary upon the circumstance. Similarly, it should go without saying that erroneous data or faulty data will clearly degrade statistical tests. All in all, it's always important to get familiar with the structure of the data and the context of the question being asked before diving into the statistics themselves.
Time to have at it! Dive in and start practicing some hypothesis testing!