Consider mentioning that overlapping confidence intervals don't imply a lack of statistical significance
yanirs opened this issue · 1 comments
In the Confidence intervals when all outcomes are 0 or 1 section, you refer to A/B testing. While it's useful to have confidence intervals for the control and test groups, it'd be good to note that even if they overlap, it doesn't mean that the difference between the groups isn't statistically significant. This is explained in this post.
It'd be nice to expand the example to show the confidence intervals for the difference of means, which can be derived with bootstrapping. It may also be worth noting that the confidence interval for the ratio of means may be quite different from the absolute difference.
Closing this issue as I ended up summarising it in this post: https://yanirseroussi.com/2019/01/08/hackers-beware-bootstrap-sampling-may-be-harmful/