A/B Testing with Holm's Procedure - by Angelina Yang

A few posts ago we talked about A/B testing with multiple metrics. We used the famous Bonferroni Correction to control our family-wise error rate when performing multiple tests. The Bonferroni method is widely used for its simplicity and broad applicability. We know its biggest drawback is that it is a conservative test. A finding that survives a Bonferroni adjustment is a credible trial outcome.