Living in the wilderness: hypothesis testing in a world that disagrees with statistical theory

Sometimes it seems paradoxical to call the famous bell curve "normal". Among all the assumptions made by traditional statistical theory, the normality assumption is notorious for the frequency it doesn't hold. My aim in this article is to show a way to test hypotheses when the normality assumption of traditional hypothesis tests is violated. In this scenario, we can't rely on theoretical results, so we need to depart from theory's ivory tower and double the bet on our data. To get there, first I briefly review what hypothesis testing is, focusing on an intuitive grasp of the reasoning behind it (no equations allowed!). Then I proceed to a case study motivated by a business problem where the normality assumption doesn't hold. This makes matters concrete and will direct our discussion. After the problem is explained, I will show that bootstrapping is a good way to fill the gaps left by theory without changing anything in the reasoning at the heart of hypothesis testing. In particular, I will show that bootstrapping leads to the right conclusion about the test. I conclude this article with a critical evaluation of bootstrapping and similar methods, pointing out their pros and cons. Many data scientists have trouble understanding hypothesis testing.