Adam Kelleher on experiment design and observational analysis – I love experiments

Causality is a basic requirement any time you're trying to make a data-driven decision about a change to system. Often, people try to use observational data to speculate about changes. You can do this if you're very careful: there are ways to control for variables that cause bias, but none of them are perfect. Check out my 2nd blog post if you're interested in the details of that. The result is that if you want to use observational data to speculate about what you should do, then you're really leaving the result up to chance. You might get lucky, and there's no bias, and so your correlative result is causal. The problem is that you just don't know. To make it concrete, I'll modify an example from this paper.