Data Mining – The Big Picture

Elsewhere, I have suggested that there are three junctures at which any data mining project may go the most wrong: 1. problem definition, 2. data acquisition and 3. model validation (see the Data Mining and Predictive Analytics Web log). Data acquisition is a superset of statistical sampling, and the text by Lohr is highly recommended for this topic. Model validation is well explained in the literature: see, for instance, Weiss and Kulikowski. Problem definition involves understanding the business problem and mapping an appropriate technical solution to it. This may not be as simple as it sounds, and it is easy to be naïve about the best way to construct a technical solution which most naturally solves the given problem.