Learning from Imbalanced Classes - Silicon Valley Data Science

If you're fresh from a machine learning course, chances are most of the datasets you used were fairly easy. Among other things, when you built classifiers, the example classes were balanced, meaning there were approximately the same number of examples of each class. Instructors usually employ cleaned up datasets so as to concentrate on teaching specific algorithms or techniques without getting distracted by other issues. Usually you're shown examples like the figure below in two dimensions, with points representing examples and different colors (or shapes) of the points representing the class: The goal of a classification algorithm is to attempt to learn a separator (classifier) that can distinguish the two. But when you start looking at real, uncleaned data one of the first things you notice is that it's a lot noisier and imbalanced.