Why My Model with 90% Accuracy Doesn't Work

When you're dealing with marketing problems like customer churn (when a customer stops using a company's product over a certain period of time) prediction, the raw dataset is often imbalanced, meaning that the classes are inherently not balanced. Basically, what this means is the percentage of your customers who churn might be a lot lower than those who don't. In this example, the binary classification problem might have an 80–20 split, with only 20% of customers discontinuing their engagement with the company and 80% continuing to make a purchase. The problem is, that 20% could be VERY important to the business's bottom line. Think about it -- a gifting company has 100,000 customers with an average value of $50 per person.