What is a key concept to understanding Machine Learning many people get wrong?

The key point is that we are not interested in all possible ML problems, most of them don't make any sense and will never occur with real data. Imagine I have a training and test set for a binary classifier, you can train a good algorithm and get 95% accuracy for the test set, now you flip the labels in the test set and the same algorithm that was excellent is a disaster. But there's a trick: the new problem doesn't make any sense. We are expecting the algorithm to learn how data works and then predict in the opposite direction. We don't have this problems with real data, they just don't make sense.