How to mess up testing your AI system

A great way to keep your wits about you when working with machine learning (ML) and artificial intelligence (AI) is to think like a teacher. After all, the point of ML/AI is that you're getting your (machine) student to learn a task by giving examples instead of explicit instructions. As any teacher will remind you: if you want to teach with examples, the examples must be good. The more complicated the task, the more examples you'll need. If you want to be able to trust that your student has learned the task, the test must be good.