You shouldn't judge a book by its cover, but a neural network can