Make Your Neural Net Confuse Dogs with Pelicans

A few years ago, one of the first things I did when learning about neural networks is to train a simple image classifier. Neural nets can do a marvelous job at telling what's in an image. However, one thing I have not asked myself back then is: "What are these nets actually learning?" Let me explain what I mean by that with an example: How do we humans recognize that a dog is a dog? I'd say we look for distinctive features like pointy ears, snout, tail, four legs, and similar things.