What shortcomings do you see with deep learning?

Another problem that the other answerers haven't mentioned is interpretability. After you've trained a neural net, it is very difficult to understand what it's actually doing. You can visualize what units in the network respond to using techniques like this: Jason Yosinski, but fundamentally, it's much harder to interpret what a CNN is doing than a simple linear model in some hand-picked feature space. This can make it difficult to use CNNs in science. Even if you have a predictive model, in some situations it's not very useful if you don't have insight into what the parameters in the model mean.