You can probably use deep learning even if your data isn't that big

Over at Simply Stats Jeff Leek posted an article entitled "Don't use deep learning your data isn't that big" that I'll admit, rustled my jimmies a little bit. To be clear, I don't think deep learning is a universal panacea and I mostly agree with his central thesis (more on that later), but I think there are several things going on at once, and I'd like to explore a few of those further in this post. Jeff takes a look at the performance of two approaches to classify handwritten 0s vs. 1s from the well known MNIST data set. He compares the performance of a 5-layer neural net with hyperbolic tangent activations to the Leekasso, which just uses the 10 pixels with the smallest marginal p-values. He shows, perhaps surprisingly, that the Leekasso outperforms the neural net when you only have a dozen or so samples. Don't use deep learning if you have 100 samples because the model will overfit and you will get bad out of sample performance.