On the Power and Limitations of Random Features for Understanding Neural Networks

Considering the one-hidden-layer example above, this corresponds to learning linear predictors over a fixed representation (chosen obliviously and randomly at initialization).