Review for NeurIPS paper: Finite Versus Infinite Neural Networks: an Empirical Study

Correctness: Given the empirical nature of the paper, it's hard to directly evaluate its correctness. In terms of the empirical methodology, for the most part I think it was superb (as I discussed in the strengths'' section above). One thing that I'm curious about relating to the paper's results is the well-known fact that infinite width networks cannot learn any representations (i.e. the kernels don't depend on the data). On the other hand, a common hypothesis about why neural networks are so powerful is that they are really great at learning useful representations. Given that, it seems like there's some tension with the results of FCNs at finite width underperforming their kernel counterparts. I wonder if the problems studied were too simple to require the learning of useful representations (or the FCNs were too shallow to learn them, given they only had 3 layers).