The Limitations of Large Width in Neural Networks: A Deep Gaussian Process Perspective

Large width limits have been a recent focus of deep learning research: modulo computational practicalities, do wider networks outperform narrower ones?