Tyler LaBonte on LinkedIn: Machine Learning Mondays #38: Every Monday I highlight an awesome

Machine Learning Mondays #38: Every Monday I highlight an awesome ML paper that gets me pumped that I'm in this field! A important theoretical questions in deep learning is how deep models generalize, since they do not obey convergence laws of classical models; in particular, neural networks achieve better generalization as they become infinitely large (called overparametrization). Recent work in this area has suggested approaches that all depend on the uniform convergence of the model, an important property which gives bounds on convergence. In their NeurIPS 2019 Best New Directions paper, CMU researchers debate the validity of this approach and show that uniform convergence-based bounds cannot possibly "explain generalization". They first show that, in practice, these bounds increase with the dataset size, which is contrary to the theoretical prediction. Then, they provide several examples of overparametrized networks which have provably vacuous uniform convergence bounds.