Deep Learning Generalization, Over-Parameterization, Extrapolation, and Decision Boundaries

Abstract: Deep neural networks have achieved great success, most notably in learning to classify images. Yet, the phenomenon of learning images is not well understood, and generalization of deep networks is considered a mystery. Recent studies have explained the generalization of deep networks within the framework of interpolation. In this talk, we will see that the task of classifying images requires extrapolation capability, and interpolation by itself is not adequate to understand deep networks. We study image classification datasets in the pixel space, the internal representations of images learned throughout the layers of trained networks, and also in the low-dimensional feature space that one can derive using wavelets/shearlets.