Linear CNNs Discover the Statistical Structure of the Dataset Using Only the Most Dominant Frequencies

A general theory on how the implicit structure in the network arises and how it depends on the We here present a stepping stone towards a deeper structure of the dataset has yet to be developed. Here we understanding of convolutional neural networks derive such a theory for the specific case of two-layer, linear (CNNs) in the form of a theory of learning in CNNs, and we provide experiments that show how our insights linear CNNs. Through analyzing the gradient descent relate to the evolution of learning in deep, non-linear equations, we discover that the evolution CNNs. Our approach is inspired by previous work on the of the network during training is determined by learning dynamics in linear fully connected neural networks the interplay between the dataset structure and the (FCNN) (Saxe et al., 2014; 2019), but we uncover the role convolutional network structure. We show that linear of convolutions and show how they fundamentally alter the CNNs discover the statistical structure of the internal dynamics of learning. We start by discussing the dataset with non-linear, ordered, stage-like transitions, two involved structures in terms of singular value decompositions and that the speed of discovery changes (SVD): on the one hand the SVD of the input-output depending on the relationship between the dataset correlation matrix, representing the statistical dataset structure and the convolutional network structure.