Regularized linear autoencoders recover the principal components, eventually

While there has been rapid progress in understanding the learning dynamics of neural networks, most such work focuses on the networks' ability to fit input-output relationships. However, many machine learning problems require learning representations with general utility.