A Fine-Grained Spectral Perspective on Neural Networks