Sparse Spectral Training and Inference on Euclidean and Hyperbolic Neural Networks