Stable Invariant Models via Koopman Spectra