SaFARi: State-Space Models for Frame-Agnostic Representation