The Surprising Effectiveness of Equivariant Models in Domains with Latent Symmetry