Latent Action Learning Requires Supervision in the Presence of Distractors