Does learning the right latent variables necessarily improve in-context learning?