Recovering Latent Causal Factor for Generalization to Distributional Shifts