Doubly Robust Inference in Causal Latent Factor Models