Doubly robust nearest neighbors in factor models