Latent Graph Inference with Limited Supervision