Using Imperfect Synthetic Data in Downstream Inference Tasks