Unifying Simulation and Inference with Normalizing Flows