Diffusion with Forward Models: Solving Stochastic Inverse Problems Without Direct Supervision