Is Your Diffusion Model Actually Denoising?

We study the inductive biases of diffusion models with a conditioning-variable, which have seen widespread application as both text-conditioned generative image models and observation-conditioned continuous control policies. We observe that when these models are queried conditionally, their generations consistently deviate from the idealized denoising process upon which diffusion models are formulated, inducing disagreement between popular sampling algorithms (e.g.