Score Approximation, Estimation and Distribution Recovery of Diffusion Models on Low-Dimensional Data

Diffusion models achieve state-of-the-art performance in image and audio generating tasks (Song and Ermon, 2019; Dathathri et al., 2019; Song et al., 2020b; Ho et al., 2020) and are one of the fundamental building blocks of the more advanced image synthesis system, e.g., DALL-E-2 (Ramesh et al., 2022) and stable diffusion (Rombach et al., 2022). A standard diffusion model (Sohl-Dickstein et al., 2015; Ho et al., 2020) consists of a forward process and a backward process: In the forward process, a data point is sequentially corrupted by Gaussian random noises and in the limit the data distribution is transformed into white noise; In the backward process, a denoising neural network is trained to sequentially remove the added noise in the data and restore the clean data point. Using the trained denoising network for the backward process, one can generate diverse and high fidelity samples by first sampling from the standard Gaussian distribution and then progressively removing noises.