Contrastive Energy Prediction for Exact Energy-Guided Diffusion Sampling in Offline Reinforcement Learning

A common paradigm for Guided sampling is a vital approach for applying introducing human preference in diffusion models is guided diffusion models in real-world tasks that embeds sampling, which includes classifier guidance (Dhariwal & human-defined guidance during the sampling procedure. Nichol, 2021), classifier-free guidance (Ho & Salimans, This paper considers a general setting 2021) and other guidance methods (Nichol et al., 2021; Ho where the guidance is defined by an (unnormalized) et al., 2022c; Zhao et al., 2022). By leveraging guided sampling, energy function. The main challenge for diffusion models can realize amazing text-to-image this setting is that the intermediate guidance during generation (Saharia et al., 2022b), video generation (Ho the diffusion sampling procedure, which is et al., 2022c;a; Yang et al., 2022; Zhou et al., 2022), controllable jointly defined by the sampling distribution and text generation (Li et al., 2022), inverse molecular the energy function, is unknown and is hard to design (Bao et al., 2022b) and reinforcement learning (Janner estimate. To address this challenge, we propose et al., 2022; Chen et al., 2022; Ajay et al., 2022).