Neural Information Processing Systems http://nips.cc/

Neural Information Processing Systems http://nips.cc/

Neural Information Processing Systems http://nips.cc/

Learning from human feedback has been shown to improve text-to-image models. These techniques first learn a reward function that captures what humans care about in the task and then improve the models based on the learned reward function. Even though relatively simple approaches (e.g., rejection sampling based on reward scores) have been investigated, fine-tuning text-to-image models with the reward function remains challenging. In this work, we propose using online reinforcement learning (RL) to fine-tune text-to-image models. We focus on diffusion models, defining the fine-tuning task as an RL problem, and updating the pre-trained text-to-image diffusion models using policy gradient to maximize the feedbacktrained reward. Our approach, coined DPOK, integrates policy optimization with KL regularization. We conduct an analysis of KL regularization for both RL fine-tuning and supervised fine-tuning. In our experiments, we show that DPOK is generally superior to supervised fine-tuning with respect to both image-text alignment and image quality.

Sampling in discrete spaces, with critical applications in simulation and opti-1 mization, has recently been boosted by significant advances in gradient-based2 approaches that exploit modern accelerators like GPUs. However, two key chal-3 lenges hinder the further research progress in discrete sampling. First, since there4 is no consensus on experimental settings, the empirical results in different research5 papers are often not comparable. Secondly, implementing samplers and target6 distributions often requires a nontrivial amount of effort in terms of calibration,7 parallelism, and evaluation. To tackle these challenges, we propose DISCS (DIS-8 Crete Sampling), a tailored package and benchmark that supports unified and9 efficient implementation and evaluations for discrete sampling in three types of10 tasks: sampling for classical graphical models, combinatorial optimization, and11 energy based generative models. Throughout the comprehensive evaluations in12 DISCS, we acquired new insights into scalability, design principles for proposal13 distributions, and lessons for adaptive sampling design.