Multimodal Large Language Models Make Text-to-Image Generative Models Align Better

Recent studies have demonstrated the exceptional potentials of leveraging human preference datasets to refine text-to-image generative models, making it to generate more human-preferred images. Despite these advances, current human preference datasets are either prohibitively expensive to construct or suffer from a lack of diversity in preference dimensions, resulting in limited applicability for instruction tuning in open-source text-to-image generative models and hinder further exploration. To address these challenges, we first leverage multimodal large language models to create VisionPrefer, a fine-grained preference dataset that captures multiple preference aspects (prompt-following, aesthetic, fidelity, and harmlessness). Then we train a corresponding reward model, VP-Score, over VisionPrefer to guide the tuning of text-to-image generative models. The preference prediction accuracy of VP-Score is validated to be comparable to that of human annotators.