More is Less: The Pitfalls of Multi-Model Synthetic Preference Data in DPO Safety Alignment

Open in new window