Plug-and-Play Training Framework for Preference Optimization