Group Robust Preference Optimization in Reward-free RLHF

While these data often come from diverse labelers' groups (e.g., different demographics, ethnicities, company teams, etc.), traditional RLHF approaches