Preference learning algorithms (e.g., RLHF and DPO) are frequently used to steer LLMs to produce generations that are more preferred by humans, but our understanding of their inner workings is still limited.

Part of this work was done during Kang Xu's internship at Shanghai Artificial Intelligence Laboratory.

Compositionality is still a challenging problem.

Clustering is a fundamental task in data science with wide-ranging applications.

However, there are three challenges.

We select a representative subset consisting of 22 datasets and propose it as the MESS benchmark.