Human feedback can alter language models in unpredictable and undesirable ways, as practitioners lack a clear understanding of what feedback data encodes. While prior work studies preferences over certain attributes (e.g., length or sycophancy), automatically extracting relevant features without pre-specifying hypotheses remains challenging. We introduce What's In My Human Feedback? (WIMHF), a method to explain feedback data using sparse autoencoders. WIMHF characterizes both (1) the preferences a dataset is capable of measuring and (2) the preferences that the annotators actually express. Across 7 datasets, WIMHF identifies a small number of human-interpretable features that account for the majority of the preference prediction signal achieved by black-box models. These features reveal a wide diversity in what humans prefer, and the role of dataset-level context: for example, users on Reddit prefer informality and jokes, while annotators in HH-RLHF and PRISM disprefer them. WIMHF also surfaces potentially unsafe preferences, such as that LMArena users tend to vote against refusals, often in favor of toxic content. The learned features enable effective data curation: re-labeling the harmful examples in Arena yields large safety gains (+37%) with no cost to general performance. They also allow fine-grained personalization: on the Community Alignment dataset, we learn annotator-specific weights over subjective features that improve preference prediction. WIMHF provides a human-centered analysis method for practitioners to better understand and use preference data.

In the early weeks of 2023, as worry about ChatGPT and other artificial intelligence tools was ratcheting up dramatically in the public conversation, a tweet passed through the many interlocking corners of Book Twitter. "Imagine if every Book is converted into an Animated Book and made 10x more engaging," it read. Huge opportunity here to disrupt Kindle and Audible." The tweet's author, Gaurav Munjal, cofounded Unacademy, which bills itself as "India's largest learning platform"--and within the edtech context, where digitally animated books can be effective teaching tools, his suggestion might read a certain way. But to a broader audience, the sweeping proclamation that AI will make "every" book "10x more engaging" seemed absurd, a solution in search of a problem, and one predicated on the idea that people who choose to read narrative prose (instead of, say, watching a film or playing a game) were somehow bored or not engaged with their unanimated tomes.