Despite the success of reinforcement learning from human feedback (RLHF) in aligning language models with human values, reward hacking, also termed reward overoptimization, remains a critical challenge. This issue primarily arises from reward misgeneralization, where reward models (RMs) compute reward using spurious features that are irrelevant to human preferences. In this work, we tackle this problem from an information-theoretic perspective and propose a framework for reward modeling, namely InfoRM, by introducing a variational information bottleneck objective to filter out irrelevant information.Notably, we further identify a correlation between overoptimization and outliers in the IB latent space of InfoRM, establishing it as a promising tool for detecting reward overoptimization.Inspired by this finding, we propose the Cluster Separation Index (CSI), which quantifies deviations in the IB latent space, as an indicator of reward overoptimization to facilitate the development of online mitigation strategies. Extensive experiments on a wide range of settings and RM scales (70M, 440M, 1.4B, and 7B) demonstrate the effectiveness of InfoRM. Further analyses reveal that InfoRM's overoptimization detection mechanism is not only effective but also robust across a broad range of datasets, signifying a notable advancement in the field of RLHF.

Also known as the phase retrieval problem, it is a fundamental challenge in nano-, bio-and astronomical imaging systems, and speech processing. The problem is ill-posed, and therefore additional assumptions on the signal and/or the measurements are necessary.

A plan for a question required the composition of REQUEST and INFORM and led to the development of two new kinds of informing speech acts, INFORMREF To plan a yes/no question about some proposition P. one should think that the and INFORMIF, and their mediating acts. The INFORMREF acts lead to hearer knows whether P is true or false (or, at least "might know"). An approximate "what," "when," and "where" questions while INFORMIF results in a yes/no representation of AGT2's knowing whether P is true or false is OR (AGT2 question.2' The reason for these new acts is that, in planning a REQUEST that BELIEVE P, AGT2 BELIEVE -- P)).'9 Such goals are often created, as modelled someone else perform an INFORM act, one only has incomplete knowledge of by our type 4 inference, when a planner does not know the truth-value of P. their beliefs and goals; but an INFORM, as originally defined can only be Typical circumstances in which an agent may acquire such disjunctive beliefs planned when one knows what is to be said.