Aligning large language models (LLMs) to human preferences typically relies on aggregating pooled feedback into a single reward model. However, this standard approach assumes that all labelers share the same underlying preferences, ignoring the fact that real-world labelers are highly heterogeneous and usually anonymous. Consequently, relying solely on binary choice data fundamentally distorts the learned policy, making the true population-average preference unidentifiable. To overcome this critical limitation, we demonstrate that augmenting preference datasets with a simple, secondary signal -- the user's response time -- can restore the identifiability of the population's average preference. By modeling each decision as a Drift-Diffusion Model (DDM), we introduce a novel, consistent estimator of heterogeneous preferences that successfully corrects the distortions of standard choice-only labels. We prove that our estimator asymptotically converges to the true average preference even in extreme cases where each anonymous labeler contributes only a single choice. Empirically, across both synthetic and real-world datasets, our method consistently outperforms standard baselines that otherwise fail and plateau at a bias floor. Because response times are essentially free to record and require zero user tracking or identification, our results bring promises and open up new opportunities for future data-collection pipelines to improve the social benefit without requiring user-level identifiers or repeated elicitations.

For labels taking values in a finite metric space, we introduce techniques new to weak supervision based on pseudo-Euclidean embeddings andtensor decompositions, providing anearly-consistent noise rate estimator.

Unlike in Metamath or Lean, we do not have access to a training set of human annotated proofs for this environment.

These images are taken from the last task, so there is no catastrophic interference.

Additional notation For a matrix A Rd1 d2, A op is the operator norm (with respect to Euclidean norms), and A F istheFrobenius norm ofA. The main intuition behind the HMM considered in this paper comes from the correlation decay phenomenon ingraphicalmodel. Informally, we expect that there is one sign flip (i.e., Si = Si+1) per 1δ samples. To begin with the analysis of the estimator in Figure 2, the following lemma is a simple, yet key tool for the proof. It establishes the variance of the random gainS.

The formal language for mathematical expressions to which our certification algorithm is applied is specified by the grammar depicted in Figure 1. The language is rich enough to cover all the examples in the main paper and this supplement. In this grammar, number is a placeholder for an arbitrary floating point number, variable is a placeholder for variable names starting with a Latin character and function is a placeholder for the supported elementary differentiable functions like exp,log and sum. Here, is used for transposition and a preceding . Here are some examples from the language (the fist example uses a transposition and the fifth and seventh example use elementwise operations): 2-norm Xw y 2: (X*w-y)'*(X*w-y) logistic log(1+exp(x)): log(1+exp(x)) 1 quadratic x2: x^2 relative entropy xlog(x/y): x*log(x/y), x>0, y>0 logistic regression Our implementation of the Hessian approach works on vectorized and normalized expression DAGs (directed acyclic graphs) for Hessians that contain every subexpression exactly once.

In physical systems a particle instead interacts only with a finite number of other particles, hence the density field remains highly fluctuating. The effect of theθ(w x) term is to select one particular half-space over which the integralisdone. To estimate the fluctuations due to a finite number of nodes, we will have to estimate the width of the output distribution for a given set of parameters. Toestimate the error inFigure 1d ofthe main text, we ask what are the values ofxk = xcosθ such that the average output plus or minus a standard deviation, divided by M, would be equal to the threshold. Since the standard deviation involves|x|2, we estimate its average value for points 3 with a givenxk, i.e.

Computational models of sequence memory have been proposed where recurrent Hopfield-like neural networks are trained with temporally asymmetric Hebbian rules.