Large-scale hypothesis testing supports probability claims about individual hypotheses, as in empirical Bayes methods for estimating local false discovery rates. We study how such claims can be interpreted as approximately calibrated forecasts of the null hypothesis, yielding interpretable error probabilities even under model misspecification. Our approach draws conceptual inspiration from probabilistic forecasting but addresses a different challenge: unlike forecasting, where labels are eventually observed, in multiple testing the ground truth is never revealed, so calibration must be assessed stochastically and established indirectly. We address this challenge by constructing a set of pseudo-labels, derived from the spacings of ordered $p$-values, which have the local false discovery rate as their regression target. Our construction unlocks existing tools for assessing and performing post-hoc calibration in multiple testing. Notably, we find on a large-scale empirical survey of published psychology and neuroscience literature that the $q$-value, a popular error measure based on the false discovery rate, can be severely miscalibrated.

Neural Information Processing Systems http://nips.cc/

We study the problem of estimating ap-dimensional s-sparse vector in a linear model with Gaussian design and additivenoise.

Workinginthestandard potential outcomes framework, we propose a data-driven modification to an arbitrary (nonparametric) prior based on the propensity score that corrects for the first-orderposteriorbias,therebyimprovingperformance.Weillustrateourmethod for Gaussian process (GP) priors using (semi-)synthetic data.

Since the VB algorithm does not depend on the unknown truth to achieve optimality, our results shed light on effective prior choices.

We study theregretincurred bytheagent, firstwhen sheknowsherrewardfunction but does not know the distribution of the task duration, and then when she does not knowher reward function, either.

We consider here the sparse Gaussian process regression (SGPR) approach introduced by Titsias [31], which is widely used in practice (see [1, 9] for implementations) and has been studied in many recent works [13,21,5,6,38,28,32,22,23].

We show that the matrix perspective function, which is jointly convex in the Cartesian product of a standard Euclidean vector space and a conformal space of symmetric matrices, has a proximity operator in an almost closed form.