In machine learning, model calibration and predictive inference are essential for producing reliable predictions and quantifying uncertainty to support decision-making.

Neural Information Processing Systems http://nips.cc/

We study the problem of offline changepoint localization in a distribution-free setting. One observes a vector of data with a single changepoint, assuming that the data before and after the changepoint are iid (or more generally exchangeable) from arbitrary and unknown distributions. The goal is to produce a finite-sample confidence set for the index at which the change occurs without making any other assumptions. Existing methods often rely on parametric assumptions, tail conditions, or asymptotic approximations, or only produce point estimates. In contrast, our distribution-free algorithm, CONformal CHangepoint localization (CONCH), only leverages exchangeability arguments to construct confidence sets with finite sample coverage. By proving a conformal Neyman-Pearson lemma, we derive principled score functions that yield informative (small) sets. Moreover, with such score functions, the normalized length of the confidence set shrinks to zero under weak assumptions. We also establish a universality result showing that any distribution-free changepoint localization method must be an instance of CONCH. Experiments suggest that CONCH delivers precise confidence sets even in challenging settings involving images or text.

Generative models at times produce "invalid" outputs, such as images with generation artifacts and unnatural sounds.

A.1 Notationandpreliminaries We consider the metric space(X,d( , )) where d : X X R+. We consider`( , ) to be the cross-entropyloss `log(M(x),y), ylogσ(M(x)) (1 y)logσ((1 M(x))) (whereσ(x) = 11+exp( x) isthe sigmoid function) orthe`2 loss.