calibration property
- Europe > Norway (0.04)
- Europe > France (0.04)
- Africa > Ethiopia > Addis Ababa > Addis Ababa (0.04)
Blackwell's Approachability for Sequential Conformal Inference
Principato, Guillaume, Stoltz, Gilles
Conformal inference [Vovk et al., 2005] provides a general procedure for constructing prediction sets with guaranteed coverage, under the assumption that the data are exchangeable. This assumption, however, is often too restrictive: it typically fails in sequential or time-dependent settings such as time series forecasting, where the distribution of observations may shift over time. To address this issue, Gibbs and Cand` es [2021] introduced Adaptive Conformal Inference (ACI), which extends Conformal Prediction (CP) to adversarial environments. ACI adapts to distribution shifts by updating prediction intervals in response to observed outcomes, ensuring that the empirical coverage converges to the desired level. While effective in maintaining coverage, ACI and its extensions generally lack efficiency guarantees-for instance, there is no control over the average length of prediction intervals in adversarial regimes. In this work, we study sequential conformal inference as a repeated two-player finite game and invoke Blackwell's theory of approachability to characterize feasible objectives. Building on this perspective, we design a calibration-based algorithm that ensures asymptotic validity while achieving asymptotic efficiency under mild assumptions. Our approach relies on the notion of opportunistic approachability [Bernstein et al., 2014], which allows the learner to exploit potential restrictions in the opponent's play. We argue that such assumptions better fit the typical use cases of ACI-such as distributional drift or regime switching-than the fully adversarial setting.
- Europe > France (0.04)
- Asia > Middle East > Jordan (0.04)
- Europe > Norway (0.04)
- Europe > France (0.04)
- Africa > Ethiopia > Addis Ababa > Addis Ababa (0.04)
No-Regret Learning with Unbounded Losses: The Case of Logarithmic Pooling
Neyman, Eric, Roughgarden, Tim
For each of $T$ time steps, $m$ experts report probability distributions over $n$ outcomes; we wish to learn to aggregate these forecasts in a way that attains a no-regret guarantee. We focus on the fundamental and practical aggregation method known as logarithmic pooling -- a weighted average of log odds -- which is in a certain sense the optimal choice of pooling method if one is interested in minimizing log loss (as we take to be our loss function). We consider the problem of learning the best set of parameters (i.e. expert weights) in an online adversarial setting. We assume (by necessity) that the adversarial choices of outcomes and forecasts are consistent, in the sense that experts report calibrated forecasts. Imposing this constraint creates a (to our knowledge) novel semi-adversarial setting in which the adversary retains a large amount of flexibility. In this setting, we present an algorithm based on online mirror descent that learns expert weights in a way that attains $O(\sqrt{T} \log T)$ expected regret as compared with the best weights in hindsight.
- North America > United States > New York > New York County > New York City (0.04)
- North America > United States > Wisconsin > Dane County > Madison (0.04)
- Europe > Norway (0.04)
- (2 more...)
On Calibration of Modern Quantized Efficient Neural Networks
We explore calibration properties at various precisions for three architectures: ShuffleNetv2, GhostNet-VGG, and MobileOne; and two datasets: CIFAR-100 and PathMNIST. The quality of calibration is observed to track the quantization quality; it is well-documented that performance worsens with lower precision, and we observe a similar correlation with poorer calibration. This becomes especially egregious at 4-bit activation regime. GhostNet-VGG is shown to be the most robust to overall performance drop at lower precision. We find that temperature scaling can improve calibration error for quantized networks, with some caveats. We hope that these preliminary insights can lead to more opportunities for explainable and reliable EdgeML.
- North America > Canada > Ontario > Waterloo Region > Waterloo (0.05)
- Europe > France (0.04)
- Asia > South Korea > Seoul > Seoul (0.04)