An Elementary Predictor Obtaining $2\sqrt{T}$ Distance to Calibration

Probabilistic predictions of binary outcomes are said to be calibrated, if, informally, they are unbiased conditional on their own predictions. For predictors that are not perfectly calibrated, there are a variety of ways to measure calibration error. Perhaps the most popular measure is Expected Calibration Error (ECE), which measures the average bias of the predictions, weighted by the frequency of the predictions. ECE has a number of difficulties as a measure of calibration, not least of which is that it is discontinuous in the predictions. Motivated by this, B lasiok et al. [2023] propose a different measure: distance to calibration, which measures how far a predictor is in l