Accurate Probability Calibration for Multiple Classifiers

In classification problems,isotonic regression has been commonly used to map the prediction scoresto posterior class probabilities. However,isotonic regression may suffer from overfitting, and the learnedmapping is often discontinuous. Besides, current efforts mainly focus on the calibration of a single classifier.As different classifiers have different strengths, a combination of themcan lead to better performance.In this paper, we propose a novelprobability calibrationapproachfor such an ensemble of classifiers.We first construct isotonic constraints on the desired probabilitiesbased on soft voting of the classifiers.Manifold information is also incorporated to combat overfitting and ensurefunction smoothness. Computationally,the extended isotonic regression modelcan be learned efficiently by a novel optimization algorithmbased on the alternating direction method of multipliers (ADMM). Experimentson a number of real-world data sets demonstrate that theproposed approach consistently outperforms independent classifiers and othercombinations of the classifiers' probabilities in terms of the Brier score and AUC.