Reliable Probabilistic Classification with Neural Networks

They have been applied to a great variety of problems and fields with very good results. However, most machine learning techniques do not provide any indication about the uncertainty of each of their predictions, which would have been very beneficial for most applications and especially for risk sensitive settings such as medical diagnosis [1]. An indication of the likelihood of each prediction being correct notifies the user of a system about how much he can rely on each prediction and enables him to take more informed decisions. A solution to this problem was given by a recently developed machine learning theory called Conformal Prediction (CP) [2]. CP can be used for extending traditional machine learning algorithms and developing methods (called Conformal Predictors) whose predictions are guaranteed to satisfy a given level of confidence without assuming anything more than that the data are independently and identically distributed (i.i.d.). More specifically, CPs produce as their predictions a set containing all the possible classifications needed to satisfy the required confidence level. To date many different CPs have been developed, see e.g.