Learning to Abstain from Binary Prediction

Consider a general practice physician treating a patient with unusual or ambiguous symptoms. The general practitioner often does not have the capability to confidently diagnose such an ailment. The doctor is faced with a difficult choice: either commit to a potentially erroneous diagnosis and act on it, which can have catastrophic consequences; orabstain from any such diagnosis and refer the patient to a specialist or hospital instead, which is safer but will certainly cost extra time and resources. Such a situation motivates the study of classifiers which are able not only to form a hypothesis about the correct classification, but also abstain entirely from making a prediction. A sufficiently self-aware abstaining classifier might abstain on examples on which it is most unsure about the label, lowering the average prediction error it suffers when it does commit to a prediction. Like the doctor in the example, however, there is typically no use in abstaining on all data, so the amount of overall abstaining is somehow restricted. The classifier must allocate limited abstentions where they will most reduce error. There has been much historical work in decision theory and machine learning on learning such abstaining classifiers (e.g.