Improving Screening Processes via Calibrated Subset Selection

Screening is an essential part of many selection processes, where an often intractable number of candidates is reduced to a shortlist of the most promising candidates for detailed--and more resource intensive--evaluation. Screening thus enables an allocation of resources that improves the overall quality of the decisions under limited resources. Examples of such screening problems are: finding patients in a large database of electronic health records to manually evaluate for qualification to take part in a medical trial [1]; the first stage of a multi-stage retrieval pipeline of a search engine [2]; or which people to reach out to with a personalized invitation to apply to a specific job posting [3]. In each of these examples, there is significant pressure to make high-quality, unbiased screening decisions quickly and efficiently, often about thousands or even millions of candidates under limited resources and additional diversity requirements [4, 5, 6, 2]. While these screening decisions have been made manually or through manually constructed rules in the past, automated predictive tools for optimizing screening decisions are becoming more prevalent [7, 8, 3, 9]. Algorithmic screening has been typically studied together with other high-stakes decision making problems as a supervised learning problem [10, 11, 12]. Under this perspective, algorithmic screening reduces to: (i) training a classifier that estimates the probability that a candidate is qualified given a set of observable features; (ii) designing a deterministic threshold rule that shortlists candidates by thresholding the candidates' probability values estimated by the classifier. Here, the classifier and the threshold rule aim to maximize a measure of average accuracy and average utility, respectively, possibly subject to diversity constraints.