Quadratic Metric Elicitation with Application to Fairness

Given a classification problem, which performance metric should the classifier optimize? This question is often faced by practitioners while developing machine learning solutions. For example, consider cancer diagnosis where the doctor applies a cost-sensitive predictive model to classify patients into cancer categories [53, 56]. Although it is clear that the chosen costs directly determine the model decisions and thus patient outcomes, it is not clear how to quantify expert intuition into precise quantitative cost tradeoffs, i.e. the performance metric. Indeed this is also true for a variety of other domains where picking the right metric is a critical challenge [8]. Hiranandani et al. [16, 17] addressed this issue by formalizing the problem of Metric Elicitation (ME), where the goal is to estimate a performance metric using preference feedback from a user. The motivation is that by employing metrics that reflect a user's innate tradeoffs, one can learn models that best capture the user preferences [16].