Metric Elicitation is a principled framework for selecting the performance metric that best reflects implicit user preferences. However, available strategies have so far been limited to binary classification. In this paper, we propose novel strategies for eliciting multiclass classification performance metrics using only relative preference feedback. We also show that the strategies are robust to both finite sample and feedback noise.

Metric Elicitation is a principled framework for selecting the performance metric that best reflects implicit user preferences. However, available strategies have so far been limited to binary classification. In this paper, we propose novel strategies for eliciting multiclass classification performance metrics using only relative preference feedback. We also show that the strategies are robust to both finite sample and feedback noise.

This paper studies the metric elicitation problem, a problem proposed by [7] (AISTATS'19 paper). The authors defined two types of metrics for multi-class classification, Diagonal Linear Performance Metric (DLPM) and Linear Performance Metric (LPM), and designed algorithms for the two type of metrics respectively. The algorithm for DLPM is relatively simple: just use binary-search like [7] and apply it for (k-1) times. The LPM case is much more complicated - the authors proposed a coordinate-wise binary-search style algorithm based on the geometry of the feasible set. Theoretical guarantees for both algorithms are provided.

Metric Elicitation is a principled framework for selecting the performance metric that best reflects implicit user preferences. However, available strategies have so far been limited to binary classification. In this paper, we propose novel strategies for eliciting multiclass classification performance metrics using only relative preference feedback. We also show that the strategies are robust to both finite sample and feedback noise.

Metric Elicitation is a principled framework for selecting the performance metric that best reflects implicit user preferences. However, available strategies have so far been limited to binary classification. In this paper, we propose novel strategies for eliciting multiclass classification performance metrics using only relative preference feedback. We also show that the strategies are robust to both finite sample and feedback noise. Papers published at the Neural Information Processing Systems Conference.