Optimizing for ROC Curves on Class-Imbalanced Data by Training over a Family of Loss Functions

Although binary classification is a well-studied problem in computer vision, training reliable classifiers under severe class imbalance remains a challenging problem. Recent work has proposed techniques that mitigate the effects of training under imbalance by modifying the loss functions or optimization methods. While this work has led to significant improvements in the overall accuracy in the multi-class case, we observe that slight changes in hyperparameter values of these methods can result in highly variable performance in terms of Receiver Operating Characteristic (ROC) curves on binary problems with severe Figure 1: Distribution of Area Under the ROC Curve (AUC) imbalance. To reduce the sensitivity to hyperparameter values obtained by training the same model on the SIIM-choices and train more general models, ISIC Melanoma classification dataset with 48 different combinations we propose training over a family of loss functions, of hyperparameters on VS loss. Results are shown instead of a single loss function. We develop at three different imbalance ratios. As the imbalance becomes a method for applying Loss Conditional more severe, model performance drops and the Training (LCT) to an imbalanced classification variance in performance drastically increases.