The Fisher-Rao Loss for Learning under Label Noise

Supervised classification is an important problem in machine learning. Training a classifier (e.g., a deep neural network) can be done by empirical risk minimisation: a numerical optimisation algorithm is applied to find the model parameters that minimise the mean value of a loss function on the training dataset. Choosing a suitable loss function is essential, since different choices can affect the performance of the resulting classifier, as well as the training dynamics. The output of a neural network trained for classification is often interpreted as giving a conditional probability distribution p(y|x) of the class y given the input x, which prompts the use of cross entropy as a loss function [1, 2, 3]. Although originally used for regression problems, the mean squared error is also used as loss function, and several works have compared these two losses [4, 5, 6, 7, 8]. Moreover, the design of new loss functions has been a topic of interest, and those are often tailored for specific problems or contexts, with many different inspirations, such as the correntropy similarity measure [9], the Wasserstein distance [10, 11], and persistent homology [12]. A case of practical interest is when training datasets are corrupted with label noise, i.e., some of the class labels may be incorrect. This is a well-studied problem in machine learning: one of its sources is crowdsourcing labelling, and it can impact the performance of the generated model [13, 14]. Many of the proposed solutions to mitigate this issue involve modifying the learning algorithms and have no theoretical guarantees of robustness.