Robust Neural Network Classification via Double Regularization

The presence of mislabelled observations in data is a notoriously challenging problem in statistics and machine learning, associated with poor generalisation properties for both traditional classifiers and, perhaps even more so, flexible classifiers like neural networks. Here we propose a novel double regularisation of the neural network training loss that combines a penalty on the complexity of the classification model and an optimal reweighting of training observations. The combined penalties result in improved generalisation properties and strong robustness against overfitting in different settings of mislabelled training data and also against variation in initial parameter values when training. We provide a theoretical justification, by proving that for logistic regression with multivariate Gaussian covariates, our proposed method can find the correct parameters exactly, i.e. estimate the parameters to exactly the same value as if there were no mislabelling. We demonstrate the double regularisation model, here denoted by DRFit, for neural net classification of (i) MNIST and (ii) CIFAR-10, in both cases with simulated mislabelling. We also illustrate that DRFit identifies mislabelled data points with very good precision. This provides strong support for DRFit as a practical of-the-shelf classifier, since, without any sacrifice in performance, we get a classifier that simultaneously reduces overfitting against mislabelling and gives an accurate measure of the trustworthiness of the labels.