When resampling/reweighting improves feature learning in imbalanced classification?: A toy-model study

Classifiers applied to such datasets tend to perform poorly for minority classes, which poses a major challenge in areas such as visual recognition. Although several methods to mitigate class imbalance have been proposed so far [6, 7, 8], recent advances of deep learning have shed new light on this issue, resulting in numerous studies from the perspective of applying those approaches to classifiers based on deep neural networks (DNNs) [5, 9, 10, 11, 12, 13, 1, 2, 14, 15, 16, 17]. Among those approaches proposed so far, we focus on two simple strategies, reweighting and resampling, which are commonly employed to mitigate class imbalance. The resampling strategy tries to balance the samples in the dataset by oversampling the minority classes and/or undersampling the majority classes, while the reweighting strategy puts an additional weight to each term of the loss in order to counterweight the class imbalance. The effectiveness of these strategies has been empirically verified in a wide range of studies [13, 1, 2, 14, 6, 7]. In spite of these pieces of work, transparent description or understanding about when they are useful or not would still be imcomplete. In particular, how class imbalance may affect the quality of feature learning would be an important problem in the context of representation learning in DNNs, but a thorough understanding of this issue is still missing. Recently, [2] reported an interesting observation that feature learning becomes better if no resampling is applied. More specifically, on the basis of their extensive experiment on visual recognition tasks using DNNs, they reported that the best classification performance was achieved when the whole network was first trained without any resampling and then only the last output layer (final classifier) was retrained with class-balanced resampling.