The importance of understanding instance-level noisy labels

A salient feature of a over-parameterized model, e.g. a deep neural network, is its ability to memorize examples Zhang et al. (2016); Neyshabur et al. (2017), and the memorization has proven to benefit the generalization Arpit et al. (2017); Feldman (2020); Feldman & Zhang (2020). Nonetheless, the potential existence of label noise, combined with the memorization effect, might lead to detrimental consequence Song et al. (2020); Yao et al. (2020a); Cheng et al. (2020b); Chen et al. (2019); Han et al. (2020); Song et al. (2020). In light of the reported empirical evidence of harms caused by over-memorizing noisy labels, we set out to understand this effect analytically. Built on a recent analytical framework Feldman (2020), we demonstrate the varying effects of memorizing noisy labels associated with instances that sit at the different spectra of the sample distribution (Section 3). Soon since the above negative effect was empirically shown, learning with noisy labels has been recognized as a challenging and important task.