Reviews: Collaborative Learning for Deep Neural Networks

Summary The paper proposes a collaborative learning strategy in which multiple classifier heads of the same deep network are trained simultaneously on the same data. Unlike ensembling, knowledge distillation, multi-task learning or training auxiliary classifiers, this seems more effective from the point of view of training time and memory utilization. In addition, the proposed training objective improves generalization and makes the network more robust to label noise. For the intermediate representations that are shared, the authors propose a backpropagation-rescaling technique so as to control the variance of gradients being propagated to the previous layers. Empirical results demonstrate the efficacy of the method w.r.t.