Everything old is new again: A multi-view learning approach to learning using privileged information and distillation

Transferring knowledge learned by a powerful model ("teacher") to a simpler model ("student") has become a theme in machine learning. The goal of the knowledge transfer is to have the teacher guide the learning process of the student, so as to achieve high prediction accuracy, or to reduce the sample complexity, which are otherwise hard for the student to achieve by itself. This learning paradigm is practically useful when it is necessary to deploy simpler models to real-world systems, which requires small memory footage or fast processing time. We focus on two specific settings of knowledge transfer in this work. The first one is learning using privileged information (LUPI) [Vapnik and Vashist, 2009], in which the teacher provides an additional set of feature representation to the student during its training process but not the test time, and the extra feature set contains richer information to make the learning problem easier for the student; an example is that the "student may normally only have access to the image of a biopsy to predict the existence of cancer, but during the training process, it also has access to the medical report of an oncologist" [Lopez-Paz et al., 2015]. The second setting is distillation [Ba and Caruana, 2014, Hinton et al.,