"NIPS Neural Information Processing Systems 8-11th December 2014, Montreal, Canada",,, "Paper ID:","1703" "Title:","How transferable are features in deep neural networks?" First provide a summary of the paper, and then address the following criteria: Quality, clarity, originality and significance. This paper aims to quantify the transferrability of features in deep neural networks, both in terms of the difference between source and target tasks and in terms of the depth of the features being transferred. To this end, the authors take an existing network (Krizhevsky et al. 2012), and performs generalization by fixing different layer depth and by transferring between different splits of the ImageNet dataset. I find the paper sufficiently interesting in the sense that, despite the many papers describing the success of feature transfer (e.g.

Recent advances in unsupervised learning have shown that unsupervised pre-training, followed by fine-tuning, can improve model generalization. However, a rigorous understanding of how the representation function learned on an unlabeled dataset affects the generalization of the fine-tuned model is lacking. Existing theoretical research does not adequately account for the heterogeneity of the distribution and tasks in pre-training and fine-tuning stage. To bridge this gap, this paper introduces a novel theoretical framework that illuminates the critical factor influencing the transferability of knowledge acquired during unsupervised pre-training to the subsequent fine-tuning phase, ultimately affecting the generalization capabilities of the fine-tuned model on downstream tasks. We apply our theoretical framework to analyze generalization bound of two distinct scenarios: Context Encoder pre-training with deep neural networks and Masked Autoencoder pre-training with deep transformers, followed by fine-tuning on a binary classification task. Finally, inspired by our findings, we propose a novel regularization method during pre-training to further enhances the generalization of fine-tuned model. Overall, our results contribute to a better understanding of unsupervised pre-training and fine-tuning paradigm, and can shed light on the design of more effective pre-training algorithms.