Unsupervised Transfer Learning via Adversarial Contrastive Training

Data representation is a fundamental aspect of machine learning that significantly influences model performance, efficiency, and interpretability Rumelhart et al. (1986); Bengio et al. (2012); LeCun et al. (2015). In the era of deep learning, neural networks have become the primary tools for data representation in computer vision and natural language processing, leveraging their capacity to automatically extract features. For instance, neural networks trained on labeled data can serve as effective feature extractors when the final layer is removed Goodfellow et al. (2016). The core idea of transfer learning is to leverage learned representations from large upstream datasets to enhance the performance of target-specific downstream tasks. A particularly effective paradigm within transfer learning is pretraining followed by fine-tuning, which has gained increasing attention for its demonstrated efficiency in various studies Schroff et al. (2015); Dhillon et al. (2020); Chen et al. (2019, 2020c). During the pretraining phase, a representation is learned using a large, general dataset with annotations, which is then transferred to the target-specific task. In the fine-tuning stage, a relatively simple model is typically trained on the learned representation to address the specific problem at hand. There is a wide variety of transfer learning methods, along with corresponding theoretical guarantees, that have been proposed.