On the Benefits of Public Representations for Private Transfer Learning under Distribution Shift

This has motivated recent works to explore the benefits of incorporating publicly available data into private training, e.g., by pretraining a model on public data and then finetuning it using private data.