86b3e165b8154656a71ffe8a327ded7d-Supplemental.pdf

Pretrained language models have achieved state-of-the-art performance when adapted to a downstream NLP task. However, theoretical analysis of these models is scarce and challenging since the pretraining and downstream tasks can be very different.