A taxonomy of Transformer based pre-trained language models (TPTLM)

General pretraining: Models like GPT-1, BERT etc are pretrained on general corpus. Language-based: Models could be trained on languages either monolingual or multilingual. TPTLM could be classified based on their architecture. A T-PTLM can be pretrained using a stack of encoders or decoders or both. Self supervised learning - SSL is one of the key ingredients in building T-PTLMs.