Post-hoc analysis of Arabic transformer models

Arabic is a Semitic language which is widely spoken with many dialects. Given the success of pre-trained language models, many transformer models trained on Arabic and its dialects have surfaced. While there have been an extrinsic evaluation of these models with respect to downstream NLP tasks, no work has been carried out to analyze and compare their internal representations. We probe how linguistic information is encoded in the transformer models, trained on different Arabic dialects. Figure 1: Data regimes of various pre-trained We perform a layer and neuron analysis Transformer models of Arabic on the models using morphological tagging tasks for different dialects of Arabic and a dialectal identification task.