Order Matters in the Presence of Dataset Imbalance for Multilingual Learning

We present a simple yet effective method of pre-training on high-resource tasks, followed by fine-tuning on a mixture of high/low-resource tasks. We provide a thorough empirical study and analysis of this method's benefits