Learning and Evaluating General Linguistic Intelligence

Advances in deep learning techniques (e.g., attention mechanisms, memory modules, and architecture search) have considerably improved natural language processing (NLP) models on many important tasks. For example, machine performance on both Chinese-English machine translation and document question answering on the Stanford question answering dataset (SQuAD; Rajpurkar et al., 2016) has been claimed to have surpassed human levels (Hassan et al., 2018; Devlin et al., 2018). While the tasks that initiated learning-based NLP models were motivated by external demands and are important applications in their own right (e.g., machine translation, question answering, automatic speech recognition, text to speech, etc.), there is a marked and troubling tendency for recent datasets to be set up to be easy to solve with little in the way of generalization or abstraction; for instance, ever larger datasets created by crowd-sourcing processes that may not well approximate the natural distributions they are intended to span, although there are some notable counterexamples (Kwiatkowski et al., 2019). When there exist multiple datasets that are representative of the exact same task from different domains (e.g., various question answering datasets), we rarely evaluate on all of them. This state of affairs promotes development of models that only work well for a specific purpose, overestimates our success at having solved the general task, fails to reward sample efficient generalization that requires the ability to discover and make use of rich linguistic structures, and ultimately limits progress.