Building Scalable, Explainable, and Adaptive NLP Models with Retrieval

Natural language processing (NLP) has witnessed impressive developments in answering questions, summarizing or translating reports, and analyzing sentiment or offensiveness. Much of this progress is owed to training ever-larger language models, such as T5 or GPT-3, that use deep monolithic architectures to internalize how language is used within text from massive Web crawls. During training, these models distill the facts they read into implicit knowledge, storing in their parameters not only the capacity to "understand" language tasks, but also highly abstract knowledge representations of entities, events, and facts the model needs for solving tasks. Despite the well-publicized success of large language models, their black-box nature hinders key goals of NLP. Researchers continue to enlarge these models, leading to striking inefficiencies as the field already pushes past 1 trillion parameters.