AI21 Labs Asks: How Much Does It Cost to Train NLP Models?

These days the total cost of training an NLP model can climb into millions of dollars. And so it is only natural that budget-constrained researchers, engineers and scientists, when planning their model-training experiments, ask the important question: How much is this going to cost? And what are the main factors affecting that price tag? Israeli research company AI21 Labs looks for answers in their recently published paper The Cost of Training NLP Models: A Concise Overview. AI21 Labs Co-CEO, Stanford University Professor of Computer Science (emeritus), and AI Index initiator Yoav Shoham describes the motivation for the project.