We Might See A 100T Language Model In 2022

Looking back at 2021, it can surely be labelled as the year of large language models, with all the tech giants releasing models to stay ahead in the innovation game. In December itself, we saw back-to-back releases – DeepMind's 280 billion parameter transformer language model, Gopher, Google's Generalist Language Model (GLaM), a trillion weight model that uses sparsity, LG AI Research's artificial intelligence language model "Exaone", with capabilities of tuning 300 billion different parameters or variables. With innovations in language models accelerating at such a massive pace, can we possibly see a 100T large language model in the very near future? The idea is surely not too far-fetched if we look at the growth that tech companies have made, bringing out improved versions of the models that exist today in a span of just a few years. After the release of the GPT-3 autoregressive language model with 175 billion machine learning parameters from Open AI in 2020 (its predecessor, GPT-2, was over 100 times smaller, at 1.5 billion parameters), major efforts have gone into bringing out more such models by tech mammoths.