The Open Source Technologies Behind One of the Biggest Language Models in History

Transformers and pre-trained models can be considered one of the most important developments in the recent years of deep learning. Beyond the research breakthroughts, Transformers have redefined the natural language understanding(NLU) space sparking a race between lead AI vendors to build bigger and more efficient neural networks. The Transformer architecture has been behind famous models such as Google's BERT, Facebook's RoBERTa or OpenAI's GPT-3. Is not surprising that many people believe that only big companies have the resources to tackle the implementation of Transformer models. Earlier this year, the deep learning community was astonished when Microsoft Research unveiled the Turing Natural Language Generation (T-NLG) model which, at the time, was considered the largest natural language processing(NLP) model in the history of artificial intelligence(AI) with 17 billion parameters.