Tens of thousands of papers involving A.I. are published each year, but it will take some time before many of them make their potential real-world impact clear. Meanwhile, the top funders of A.I. -- the Alphabets, Apples, Facebooks, Baidus, and other unicorns of this world -- continue to hone much of their most exciting technology behind closed doors. In other words, when it comes to artificial intelligence, it's impossible to do a rundown of the year's most important developments in the way that, say, you might list the 10 most listened-to tracks on Spotify. But A.I. has undoubtedly played an enormous role in 2020 in all sorts of ways. Here are six of the main developments and emerging themes seen in artificial intelligence during 2020.

Transformers and pre-trained models can be considered one of the most important developments in the recent years of deep learning. Beyond the research breakthroughts, Transformers have redefined the natural language understanding(NLU) space sparking a race between lead AI vendors to build bigger and more efficient neural networks. The Transformer architecture has been behind famous models such as Google's BERT, Facebook's RoBERTa or OpenAI's GPT-3. Is not surprising that many people believe that only big companies have the resources to tackle the implementation of Transformer models. Earlier this year, the deep learning community was astonished when Microsoft Research unveiled the Turing Natural Language Generation (T-NLG) model which, at the time, was considered the largest natural language processing(NLP) model in the history of artificial intelligence(AI) with 17 billion parameters.

As we predicted in our AI Trends 2020, NLP is the year's leading trend. But research on self-regenerating structures has likewise been both surprising and fascinating. February was rich in news on self-regenerating machines and NLP-related events and research. The tech behemoths are racing to build larger and more efficient language models, which are both costly and technologically challenging, if not daunting. Microsoft is a relatively new player in the game, and follows in the footsteps of Google with itsBERT model and OpenAI, which brought outGPT-2.

As claimed by the authors, the bigger the models, the better it generalizes to perform better on downstream tasks, especially in the low-data regime. This also means that the technology stack, including libraries and hardware, used to train these enormous language models have to be optimized, which is the innovative part proposed by T-NLG. Optimal hardware setup and a reduction in model-parallelism degree allow T-NLG to be trained more efficiently with fewer GPUs (256) than the state-of-the-art at the time, Megatron-ML (1024 NVIDIA GPUs). It will be interesting to see how Microsoft researchers apply T-NLG in production across their different products and build more fluent chatbots and digital assistants for improving customer experience. Transformer models efficiently extend the context window that helps to improve language understanding while at the same time requiring less computation and speeding up training via a self-attention mechanism.

A Generative Adversarial Network for AI-Aided Chair Design Researchers present a deep neural network for improving human design of chairs which consists of an image synthesis module and a super-resolution module. They select one of the candidates as a design prototype and create a real-life chair based on it. According to the researcher team, this is the first physical chair created with the help of deep neural networks, which bridges the gap between AI and design. This is the largest NLP model ever trained, with 17 billion parameters. T-NLG has achieved SOTA performance on mainstream NLP tasks.

New model has "real business impact" Microsoft has unveiled the world's largest deep learning language model to-date: a 17 billion-parameter "Turing Natural Language Generation (T-NLG)" model that the company believes will pave the way for more fluent chatbots and digital assistants. The T-NLG "outperforms the state of the art" on a several benchmarks, including summarisation and question answering, Microsoft claimed in a new research blog, as the company stakes its claim to a potentially dominant position in one of the most closely watched new technologies, natural language processing. Deep learning language models like BERT, developed by Google, have hugely improved the powers of natural language processing, by training on colossal data sets with billions of parameters to learn the contextual relations between words. Bigger is not always better, those working on language models may recognise, but Microsoft scientist Corby Rosset said his team "have observed that the bigger the model and the more diverse and comprehensive the pretraining data, the better it performs at generalizing to multiple downstream tasks even with fewer training examples." He emphasised: "Therefore, we believe it is more efficient to train a large centralized multi-task model and share its capabilities across numerous tasks."

Massive deep learning language models (LM), such as BERT and GPT-2, with billions of parameters learned from essentially all the text published on the internet, have improved the state of the art on nearly every downstream natural language processing (NLP) task, including question answering, conversational agents, and document understanding among others. Better natural language generation can be transformational for a variety of applications, such as assisting authors with composing their content, saving one time by summarizing a long piece of text, or improving customer experience with digital assistants. Following the trend that larger natural language models lead to better results, Microsoft is introducing Turing Natural Language Generation (T-NLG), the largest model ever published at 17 billion parameters, which outperforms the state of the art on a variety of language modeling benchmarks and also excels when applied to numerous practical tasks, including summarization and question answering. This work would not be possible without breakthroughs produced by the DeepSpeed library (compatible with PyTorch) and ZeRO optimizer, which can be explored more in this accompanying blog post. We are releasing a private demo of T-NLG, including its freeform generation, question answering, and summarization capabilities, to a small set of users within the academic community for initial testing and feedback.