Microsoft's UniLM AI achieves state-of-the-art performance on summarization and language generation

Language model pretraining, a technique that "teaches" machine learning systems contextualized text representations by having them predict words based on their contexts, has advanced the state of the art across a range of natural language processing objectives. However, models like Google's BERT, which are bidirectional in design (meaning they draw on left-of-word and right-of-word context to form predictions), aren't well-suited to the task of natural language generation with substantial modification. That's why scientists at Microsoft Research investigated an alternative approach dubbed UNIfied pre-trained Language Model (UniLM), which completes unidirectional, sequence-to-sequence, and bidirectional prediction tasks and which can be fine-tuned for both natural language understanding and generation. They claim it compares favorably to BERT on popular benchmarks, achieving state-of-the-art results on a sampling of abstract summarization, generative question answering, and language generation data sets. UniLM is a multi-layer network at its core, made up of Transformer AI models jointly pretrained on large amounts of text and optimized for language modeling.