Google's BigBird Model Improves Natural Language and Genomics Processing

Researchers at Google have developed a new deep-learning model called BigBird that allows Transformer neural networks to process sequences up to 8x longer than previously possible. Networks based on this model achieved new state-of-the-art performance levels on natural-language processing (NLP) and genomics tasks. The team described the model and a set of experiments in a paper published on arXiv. BigBird is a new self-attention model that reduces the neural-network complexity of Transformers, allowing for training and inference using longer input sequences. By increasing sequence length up to 8x, the team was able to achieve new state-of-the-art performance on several NLP tasks, including question-answering and document summarization.